A Prometheus metrics exporter for AWS that fills in gaps CloudWatch doesn't cover

Overview

YAAE (Yet Another AWS Exporter)

A Prometheus metrics exporter for AWS that fills in gaps CloudWatch doesn't cover

About

This exporter is meant to expose metrics that aren't included in CloudWatch for Prometheus to scrape. When run in conjuction with a CloudWatch exporter (such as YACE or the Prometheus Community CloudWatch Exporter), this exporter provides increased visbility into your AWS ecosystem.

A full list of scrapers and metrics is found below.

Image

cashapp/yet-another-aws-exporter:<TAG>

Exported Metrics

Features

  • Self-documenting scraper/metric structure architecture
  • All metrics/scrapers can be disabled
  • IAM permissions can be generated off only selected scrapers
  • Internal metrics for tracking error rates and scrape duration
  • Structured JSON logging

Configuration File

YAAE looks for a config file next to the binary yaae binary named yaae.yaml. To pass a custom path to a config file, use the -c flag:

yaae -c /path/to/yaae.yaml

A config file example can be found in the examples directory.

AWS IAM Permissions

This exporter should never require anything more than read-only permissions. If you're running every scraper available, you'll need the following permissions on the role/user that the exporter is running under:

{
    "Version": "2012-10-17",
    "Statement": [
        {
            "Sid": "yaae",
            "Effect": "Allow",
            "Action": [
                "eks:DescribeCluster",
                "eks:ListClusters",
                "ec2:DescribeVpcs",
                "ec2:DescribeSubnets",
                "iam:GetAccountSummary"
            ],
            "Resource": "*"
        }
    ]
}

Not using every exporter? Run yaae scrapers list-iam-permissions -c <PATH TO CONFIG FILE> to generate a new list of permissions.

Contributing

If you're interested in contributing, we'd love the help! See the contributing guidelines for initial details.

Comments
  • [Feature]: Allow Scrapers To Create Multiple Timeseries

    [Feature]: Allow Scrapers To Create Multiple Timeseries

    Is your feature request related to a problem? Please describe. There is an explicit 1-to-1 relationship between scrapers and timeseries, and this creates a some redundancy when two or more scrapers make the same API calls. For example, iam-resource-usage and iam-quota scrapers are essentially the same scraper implementation but are duplicated we can't return two different time series.

    Describe the solution you'd like The Scraper should support being able to return results for multiple metrics.

    Describe alternatives you've considered Being able to pass the output of one scraper to another so that only one scraper has to make the API call. This is tricky though with a simple goroutine setup and would overcomplicate things when we could just allow a scraper to return n timeseries.k

    Additional context This could also be used for the vpc-info and subnet-available-ips scrapers. We'll likely encounter lots of places where one or more timeseries make sense.

    enhancement 
    opened by jonwinton 1
  • Add Helm Chart Config Support

    Add Helm Chart Config Support

    Description

    This PR adds support for the yaae.yaml config to the Helm chart so scrapers can be disabled easily.

    Types of changes

    • [x] New feature (non-breaking change which adds functionality)

    It's a new Helm chart feature, but we're still in pre-0.1.x so only bumping as a patch upgrade of the chart.

    Checklist:

    • [x] My change requires a change to the documentation.
    • [x] I have updated the documentation accordingly.
    opened by jonwinton 0
  • Scrapers Support Multiple Metrics

    Scrapers Support Multiple Metrics

    Description

    As discussed in the issue, there is case for a Scraper supporting multiple timeseries from a single scrape function. This PR aims to add that functionality.

    The approach taken is to add a map of Metric structs to the Scraper so that each individual timeseries has an alias on the Scraper. This then translates into Scraper Functions returning a map of ScraperResult structs, with each key of the returned map matching the alias of the Metric on the Scraper.

    This is a large refactor, and thus changes a lot of things: docs, tests, and each individual scraper. Because of that, the PR includes the following:

    • Update the scraper docs to match the new structure
    • Consolidate scrapers which collect the same type of metric.
      • iam-quota and iam-resource-usage are consolidated
      • vpc-info and subnet-available-ips are consolidated
    • The Collector has to update to know to iterated through the Metrics property on a Scraper
    • The commands to list the scrapers need to update to look into Metrics
    • Metric docs are updated with the new metrics that are now exported

    Link To Issue

    Closes issue #1

    Types of changes

    • [ ] Bug fix (non-breaking change which fixes an issue)
    • [x] New feature (non-breaking change which adds functionality)
    • [ ] Breaking change (fix or feature that would cause existing functionality to not work as expected)

    Because the API is not yet stable, I'm marking this as a new feature.

    Checklist:

    • [x] My change requires a change to the documentation.
    • [x] I have updated the documentation accordingly.
    opened by jonwinton 0
  • [Docs]: Add Command Information

    [Docs]: Add Command Information

    Please describe the question you have or the feature that isn't documented

    Right now there are three commands that can be run:

    • yaae serve: starts the exporter server
    • yaae scrapers list: compiles list of scrapers in the project
    • yaae scrapers list-iam-permissions: outputs the AWS IAM permissions needed for scrapers

    We need to document each one of these commands and give context for when they're useful.

    documentation triage 
    opened by jonwinton 0
  • [Feature]: Per-Scraper Config

    [Feature]: Per-Scraper Config

    Is your feature request related to a problem? Please describe. Right now, our config file is only good for disabling scrapers and scrapers are "dumb" and capture all the values around a single resource. For example, the vpc-info scraper grabs info about ALL VPCs rather than looking at a list of VPCs.

    Most of the time, this is alright, but in cases where we want to have more control over how a scraper runs we should enable passing down config to each scraper.

    Describe the solution you'd like The config file should includes a scrapers map, with each key corresponding to a scraper's ID.

    scrapers:
        vpcInfo:
            ...map of config values
    

    Then we pass in this in to the scraper at scrape function invocation time.

    Describe alternatives you've considered At exporter startup time, we could could pass in the config to each scraper at registration time and store the config in memory for each invocation. This could save on yaml unmarshaling time for each invocation.

    Additional context N/A

    enhancement triage 
    opened by jonwinton 0
  • [Feature]: Calculate vCPU Usage Metric

    [Feature]: Calculate vCPU Usage Metric

    Is your feature request related to a problem? Please describe. For big consumers of EC2 instances, one of the more frustrating service quotas is vCPU usage. This is often a quota that, if hit, can inhibit scaling and create/exacerbate downtime. It would be nice to calculate the number of vCPUs in a region and expose that metric.

    Here's a doc on how to calculate usage: https://aws.amazon.com/premiumsupport/knowledge-center/ec2-on-demand-instance-vcpu-increase/

    Describe the solution you'd like Describe instances in a region and calculate the number of total vCPUs being used. Need to filter for only active running instances.

    Describe alternatives you've considered N/A

    Additional context N/A

    enhancement triage 
    opened by jonwinton 0
Releases(v0.0.2)
Owner
Cash App
Cash App
Openvpn exporter - Prometheus OpenVPN exporter For golang

Prometheus OpenVPN exporter Please note: This repository is currently unmaintain

Serialt 0 Jan 2, 2022
Amplitude-exporter - Amplitude charts to prometheus exporter PoC

Amplitude exporter Amplitude charts to prometheus exporter PoC. Work in progress

Andrey S. Kolesnichenko 1 May 26, 2022
Vulnerability-exporter - A Prometheus Exporter for managing vulnerabilities in kubernetes by using trivy

Kubernetes Vulnerability Exporter A Prometheus Exporter for managing vulnerabili

null 24 Dec 4, 2022
Netstat exporter - Prometheus exporter for exposing reserved ports and it's mapped process

Netstat exporter Prometheus exporter for exposing reserved ports and it's mapped

Amir Hamzah 0 Feb 3, 2022
📡 Prometheus exporter that exposes metrics from SpaceX Starlink Dish

Starlink Prometheus Exporter A Starlink exporter for Prometheus. Not affiliated with or acting on behalf of Starlink(™) ?? Starlink Monitoring System

DanOpsTech 82 Nov 25, 2022
Prometheus exporter for Chia node metrics

chia_exporter Prometheus metric collector for Chia nodes, using the local RPC API Building and Running With the Go compiler tools installed: go build

Kevin Retzke 33 Sep 19, 2022
NVIDIA GPU metrics exporter for Prometheus leveraging DCGM

DCGM-Exporter This repository contains the DCGM-Exporter project. It exposes GPU metrics exporter for Prometheus leveraging NVIDIA DCGM. Documentation

NVIDIA Corporation 229 Dec 7, 2022
A Prometheus exporter which scrapes metrics from CloudLinux LVE Stats 2

CloudLinux LVE Exporter for Prometheus LVE Exporter - A Prometheus exporter which scrapes metrics from CloudLinux LVE Stats 2 Help on flags: -h, --h

Tsvetan Gerov 1 Nov 2, 2021
Prometheus metrics exporter for libvirt.

Libvirt exporter Prometheus exporter for vm metrics written in Go with pluggable metric collectors. Installation and Usage If you are new to Prometheu

Jasper 3 Jul 4, 2022
Prometheus Exporter for Kvrocks Metrics

Prometheus Kvrocks Metrics Exporter This is a fork of oliver006/redis_exporter to export the kvrocks metrics. Building and running the exporter Build

Kvrocks Labs 13 Sep 7, 2022
A prometheus exporter which reports metrics about your Gmail inbox.

prometheus-gmail-exporter-go A prometheus exporter for gmail. Heavily inspired by https://github.com/jamesread/prometheus-gmail-exporter, but written

Richard Towers 3 Nov 15, 2022
LLS-Exporter exports fuel level sensor data (rs-485 lls protocol) as prometheus metrics

LLS Exporter LLS Exporter reads rs485/rs232 data from serial port, decodes lls protocol and exports fuel level sensor data as prometheus metrics. Lice

Viktor Kuzmin 0 Dec 14, 2021
Openshift's hpessa-exporter allows users to export SMART information of local storage devices as Prometheus metrics, by using HPE Smart Storage Administrator tool

hpessa-exporter Overview Openshift's hpessa-exporter allows users to export SMART information of local storage devices as Prometheus metrics, by using

Shachar Sharon 0 Jan 17, 2022
Exporter your cypress.io dashboard into prometheus Metrics

Cypress.io dashboard Prometheus exporter Prometheus exporter for a project from Cypress.io dashboards, giving the ability to alert, make special opera

Romain Guilmont 4 Feb 8, 2022
Github exporter for Prometheus metrics. Written in Go, with love ❤️

Github exporter for Prometheus This is a Github exporter for Prometheus metrics exposed by Github API. Written in Go with pluggable metrics collectors

Konradas Bunikis 2 Oct 5, 2022
Kepler (Kubernetes-based Efficient Power Level Exporter) uses eBPF to probe energy related system stats and exports as Prometheus metrics

kepler Kepler (Kubernetes Efficient Power Level Exporter) uses eBPF to probe energy related system stats and exports as Prometheus metrics Architectur

Sustainable Computing 193 Nov 29, 2022
Sensu-go-postgres-metrics - The sensu-go-postgres-metrics is a sensu check that collects PostgreSQL metrics

sensu-go-postgres-metrics Table of Contents Overview Known issues Usage examples

Scott Cupit 0 Jan 12, 2022
The metrics-agent collects allocation metrics from a Kubernetes cluster system and sends the metrics to cloudability

metrics-agent The metrics-agent collects allocation metrics from a Kubernetes cluster system and sends the metrics to cloudability to help you gain vi

null 0 Jan 14, 2022
Export Prometheus metrics from journald events using Prometheus Go client library

journald parser and Prometheus exporter Export Prometheus metrics from journald events using Prometheus Go client library. For demonstration purposes,

Mike Sgarbossa 0 Jan 3, 2022