Type-safe Prometheus metrics builder library for golang

Overview

gotoprom

A Prometheus metrics builder

Build Status Coverage Status GoDoc Mentioned in Awesome Go

gotoprom offers an easy to use declarative API with type-safe labels for building and using Prometheus metrics. It doesn't replace the official Prometheus client but adds a wrapper on top of it.

gotoprom is built for developers who like type safety, navigating the code using IDEs and using a “find usages” functionality, making refactoring and debugging easier at the cost of performance and writing slightly more verbose code.

Motivation

Main motivation for this library was to have type-safety on the Prometheus labels, which are just a map[string]string in the original library, and their values can be reported even without mentioning the label name, just relying on the order they were declared in.

For example, it replaces:

httpReqs := prometheus.NewCounterVec(
    prometheus.CounterOpts{
        Name: "http_requests_total",
        Help: "How many HTTP requests processed, partitioned by status code and HTTP method.",
    },
    []string{"code", "method"},
)
prometheus.MustRegister(httpReqs)

// ...

httpReqs.WithLabelValues("404", "POST").Add(42)

With:

var metrics = struct{
	Reqs func(labels) prometheus.Counter `name:"requests_total" help:"How many HTTP requests processed, partitioned by status code and HTTP method."`
}

type labels struct {
	Code   int    `label:"code"`
	Method string `label:"method"`
}

gotoprom.MustInit(&metrics, "http")

// ...

metrics.Reqs(labels{Code: 404, Method: "POST"}).Inc()

This way it's impossible to mess the call by exchanging the order of "POST" & "404" params.

Usage

Define your metrics:

var metrics struct {
	SomeCounter                      func() prometheus.Counter   `name:"some_counter" help:"some counter"`
	SomeHistogram                    func() prometheus.Histogram `name:"some_histogram" help:"Some histogram with default prometheus buckets" buckets:""`
	SomeHistogramWithSpecificBuckets func() prometheus.Histogram `name:"some_histogram_with_buckets" help:"Some histogram with custom buckets" buckets:".01,.05,.1"`
	SomeGauge                        func() prometheus.Gauge     `name:"some_gauge" help:"Some gauge"`
	SomeSummaryWithSpecificMaxAge    func() prometheus.Summary   `name:"some_summary_with_specific_max_age" help:"Some summary with custom max age" max_age:"20m" objectives:"0.50,0.95,0.99"`

	Requests struct {
		Total func(requestLabels) prometheus.Count `name:"total" help:"Total amount of requests served"`
	} `namespace:"requests"`
}

type requestLabels struct {
	Service    string `label:"service"`
	StatusCode int    `label:"status"`
	Success    bool   `label:"success"`
}

Initialize them:

func init() {
	gotoprom.MustInit(&metrics, "namespace")
}

Measure stuff:

metrics.SomeGauge().Set(100)
metrics.Requests.Total(requestLabels{Service: "google", StatusCode: 404, Success: false}).Inc()

Custom metric types

By default, only some basic metric types are registered when gotoprom is intialized:

  • prometheus.Counter
  • prometheus.Histogram
  • prometheus.Gauge
  • prometheus.Summary

You can extend this by adding more types, for instance, if you want to observe time and want to avoid repetitive code you can create a prometheusx.TimeHistogram:

package prometheusx

import (
	"reflect"
	"time"

	"github.com/cabify/gotoprom"
	"github.com/cabify/gotoprom/prometheusvanilla"
	"github.com/prometheus/client_golang/prometheus"
)

var (
	// TimeHistogramType is the reflect.Type of the TimeHistogram interface
	TimeHistogramType = reflect.TypeOf((*TimeHistogram)(nil)).Elem()
)

func init() {
	gotoprom.MustAddBuilder(TimeHistogramType, RegisterTimeHistogram)
}

// RegisterTimeHistogram registers a TimeHistogram after registering the underlying prometheus.Histogram in the prometheus.Registerer provided
// The function it returns returns a TimeHistogram type as an interface{}
func RegisterTimeHistogram(name, help, namespace string, labelNames []string, tag reflect.StructTag) (func(prometheus.Labels) interface{}, prometheus.Collector, error) {
	f, collector, err := prometheusvanilla.BuildHistogram(name, help, namespace, labelNames, tag)
	if err != nil {
		return nil, nil, err
	}

	return func(labels prometheus.Labels) interface{} {
		return timeHistogramAdapter{Histogram: f(labels).(prometheus.Histogram)}
	}, collector, nil
}

// TimeHistogram offers the basic prometheus.Histogram functionality
// with additional time-observing functions
type TimeHistogram interface {
	prometheus.Histogram
	// Duration observes the duration in seconds
	Duration(duration time.Duration)
	// Since observes the duration in seconds since the time point provided
	Since(time.Time)
}

type timeHistogramAdapter struct {
	prometheus.Histogram
}

// Duration observes the duration in seconds
func (to timeHistogramAdapter) Duration(duration time.Duration) {
	to.Observe(duration.Seconds())
}

// Since observes the duration in seconds since the time point provided
func (to timeHistogramAdapter) Since(duration time.Time) {
	to.Duration(time.Since(duration))
}

So you can later define it as:

var metrics struct {
	DurationSeconds func() prometheusx.TimeHistogram `name:"duration_seconds" help:"Duration in seconds" buckets:".001,.005,.01,.025,.05,.1"`
}

func init() {
	gotoprom.MustInit(&metrics, "requests")
}

And use it as:

// ...
defer metrics.DurationSeconds().Since(t0)
// ...

Replacing metric builders

If you don't like the default metric builders, you can replace the DefaultInitializer with your own one.

Performance

Obviously, there's a performance cost to perform the type-safety mapping magic to the original Prometheus client's API.

In general terms, it takes 3x to increment a counter than with vanilla Prometheus, which is around 600ns (we're talking about a portion of a microsecond, less than a thousandth of a millisecond)

$ go test -bench . -benchtime 3s
goos: darwin
goarch: amd64
pkg: github.com/cabify/gotoprom
BenchmarkVanilla-4    	10000000	       387 ns/op
BenchmarkGotoprom-4   	 5000000	      1049 ns/op
PASS
ok  	github.com/cabify/gotoprom	10.611s

In terms of memory, there's a also a 33% increase in terms of space, and 3x increase in allocations:

$ go test -bench . -benchmem
goos: darwin
goarch: amd64
pkg: github.com/cabify/gotoprom
BenchmarkVanilla-4    	 5000000	       381 ns/op	     336 B/op	       2 allocs/op
BenchmarkGotoprom-4   	 1000000	      1030 ns/op	     432 B/op	       6 allocs/op
PASS
ok  	github.com/cabify/gotoprom	3.369s

This costs are probably assumable in most of the applications, especially when measuring network accesses, etc. which are magnitudes higher.

Comments
  • Remove the default DefaultBuckets, make them configurable

    Remove the default DefaultBuckets, make them configurable

    When buckets for a Histogram are not defined explicitly, we set some DefaultBuckets: https://github.com/cabify/gotoprom/blob/b6bf805c5c9ff2e7dad23c127ef5e9876453c522/prometheusvanilla/builders.go#L123-L125

    Those may fit you or may not, fit you. We actually already changed them once and that was a breaking change for our customers.

    I'd say that there should not be any default values and everyone should provide explicitly the buckets they want.

    OTOH, since there's a clear wish for having default values here, we can make the default values being optional.

    So I think that the solution is a Builder factory, like:

    // HistogramBuilder returns a builder for the `prometheus.Histogram` metric type with the
    // default buckets set to the provided ones.
    // Default buckets will be used if specific ones were not provided through a tag.
    // If default buckets are set to nil and metric does not provide ones, metric building will fail. 
    func HistogramBuilder(defaultBuckets []float64) gotoprom.Builder {
        ...
    }
    

    With default initializer configured as prometheusvanilla.HistogramBuilder(nil)

    opened by colega 1
  • Release v1.0.0

    Release v1.0.0

    We've been stuck to v0.x.y because github.com/prometheus/client_golang was on v0.i.j, but since official prometheus v1.0.0 is out and considered stable, we can release a stable version too (with some breaking changes, see the milestone)

    opened by colega 1
  • Change the builder for prometheus.Observer by a builder for prometheus.Histogram

    Change the builder for prometheus.Observer by a builder for prometheus.Histogram

    Initially we didn't offer a builder for summaries and we used the prometheus.Observer target interface to bind the builder for prometheus.Histogram. That's not exactly correct and the v1.0.0 release is a good moment to fix that.

    This would be a breaking change so we should reduce the pain on this for our customers (since upgrading from v0 to v1 does not change the package).

    I'd propose to rename BuildObserver to BuildHistogram and register that builder for prometheus.Histogram and stop registering it for prometheus.Observer.

    State that in the release notice allowing people to:

    • Change their code to the new type in the builder (otherwise it would not build the metrics)
    • Register BuildHistogram as a builder for prometheus.Observer in init() of their main package if someone depends on a third party code building the metrics which doesn't update.
    opened by colega 1
  • Allow passing summary objectives through the tags

    Allow passing summary objectives through the tags

    Prometheus removed the default 0.5, 0.9 and 0.95 summary objectives in the v1.0.0, so now the summaries this library builds have no objectives at all, and act like an histogram with no buckets (only a sum and a count are exposed).

    We should add the ability of providing those objectives through the tags of the summary when it's defined.

    good first issue 
    opened by colega 1
  • Added support of empty values for labels

    Added support of empty values for labels

    Maybe I should call it "default"? We can explicitly report a value if none was set on a string. It can become too repetitive in the code if you have 5 labels in a metric.

    opened by colega 1
  • Use golangci-lint as linter

    Use golangci-lint as linter

    This is faster than goimports and includes more linters.

    Also removed non user-runnable targets from Makefile.

    Finally, applied changes to the code to comply with the code.

    opened by colega 0
  • Summaries: require objectives to be explicit

    Summaries: require objectives to be explicit

    Just like https://github.com/cabify/gotoprom/pull/27 but for Summaries, same reasons apply, TL;DR: explicit is better than implicit.

    Also fixed summary builder not failing when objectives were malformed.

    opened by colega 0
  • Histograms: require buckets to be explicit

    Histograms: require buckets to be explicit

    Default buckets were removed and buckets are now required: writing the buckets is a 5 second effort while forgetting about them and having wrong buckets in production is a lot of work (and maybe even an incident if wrong buckets don't provide enough visibility on an issue)

    So now any code without buckets on the histograms will fail to intialize that metrics.

    Finally, empty buckets tag is now allowed and it builds an empty buckets slice, which will cause prometheus library to use its own default buckets.

    Notice that this change identified an issue in one of our own tests where buckets tag was wrong (metricsbuckets) and it was unnoticed because default buckets where used instead.

    Solves https://github.com/cabify/gotoprom/issues/23

    opened by colega 0
  • Build prometheus.Histogram instead of Observer

    Build prometheus.Histogram instead of Observer

    The unprecise usage of prometheus.Observer instead of prometheus.Histogram may be misleading, since both Histogram and Summary interfaces embed the Observer interface.

    This is a breaking change since previous code won't compile with this version, so there's no better moment (except for v2) for doing this than going v1.

    Solves https://github.com/cabify/gotoprom/issues/21 although without offering a backwards compatible mechanism because of how packages are loaded and initialized.

    opened by colega 0
  • Add objectives to summaries from struct tag

    Add objectives to summaries from struct tag

    Prometheus has deprecated the default values for objectives and they now need to be explicitly specified. Default values are those that client_golang library had before deprecation in v1. If no objectives are specified, default objectives are used.

    Solves https://github.com/cabify/gotoprom/issues/20.

    opened by mapno 0
  • Shorten default list of buckets

    Shorten default list of buckets

    From 13 time series to 7 generated in a histogram as a "sensible default" with network requests in mind. These can be overwritten by any metric definition.

    opened by simonsargeant 0
Releases(v1.1.0)
  • v1.1.0(Jan 29, 2020)

  • v1.0.0(Jan 10, 2020)

    Added

    • Support for empty buckets tag, which will generate nil buckets for the prometheus Histogram and use default prometheus buckets.
    • Support for empty objectives tag, which will generate nil objectives for the prometheus Summary and use an empty objectives map after all.

    Changed

    • Breaking: prometheus.Histogram is now used to build histograms, instead of prometheus.Observer, which means that previous code building prometheus.Observer won't compile anymore.

    Removed

    • Breaking: default buckets on histograms. All histogram should explicitly specify their buckets now or they will fail to build.
    • Breaking: default objectives on summaries. All summaries should explicitly specify their objectives now or they will fail to build.

    Fixed

    • Summary building was not failing with malformed objectives.

    Migration instructions

    If you're migrating from a v0.x.y, you will need to:

    • Replace Metric func() prometheus.Observer by Metric func() prometheus.Histogram
    • On prometheus.Histogram metrics, add buckets:"" which will use the prometheus.DefBuckets bucekts, or even better, define yours.
    • On prometheus.Summary metrics, add objectives:".5,.95,.99" to keep using the same objectives, or define yours.
    Source code(tar.gz)
    Source code(zip)
  • v0.3.0(Oct 10, 2019)

  • v0.2.1(Jun 5, 2019)

  • v0.1.1(May 8, 2019)

Owner
Cabify
Cabify
Generate type-safe Go converters by simply defining an interface

goverter a "type-safe Go converter" generator goverter is a tool for creating type-safe converters. All you have to do is create an interface and exec

Jannis Mattheis 175 Nov 23, 2022
Type-safe atomic values for Go

Type-safe atomic values for Go One issue with Go's sync/atomic package is that there is no guarantee from the type system that operations on an intege

Alec Thomas 17 Apr 8, 2022
[TOOL, CLI] - Filter and examine Go type structures, interfaces and their transitive dependencies and relationships. Export structural types as TypeScript value object or bare type representations.

typex Examine Go types and their transitive dependencies. Export results as TypeScript value objects (or types) declaration. Installation go get -u gi

Daniel T. Gorski 154 Nov 2, 2022
AutoK3s GEO collects metrics about locates remote IP-address and exposes metrics to InfluxDB.

AutoK3s GEO AutoK3s GEO collects metrics about locates remote IP-address and exposes metrics to InfluxDB. Thanks to https://freegeoip.live/ which prov

Jason 0 Jun 16, 2022
A tool to run queries in defined frequency and expose the count as prometheus metrics.

A tool to run queries in defined frequency and expose the count as prometheus metrics. Supports MongoDB and SQL

S Santhosh Nagaraj 19 Jul 1, 2022
Prometheus support for go-metrics

go-metrics-prometheus This is a reporter for the go-metrics library which will post the metrics to the prometheus client registry . It just updates th

Csergő Bálint 76 Nov 13, 2022
rsync wrapper (or output parser) that pushes metrics to prometheus

rsync-prom An rsync wrapper (or output parser) that pushes metrics to prometheus. This allows you to then build dashboards and alerting for your rsync

Michael Stapelberg 34 Nov 25, 2022
🚀 Backend for Online Courses Builder | SaaS Product

Backend app for Online Course Constructor Platform Build & Run (Locally) Prerequisites go 1.15 docker golangci-lint (optional, used to run code checks

Maksim Zhashkevych 102 Nov 25, 2022
A Visual Go REST API boilerplate builder.

A Visual Go REST API boilerplate builder. The boilerplate builder will export a Go web server with 0 dependencies, besides the ones you add.

The Strukture IDE 55 Jul 8, 2022
Go port of Coda Hale's Metrics library

go-metrics Go port of Coda Hale's Metrics library: https://github.com/dropwizard/metrics. Documentation: http://godoc.org/github.com/rcrowley/go-metri

Richard Crowley 3.3k Nov 27, 2022
atomic measures + Prometheus exposition library

About Atomic measures with Prometheus exposition for the Go programming language. This is free and unencumbered software released into the public doma

Pascal S. de Kloe 24 Sep 27, 2022
Prometheus instrumentation library for Go applications

Prometheus Go client library This is the Go client library for Prometheus. It has two separate parts, one for instrumenting application code, and one

Prometheus 4.2k Nov 21, 2022
Toy program for benchmarking safe and unsafe ways of saving a file

save-a-file benchmarks the many strategies an editor could use to save a file. Example output on a SSD: ext4: $ ./save-a-file ~/tmp/foo 29.195µs per s

null 2 Sep 13, 2021
Start of a project that would let people stay informed about safe running spaces in their area.

SafeRun Start of a project that would let people stay informed about safe running spaces in their area. Too many people I'm friends with feel unsafe w

Ryan Dunning 0 Feb 11, 2022
In one particular project, i had to import some key/value data to Prometheus. So i have decided to create my custom-built Node Exporter in Golang.

In one particular project, i had to import some key/value data to Prometheus. So i have decided to create my custom-built Node Exporter in Golang.

Hamid Hosseinzadeh 1 May 19, 2022
Analyze the binary outputted by `go build` to get type information etc.

Analyze the binary outputted by go build to get type information etc.

Masaaki Goshima 16 Oct 5, 2022
a tool for getting metrics in containers

read metrics in container if environment is container, the cpu ,memory is relative to container, else the metrics is relative to host. juejing link :

null 6 Oct 13, 2022
Collect and visualize metrics from Brigade 2

Brigade Metrics: Monitoring for Brigade 2 Brigade Metrics adds monitoring capabilities to a Brigade 2 installation. It utilizes Brigade APIs to export

Brigade 3 Sep 8, 2022
IBus Engine for GoVarnam. An easy way to type Indian languages on GNU/Linux systems.

IBus Engine For GoVarnam An easy way to type Indian languages on GNU/Linux systems. goibus - golang implementation of libibus Thanks to sarim and haun

Varnamproject 10 Feb 10, 2022