Turn Nginx logs into Prometheus metrics

Last update: Dec 13, 2022

Overview

Nginx log parser and Prometheus exporter

This service parses incoming syslog messages from Nginx sent over UDP and converts them into Prometheus metrics exported through the built-in HTTP server.

If the prefix-list is specified then per-URI statistics are generated. Using this feature without URI-prefix list is dangerous because it leads to an unbounded memory usage. It's therefore enabled only with a limited prefix list.

The URI prefix list is a plain-text file with a single prefix per line, e.g.
```
/api/call1.json
/api/call2.json
/api/call3.json
```
URIs received from nginx are stripped of any query parameters - only the part before '?' is used.
It optionally supports country lookup for client IPs using MaxMind GeoIP database.

Nginx configuration snippet

log_format collector '$remote_addr|$scheme|$host|$request_method|$server_protocol|$request_uri|$status|$request_time|$request_length|$bytes_sent';
access_log syslog:server=1.1.1.1:1514,tag=nginx collector;

Getting it

Get RPM & DEB directly from releases, only linux-amd64 build currently available.
Build youself:

go get github.com/blind-oracle/nginx-prometheus
cd $GOPATH/src/github.com/blind-oracle/nginx-prometheus
go build

Usage example

If you're using RPM/DEB packages then options are set in /etc/default/nginx-prometheus

./nginx-prometheus \
-debug false \
-listenSyslog 0.0.0.0:1514 \
-listenHTTP 0.0.0.0:1514 \
-geoipCountryDB /etc/nginx-prometheus/country.mmdb \
-uriPrefixFile /etc/nginx-prometheus/uriPrefixes.txt

debug - prints every syslog message received if true (defaults to false)
listenSyslog - ip:port on which to listen for UDP Syslog messages (defaults to 0.0.0.0:1514)
listenHTTP - ip:port on which to listen for incoming HTTP requests from Prometheus (defaults to 0.0.0.0:11080)
geoipCountryDB - path to MaxMind country GeoIP database (optional)
uriPrefixFile - path to a URI prefix list (optional)

Grafana dashboard

Add Prometheus source to Grafana and import grafana-dashboard.json for most common graphs.

Prometheus metrics for Go database/sql via VictoriaMetrics/metrics

sqlmetrics Prometheus metrics for Go database/sql via VictoriaMetrics/metrics Features Simple API. Easy to integrate. Install Go version 1.16+ go get

22 Dec 16, 2022

Export Prometheus metrics from journald events using Prometheus Go client library

journald parser and Prometheus exporter Export Prometheus metrics from journald events using Prometheus Go client library. For demonstration purposes,

0 Jan 3, 2022

Mogo: a lightweight browser-based logs analytics and logs search platform for some datasource(ClickHouse, MySQL, etc.)

mogo Mogo is a lightweight browser-based logs analytics and logs search platform

986 Dec 30, 2022

Exporter your cypress.io dashboard into prometheus Metrics

Cypress.io dashboard Prometheus exporter Prometheus exporter for a project from Cypress.io dashboards, giving the ability to alert, make special opera

4 Feb 8, 2022

Prometheus nginx socket exporter

stand-alone version of k8s/ingress-nginx' prometheus exporter based on openresty/lua-nginx-module how it works exporter opens socket (/tmp/prometheus-

4 Aug 8, 2022

Prometheus Common Data Exporter can parse JSON, XML, yaml or other format data from various sources (such as HTTP response message, local file, TCP response message and UDP response message) into Prometheus metric data.

Prometheus Common Data Exporter Prometheus Common Data Exporter 用于将多种来源(如http响应报文、本地文件、TCP响应报文、UDP响应报文)的Json、xml、yaml或其它格式的数据，解析为Prometheus metric数据。

7 May 18, 2022

Turn any program that uses STDIN/STDOUT into a WebSocket server. Like inetd, but for WebSockets.

websocketd websocketd is a small command-line tool that will wrap an existing command-line interface program, and allow it to be accessed via a WebSoc

16.6k Dec 31, 2022

turn a directory into a GUI, slash example of VNC-based GUI

dirgui @rsnous on Jan 11, 2021: "idea: filesystem-GUI adapter, where a directory turns into a form, executable files inside that directory turn into

32 May 3, 2022

yml2dot - Turn YAML into beautiful Graph

yml2dot - Turn YAML into beautiful Graph Use Cases Visualize your YAML files as Graph Generate additional info from your source code

111 Dec 30, 2022

Turn shell commands into web services

webify Turn functions and commands into web services For a real world example, see turning a Python function into a web service. Overview webify is a

928 Dec 22, 2022

Turn asterisk-indented text lines into mind maps

Crumbs Turn asterisk-indented text lines into mind maps. Organize your notes in a hierarchical tree structure, using a simple text editor. an asterisk

99 Jan 8, 2023

A CLI to turn Go's benchmark output into pictures

benchdraw benchdraw allows you to make easy to read picture plots from data in Go's benchmark format, implemented in pure Go. Benchdraw does not try t

346 Nov 28, 2022

GoPrisma - A Go wrapper for prisma to turn databases into GraphQL APIs using Go.

GoPrisma - a Go wrapper for the Prisma Engines What's this? Introspect a database and use it as a GraphQL API using Go. Supported Databases: SQLite Po

71 Dec 20, 2022

webify - Turn functions and commands into web services

webify is a very basic CGI server which forwards all requests to a single script. A design goal is to be as zero-config as possible.

928 Dec 22, 2022

Turn any key/value index into a high-performance two-dimensional spatial index

modular-spatial-index For the demo that this animated gif was generated from

5 Mar 6, 2022

Turn a Spotify playlist into a YouTube playlist

spotlist Turn a Spotify playlist into a YouTube playlist How to use: Download latest version from Releases. (Quick Links: Mac/Windows) Run it Enter Sp

5 Sep 8, 2022

A plugin that turn hashicorp vault into blockchain wallet.

dq-vault - Hashicorp vault BTC/ETH plugin This vault plugin stores a user's mnemonic inside vault in an encrypted manner. The plugin uses this stored

9 Dec 7, 2022

Turn a heterogeneous pile of text docs into a single web page with good search.

Codex Codex turns an unstructured pile of heterogeneous documents into a single interactive web document. Your input documents maybe in markdown, TeX,