Turn Nginx logs into Prometheus metrics

Overview

Nginx log parser and Prometheus exporter

Go Report Card Coverage Status Build Status Mentioned in Awesome Go

This service parses incoming syslog messages from Nginx sent over UDP and converts them into Prometheus metrics exported through the built-in HTTP server.

  • If the prefix-list is specified then per-URI statistics are generated. Using this feature without URI-prefix list is dangerous because it leads to an unbounded memory usage. It's therefore enabled only with a limited prefix list.

    The URI prefix list is a plain-text file with a single prefix per line, e.g.

    /api/call1.json
    /api/call2.json
    /api/call3.json

    URIs received from nginx are stripped of any query parameters - only the part before '?' is used.

  • It optionally supports country lookup for client IPs using MaxMind GeoIP database.

Nginx configuration snippet

log_format collector '$remote_addr|$scheme|$host|$request_method|$server_protocol|$request_uri|$status|$request_time|$request_length|$bytes_sent';
access_log syslog:server=1.1.1.1:1514,tag=nginx collector;

Getting it

  • Get RPM & DEB directly from releases, only linux-amd64 build currently available.

  • Build youself:

go get github.com/blind-oracle/nginx-prometheus
cd $GOPATH/src/github.com/blind-oracle/nginx-prometheus
go build

Usage example

If you're using RPM/DEB packages then options are set in /etc/default/nginx-prometheus

./nginx-prometheus \
-debug false \
-listenSyslog 0.0.0.0:1514 \
-listenHTTP 0.0.0.0:1514 \
-geoipCountryDB /etc/nginx-prometheus/country.mmdb \
-uriPrefixFile /etc/nginx-prometheus/uriPrefixes.txt
  • debug - prints every syslog message received if true (defaults to false)
  • listenSyslog - ip:port on which to listen for UDP Syslog messages (defaults to 0.0.0.0:1514)
  • listenHTTP - ip:port on which to listen for incoming HTTP requests from Prometheus (defaults to 0.0.0.0:11080)
  • geoipCountryDB - path to MaxMind country GeoIP database (optional)
  • uriPrefixFile - path to a URI prefix list (optional)

Grafana dashboard

Add Prometheus source to Grafana and import grafana-dashboard.json for most common graphs.

You might also like...
Prometheus metrics for Go database/sql via VictoriaMetrics/metrics

sqlmetrics Prometheus metrics for Go database/sql via VictoriaMetrics/metrics Features Simple API. Easy to integrate. Install Go version 1.16+ go get

Export Prometheus metrics from journald events using Prometheus Go client library

journald parser and Prometheus exporter Export Prometheus metrics from journald events using Prometheus Go client library. For demonstration purposes,

Mogo: a lightweight browser-based logs analytics and logs search platform for some datasource(ClickHouse, MySQL, etc.)
Mogo: a lightweight browser-based logs analytics and logs search platform for some datasource(ClickHouse, MySQL, etc.)

mogo Mogo is a lightweight browser-based logs analytics and logs search platform

Exporter your cypress.io dashboard into prometheus Metrics

Cypress.io dashboard Prometheus exporter Prometheus exporter for a project from Cypress.io dashboards, giving the ability to alert, make special opera

Prometheus nginx socket exporter
Prometheus nginx socket exporter

stand-alone version of k8s/ingress-nginx' prometheus exporter based on openresty/lua-nginx-module how it works exporter opens socket (/tmp/prometheus-

Prometheus Common Data Exporter can parse JSON, XML, yaml or other format data from various sources (such as HTTP response message, local file, TCP response message and UDP response message) into Prometheus metric data.
Prometheus Common Data Exporter can parse JSON, XML, yaml or other format data from various sources (such as HTTP response message, local file, TCP response message and UDP response message) into Prometheus metric data.

Prometheus Common Data Exporter Prometheus Common Data Exporter 用于将多种来源(如http响应报文、本地文件、TCP响应报文、UDP响应报文)的Json、xml、yaml或其它格式的数据,解析为Prometheus metric数据。

Turn any program that uses STDIN/STDOUT into a WebSocket server. Like inetd, but for WebSockets.

websocketd websocketd is a small command-line tool that will wrap an existing command-line interface program, and allow it to be accessed via a WebSoc

turn a directory into a GUI, slash example of VNC-based GUI

dirgui @rsnous on Jan 11, 2021: "idea: filesystem-GUI adapter, where a directory turns into a form, executable files inside that directory turn into

yml2dot - Turn YAML into beautiful Graph
yml2dot - Turn YAML into beautiful Graph

yml2dot - Turn YAML into beautiful Graph Use Cases Visualize your YAML files as Graph Generate additional info from your source code

Turn shell commands into web services
Turn shell commands into web services

webify Turn functions and commands into web services For a real world example, see turning a Python function into a web service. Overview webify is a

Turn asterisk-indented text lines into mind maps
Turn asterisk-indented text lines into mind maps

Crumbs Turn asterisk-indented text lines into mind maps. Organize your notes in a hierarchical tree structure, using a simple text editor. an asterisk

A CLI to turn Go's benchmark output into pictures

benchdraw benchdraw allows you to make easy to read picture plots from data in Go's benchmark format, implemented in pure Go. Benchdraw does not try t

GoPrisma - A Go wrapper for prisma to turn databases into GraphQL APIs using Go.

GoPrisma - a Go wrapper for the Prisma Engines What's this? Introspect a database and use it as a GraphQL API using Go. Supported Databases: SQLite Po

webify - Turn functions and commands into web services
webify - Turn functions and commands into web services

webify is a very basic CGI server which forwards all requests to a single script. A design goal is to be as zero-config as possible.

Turn any key/value index into a high-performance two-dimensional spatial index
Turn any key/value index into a high-performance two-dimensional spatial index

modular-spatial-index For the demo that this animated gif was generated from

Turn a Spotify playlist into a YouTube playlist
Turn a Spotify playlist into a YouTube playlist

spotlist Turn a Spotify playlist into a YouTube playlist How to use: Download latest version from Releases. (Quick Links: Mac/Windows) Run it Enter Sp

A plugin that turn hashicorp vault into blockchain wallet.
A plugin that turn hashicorp vault into blockchain wallet.

dq-vault - Hashicorp vault BTC/ETH plugin This vault plugin stores a user's mnemonic inside vault in an encrypted manner. The plugin uses this stored

Turn a heterogeneous pile of text docs into a single web page with good search.
Turn a heterogeneous pile of text docs into a single web page with good search.

Codex Codex turns an unstructured pile of heterogeneous documents into a single interactive web document. Your input documents maybe in markdown, TeX,

Turn a grid of frames (stored in a single image) into a video file
Turn a grid of frames (stored in a single image) into a video file

reel2vid Convert a video that is encoded into one large image as many images sid

Releases(v1.1.0)
Owner
Igor Novgorodov
Software/System Engineer
Igor Novgorodov
Gin Server Demo, with Features: Swagger UI, Prometheus, Shell scripts

Gin Server Gin Server Features Swagger UI Docker Prometheus Shell generator Production Cypher Service And we can get the authorization key SECURITY_CA

song 0 Nov 1, 2021
nginx exporter for Prometheus copy code from ingress-nginx use for monit nginx

nginx_exporter 介绍 从ingress-nginx官方代码中的expoter迁移出来 用来监控虚拟机上的nginx的expoter 基于官方 controller-v0.49.3 版本移植的代码 软件架构 nginx_socket 通过lua模块monitor.lua 将nginx l

null 0 Dec 8, 2021
nginx exporter for Prometheus copy code from ingress-nginx use for monit nginx

nginx_exporter 介绍 从ingress-nginx官方代码中的expoter迁移出来 用来监控虚拟机上的nginx的expoter 基于官方 controller-v0.49.3 版本移植的代码 软件架构 nginx_socket 通过lua模块monitor.lua 将nginx l

嘉联支付有限公司 3 Dec 20, 2021
Json-log-exporter - A Nginx log parser exporter for prometheus metrics

json-log-exporter A Nginx log parser exporter for prometheus metrics. Installati

horan 0 Jan 5, 2022
Nginx-Log-Analyzer is a lightweight (simplistic) log analyzer for Nginx.

Nginx-Log-Analyzer is a lightweight (simplistic) log analyzer, used to analyze Nginx access logs for myself.

Mao Mao 28 Oct 26, 2022
An implementation of an Ingress controller for NGINX and NGINX Plus

?? Help make the NGINX Ingress Controller better by participating in our survey!

null 0 Dec 17, 2021
The open and composable observability and data visualization platform. Visualize metrics, logs, and traces from multiple sources like Prometheus, Loki, Elasticsearch, InfluxDB, Postgres and many more.

The open-source platform for monitoring and observability. Grafana allows you to query, visualize, alert on and understand your metrics no matter wher

Grafana Labs 52.4k Nov 20, 2022
Flowlogs2metrics - Transform flow logs into metrics

Overview Flow-Logs to Metrics (a.k.a. FL2M) is an observability tool that consum

null 23 Nov 17, 2022
Sensu-go-postgres-metrics - The sensu-go-postgres-metrics is a sensu check that collects PostgreSQL metrics

sensu-go-postgres-metrics Table of Contents Overview Known issues Usage examples

Scott Cupit 0 Jan 12, 2022
The metrics-agent collects allocation metrics from a Kubernetes cluster system and sends the metrics to cloudability

metrics-agent The metrics-agent collects allocation metrics from a Kubernetes cluster system and sends the metrics to cloudability to help you gain vi

null 0 Jan 14, 2022