nginx exporter for Prometheus copy code from ingress-nginx use for monit nginx

Overview

nginx_exporter

介绍

从ingress-nginx官方代码中的expoter迁移出来 用来监控虚拟机上的nginx的expoter
基于官方 controller-v0.49.3 版本移植的代码

软件架构

nginx_socket 通过lua模块monitor.lua 将nginx log 以json格式发到 /tmp/prometheus-nginx.socket, nginx_exporter 通过这个socket获得数据并组装成metrics。

nginx_process 通过采集 /proc/PID/ 目录下面的数据,监控cpu、memory、IO ;

nginx_status 通过 http_stub_status_module 模块采集nginx的连接数据,nginx编译时要加上 --with-http_stub_status_module; 配置文件要加上

    #监控使用端口
    server {
        listen 8021;
        location /stub_status{
           stub_status on;
           access_log off;
           allow 127.0.0.1;
           # deny all;
        }
    }        

nginx_certificate 证书监控模块,待完成

依赖

1) nginx 必须要编译有lua模块
2) lua 必须要有 cjson 模块

安装luajit 和 cjson

yum install gcc -y

cd /usr/local/src/
wget --no-check-certificate https://luajit.org/download/LuaJIT-2.0.5.zip 
unzip LuaJIT-2.0.5.zip
cd LuaJIT-2.0.5/
make install PREFIX=/usr/local/luajit 

cd /usr/local/src/
wget --no-check-certificate https://kyne.com.au/~mark/software/download/lua-cjson-2.1.0.zip
unzip lua-cjson-2.1.0.zip 
cd lua-cjson-2.1.0/
# 这里要修改makefile文件,不然编译报错
sed -i 's#^LUA_INCLUDE_DIR = .*#LUA_INCLUDE_DIR =   /usr/local/src/LuaJIT-2.0.5/src#' Makefile
make && make install 

使用说明

  1. 将Lua脚本copy到 /data/nginx/lua 目录(这个目录可以自己定义,和nginx配置文件一致就行);

  2. 修改nginx的http模块配置,新增如下配置

http {

    # lua脚本的目录路径
    lua_package_path "/data/nginx/lua/?.lua;;";

    init_by_lua_block {
        collectgarbage("collect")

        -- init modules
        local ok, res

        ok, res = pcall(require, "monitor")
        if not ok then
                error("require failed: " .. tostring(res))
        else
                monitor = res
        end

        ok, res = pcall(require, "plugins")
        if not ok then
                error("require failed: " .. tostring(res))
        else
                plugins = res
        end
        -- load all plugins that'll be used here
        plugins.init({  })
    }

    init_worker_by_lua_block {
        monitor.init_worker(10000)
        plugins.run()
    }

    log_by_lua_block {
        monitor.call()
        plugins.run()
    }
    
......
}
  1. 启动nginx_exproter
# 编译ngx_exporter
git clone https://gitee.com/xianglinzeng/nginx_exporter.git
cd nginx_exporte
go mod tidy
go build -o nginx_exporter

# nginx_exporter 参数
# -port 指定启动端口,默认9123端口
# -v    指定日志级别  1 2 3 4 5 越高日志越详细,默认是2,不指定也行,调试使用5
#  -statuspath string
#    	http_stub_status_module 模块的监听路径,默认/stub_status (default "/stub_status")
#  -statusport string
#    	http_stub_status_module 模块的监听端口,默认8021 (default "8021")
./nginx_exporter -port=9999 -v=5


# 使用systemd管理
mkdir /opt/nginx_exporter
cp nginx_exporter /opt/nginx_exporter

cat <<EOF > /usr/lib/systemd/system/nginx_exporter.service
[Unit]
Description=nginx_exporter
After=network.target

[Service]
User=root
Group=root
Type=simple
ExecStart=/opt/nginx_exporter/nginx_exporter -port=9123 -v=2
Restart=on-failure

[Install]
WantedBy=multi-user.target
EOF

systemctl daemon-reload
systemctl daemon-reload
systemctl restart nginx_exporter
systemctl status nginx_exporter.service
systemctl enable nginx_exporter

curl localhost:9123/metrics
  1. 使用ServiceMonitor添加到prometheus-opertor
---
apiVersion: monitoring.coreos.com/v1
kind: ServiceMonitor
metadata:
  labels:
    prometheus: k8s
    k8s-apps: nginx-exporter
  name: nginx-exporter-sm
  namespace: monitoring
spec:
  endpoints:
  - port: metrics
    interval: 10s
    Scheme: http
    path: /metrics
  jobLabel: k8s-app
  selector:
    matchLabels:
      metrics: nginx-exporter

---
apiVersion: v1
kind: Service
metadata:
  labels:
    # ServiceMonitor 自动发现的关键label
    metrics: nginx-exporter
  name: nginx-exporter
  namespace: monitoring
spec:
  ports:
  - name: metrics
    #对应 ServiceMonitor 中spec.endpoints.port
    port: 9123
    targetPort: 9123
---
apiVersion: v1
kind: Endpoints
metadata:
  name: nginx-exporter
  namespace: monitoring
  labels:
    metrics: nginx-exporter
subsets:
  - addresses:
      - ip: 172.20.4.117
    ports:
      - name: metrics
        port: 9123
        protocol: TCP 

告警规则

报警名称 表达式 采集数据时间(分钟) 报警触发条件
NginxHighHttp4xxErrorRate sum(rate(nginx_http_requests_total{status=~"^4.."}[1m])) / sum(rate(nginx_http_requests_total[1m])) * 100 > 5 5 HTTP 4xx错误率过高。
NginxHighHttp5xxErrorRate sum(rate(nginx_http_requests_total{status=~"^5.."}[1m])) / sum(rate(nginx_http_requests_total[1m])) * 100 > 5 5 HTTP 5xx错误率过高。
NginxLatencyHigh histogram_quantile(0.99, sum(rate(nginx_http_request_duration_seconds_bucket[10m])) by (host, node)) > 10 5 延迟过高。

grafana

grafana的dashboard在grafana目录,预览如下:

输入图片说明

输入图片说明

输入图片说明

You might also like...
kubectl plugin for generating nginx-ingress compatible basic-auth secrets on kubernetes clusters

kubectl-htpasswd kubectl plugin for easily generating hashed basic auth secrets. Supported hash algorithms bcrypt Examples Create the secret on the cl

Github billing exporter - Billing exporter for GitHub organizations

GitHub billing exporter Forked From: https://github.com/borisputerka/github_bill

📡  Prometheus exporter that exposes metrics from SpaceX Starlink Dish
📡 Prometheus exporter that exposes metrics from SpaceX Starlink Dish

Starlink Prometheus Exporter A Starlink exporter for Prometheus. Not affiliated with or acting on behalf of Starlink(™) 📡 Starlink Monitoring System

Prometheus exporter for Chia node metrics

chia_exporter Prometheus metric collector for Chia nodes, using the local RPC API Building and Running With the Go compiler tools installed: go build

Nvidia GPU exporter for prometheus using nvidia-smi binary
Nvidia GPU exporter for prometheus using nvidia-smi binary

nvidia_gpu_exporter Nvidia GPU exporter for prometheus, using nvidia-smi binary to gather metrics. Introduction There are many Nvidia GPU exporters ou

NVIDIA GPU metrics exporter for Prometheus leveraging DCGM

DCGM-Exporter This repository contains the DCGM-Exporter project. It exposes GPU metrics exporter for Prometheus leveraging NVIDIA DCGM. Documentation

Prometheus exporter for Amazon Elastic Container Service (ECS)

ecs_exporter 🚧 🚧 🚧 This repo is still work in progress and is subject to change. This repo contains a Prometheus exporter for Amazon Elastic Contai

Prometheus exporter for DeadMansSnitch

DeadMansSnitch Exporter Prometheus exporter for DeadMansSnitch information (snitches) Configuration Usage: deadmanssnitch-exporter [OPTIONS] Applic

A prometheus exporter for monitoring FIO nodeos nodes.
A prometheus exporter for monitoring FIO nodeos nodes.

fio-prometheus-exporter This is a simple prometheus exporter for FIO nodeos nodes. It can connect to multiple nodes to display a few critical statisti

Owner
嘉联支付有限公司
嘉联支付有限公司
K8s-ingress-health-bot - A K8s Ingress Health Bot is a lightweight application to check the health of the ingress endpoints for a given kubernetes namespace.

k8s-ingress-health-bot A K8s Ingress Health Bot is a lightweight application to check the health of qualified ingress endpoints for a given kubernetes

Aaron Tam 0 Jan 2, 2022
Json-log-exporter - A Nginx log parser exporter for prometheus metrics

json-log-exporter A Nginx log parser exporter for prometheus metrics. Installati

horan 0 Jan 5, 2022
An implementation of an Ingress controller for NGINX and NGINX Plus

?? Help make the NGINX Ingress Controller better by participating in our survey!

null 0 Dec 17, 2021
Openvpn exporter - Prometheus OpenVPN exporter For golang

Prometheus OpenVPN exporter Please note: This repository is currently unmaintain

Serialt 0 Jan 2, 2022
Amplitude-exporter - Amplitude charts to prometheus exporter PoC

Amplitude exporter Amplitude charts to prometheus exporter PoC. Work in progress

Andrey S. Kolesnichenko 1 May 26, 2022
Vulnerability-exporter - A Prometheus Exporter for managing vulnerabilities in kubernetes by using trivy

Kubernetes Vulnerability Exporter A Prometheus Exporter for managing vulnerabili

null 23 Aug 17, 2022
Netstat exporter - Prometheus exporter for exposing reserved ports and it's mapped process

Netstat exporter Prometheus exporter for exposing reserved ports and it's mapped

Amir Hamzah 0 Feb 3, 2022
Simple HPA base NGINX Ingress

Simple HPA Base Ingress Access Log Support NGINX Ingress How to Use Requirement Kubernetes NGINX Ingress Quick Start 1. Deploy Change deploy.yamlENV,

有关部门 2 Dec 27, 2021
A Pulumi NGINX Ingress Controller component

Pulumi NGINX Ingress Controller Component This repo contains the Pulumi NGINX Ingress Controller component for Kubernetes. This ingress controller use

Pulumi 8 Aug 10, 2022
An Ingress controller for Kubernetes using NGINX as a reverse proxy and load balancer

NGINX Ingress Controller Overview ingress-nginx is an Ingress controller for Kubernetes using NGINX as a reverse proxy and load balancer. Learn more a

null 0 Nov 15, 2021