Nginx-Log-Analyzer is a lightweight (simplistic) log analyzer for Nginx.

Overview

Nginx-Log-Analyzer

Actions Status codecov Go Version Go Report Card Release License

README English | 中文

What is it

Nginx-Log-Analyzer is a lightweight (simplistic) log analyzer, used to analyze Nginx access logs for myself.

Nginx-Log-Analyzer is written in Go programming language, needs only a 2 MB executable file to run, currently supported features are as follows:

  • Filter logs based on the request time
  • Support multiple log format configurations
    • combined (Nginx default configuration)
    • JSON
  • Analyze multiple files at the same time
  • Analyze .gz compressed files
  • Support a variety of statistical indicators

Advantages compared to GoAccess

GoAccess is an excellent and powerful real-time web log analyzer, interactive viewer that runs in a terminal in *nix systems or through your browser. But as far as I know, GoAccess seems does not support counting URI response time by percentile, Nginx-Log-Analyzer supports this feature.

If I knew about GoAccess before developing Nginx-Log-Analyzer, I might choose to use it directly. GoAccess is so powerful, I love GoAccess.

Advantages compared to ELK

Although ELK is powerful, it is troublesome to install and configure, and it also has certain requirements for machine performance. Nginx-Log-Analyzer is more lightweight and easier to use, suitable for some simple log analysis scenarios.

Quick start

Installation

Just download the binary executable file for the corresponding platform from the GitHub Release page of Nginx-Log-Analyzer.

GeoIP2 and GeoLite2

GeoIP2 is a commercial IP geolocation database, need to pay to use it. GeoLite2 is a free and low-precision version of GeoIP2, distribute by Attribution-ShareAlike 4.0 International license, download by logging in to the MaxMind official website.

When using Nginx-Log-Analyzer, if you need to resolve the geographic location of the IP (that is, use the -t 4 mode), then you will need to download the GeoIP2 or GeoLite2 City database file, save it to the City.mmdb file in the default configuration directory ${HOME}/.config/nginx-log-analyzer/. The corresponding shell commands are as follows:

~$ mkdir -p ${HOME}/.config/nginx-log-analyzer
~$ tar -xzf GeoLite2-City_20211109.tar.gz
~$ cp GeoLite2-City_20211109/GeoLite2-City.mmdb ${HOME}/.config/nginx-log-analyzer/City.mmdb

Configure Nginx

Nginx-Log-Analyzer parses Nginx access logs in combined format by default, which means that the logs will contain the following fields:

  • $remote_addr
  • $remote_user
  • $time_local
  • $request
  • $status
  • $body_bytes_sent
  • $http_referer
  • $http_user_agent

When using Nginx-Log-Analyzer, if you need more types of statistical indicators, then you will need to use the -lf json option to specify the log parsing mode to the JSON format, and need to add the following log_format and access_log directives in the Nginx configuration:

log_format json_log escape=json '{"remote_addr":"$remote_addr",'
                                '"time_local":"$time_local",'
                                '"request":"$request",'
                                '"status":$status,'
                                '"body_bytes_sent":$body_bytes_sent,'
                                '"http_user_agent":"$http_user_agent",'
                                '"request_time":$request_time}';
access_log /path/to/access.json.log json_log;
  • The log_format directive can only appear in the http context;
  • The access_log directive could appear in the http, server, location context, and should use the log_format declared above;
  • You can make multiple access_logs at the same time without deleting the original configuration. e.g.
    access_log /path/to/access.log;
    access_log /path/to/access.json.log json_log;
    

Related document: http://nginx.org/en/docs/http/ngx_http_log_module.html

Command line options

show version -v

The -v options show Nginx-Log-Analyzer's build version, build time, and Git Commit at build time.

specify the configuration directory -d

The -d option specify the configuration directory that Nginx-Log-Analyzer required at runtime, the default value is ${HOME}/.config/nginx-log-analyzer/.

specify the log format -lf

The -lf option specify the log format parsed by Nginx-Log-Analyzer, available values are combined and json, the default value is combined.

specify the analysis type -t

The -t option specify the type of this analysis, the analysis type and corresponding statistical indicators are as follows:

Supported Analysis Type -t Statistical Indicators Required Fields or Libraries
0 PV and UV $remote_addr
1 Most visited IPs $remote_addr
2 Most visited URIs $request
3 Most visited User-Agents $http_user_agent
4 Most visited user countries and cities $remote_addr, MaxMind GeoIP2 or GeoLite2 City Database
5 Most frequent response status $status, $request
6 Largest average response time URIs $request, $request_time
7 Largest percentile response time URIs, e.g. p1(min), p50(median), p95, p100(max) $request, $request_time

limit the analysis start and end time -ta -tb

-ta and -tb options are used to filter logs based on the request time, ta is the abbreviation of time after, tb is the abbreviation of time before.

-ta and -tb options required the $time_local field in log_format directive of Nginx configuration.

limit the output lines number -n -n2

-n and -n2 options are used to limit the number of output lines of Nginx-Log-Analyzer, -n2 option only works in -t 4 mode.

specify the percentile value -p

The -p option specify the percentile value in the -t 7 mode, the default value is 95.

Usages

Filter logs based on the request time

image

Analyze multiple files at the same time

image

Analyze .gz compressed files

image

Count the most visited IPs

image

Count the most visited URIs

image

Count the most visited User-Agents

image

Count the most visited countries and cities

image

Count the most frequently response status

image

count the largest URI average response times

image

count the largest URI percentile response times

image

FQA

Q: Will it support real-time analysis in the future?

A: No. If you want this feature, it is recommended to use solutions such as GoAccess, ELK, Grafana + Time Series DBMS.

License

GeoLite2 Database License

Nginx-Log-Analyzer License

Copyright (c) 2021 fantasticmao

Owner
Mao Mao
Java developer, also interested in C, Go, JavaScript, Python.
Mao Mao
Log-analyzer - Log analyzer with golang

Log Analyzer what do we have here? Objective Installation and Running Applicatio

Lawrence Agbani 0 Jan 27, 2022
Log-server - Implement log server for gwaylib/log/adapter/rmsq

Implement server of github.com/gwaylib/log Base on https://github.com/gwaycc/lserver Build . env.sh cd cmd/web go build Deploy Install supd(Debian sy

null 0 Jan 3, 2022
Nginx JSON Log Analyze

Nginx-JSON-Log-Analyze Nginx Configuration log_format json_log escape=json '{"time_iso8601":"$time_iso8601",' '"remote

Mao Mao 24 Jul 31, 2022
BRUS - Parses your web server (e.g. nginx) log files and checks with GreyNoise how much noise your website is exposed to.

BRUS bbbbbb rrrrrr u u sssss b b r r u u s bbbbbb rrrrrr u u sssss b b r r u u s bbbbbb r r

dubs3c 1 May 29, 2022
An golang log lib, supports tracking and level, wrap by standard log lib

Logex An golang log lib, supports tracing and level, wrap by standard log lib How To Get shell go get gopkg.in/logex.v1 source code import "gopkg.in/

chzyer 39 Apr 15, 2022
Distributed-Log-Service - Distributed Log Service With Golang

Distributed Log Service This project is essentially a result of my attempt to un

Hamza Yusuff 6 Jun 1, 2022
EdgeLog is a lightweight log management system, and Agent is a part of EdgeLog system

EdgeLog is a lightweight log management system, and Agent is a part of EdgeLog system. It is installed on host machine and its main duty is to collect host program log statics.

null 2 Mar 2, 2022
a golang log lib supports level and multi handlers

go-log a golang log lib supports level and multi handlers Use import "github.com/siddontang/go-log/log" //log with different level log.Info("hello wo

siddontang 31 Jun 15, 2022
Structured log interface

Structured log interface Package log provides the separation of the logging interface from its implementation and decouples the logger backend from yo

teris.io 25 Jul 29, 2022
lumberjack is a log rolling package for Go

lumberjack Lumberjack is a Go package for writing logs to rolling files. Package lumberjack provides a rolling logger. Note that this is v2.0 of lumbe

Nate Finch 3.5k Aug 4, 2022
CoLog is a prefix-based leveled execution log for Go

What's CoLog? CoLog is a prefix-based leveled execution log for Go. It's heavily inspired by Logrus and aims to offer similar features by parsing the

null 156 Aug 2, 2022
OpenTelemetry log collection library

opentelemetry-log-collection Status This project was originally developed by observIQ under the name Stanza. It has been contributed to the OpenTeleme

OpenTelemetry - CNCF 88 Jul 11, 2022
A simple web service for storing text log files

logpaste A minimalist web service for uploading and sharing log files. Run locally go run main.go Run in local Docker container The Docker container a

Michael Lynch 234 Aug 9, 2022
exo: a process manager & log viewer for dev

exo: a process manager & log viewer for dev exo- prefix – external; from outside. Features Procfile compatible process manager.

Deref 327 Aug 1, 2022
Write log entries, get X-Ray traces.

logtoxray Write to logs, get X-Ray traces. No distributed tracing instrumenation library required. ?? ?? ?? THIS PROJECT IS A WORK-IN-PROGRESS PROTOTY

JBD 27 Apr 24, 2022
Binalyze logger is an easily customizable wrapper for logrus with log rotation

logger logger is an easily customizable wrapper for logrus with log rotation Usage There is only one function to initialize logger. logger.Init() When

Binalyze 26 Nov 18, 2021
Log-structured virtual disk in Ceph

lsd_ceph Log-structured virtual disk in Ceph 1. Vision and Goals of the Project Implement the basic librbd API to work with the research block device

null 3 Dec 13, 2021
Multi-level logger based on go std log

mlog the mlog is multi-level logger based on go std log. It is: Simple Easy to use NOTHING ELSE package main import ( log "github.com/ccpaging/lo

null 0 May 18, 2022
Simple log parser written in Golang

Simple log parser written in Golang

Matteo Baiguini 0 Oct 31, 2021