Logstash like, written in golang

Overview

gogstash

Logstash like, written in golang

Build Status

curl 'https://github.com/tsaikd/gogstash/releases/download/0.1.8/gogstash-Linux-x86_64' -SLo gogstash && chmod +x gogstash
  • Configure for ubuntu-sys.json (example)
{
	"input": [
		{
			"type": "exec",
			"command": "sh",
			"interval": 60,
			"message_prefix": "%{@timestamp} [df] ",
			"args": ["-c", "df -B 1 / | sed 1d"]
		},
		{
			"type": "exec",
			"command": "sh",
			"interval": 60,
			"message_prefix": "%{@timestamp} [diskstat] ",
			"args": ["-c", "grep '0 [sv]da ' /proc/diskstats"]
		},
		{
			"type": "exec",
			"command": "sh",
			"interval": 60,
			"message_prefix": "%{@timestamp} [loadavg] ",
			"args": ["-c", "cat /proc/loadavg"]
		},
		{
			"type": "exec",
			"command": "sh",
			"interval": 60,
			"message_prefix": "%{@timestamp} [netdev] ",
			"args": ["-c", "grep '\\beth0:' /proc/net/dev"]
		},
		{
			"type": "exec",
			"command": "sh",
			"interval": 60,
			"message_prefix": "%{@timestamp} [meminfo]\n",
			"args": ["-c", "cat /proc/meminfo"]
		}
	],
	"output": [
		{
			"type": "report"
		},
		{
			"type": "redis",
			"key": "gogstash-ubuntu-sys-%{host}",
			"host": ["127.0.0.1:6379"]
		}
	]
}
  • Configure for dockerstats.json (example)
{
	"input": [
		{
			"type": "dockerstats"
		}
	],
	"output": [
		{
			"type": "report"
		},
		{
			"type": "redis",
			"key": "gogstash-docker-%{host}",
			"host": ["127.0.0.1:6379"]
		}
	]
}
  • Config format with YAML for dockerstats.json (example)
input:
  - type: dockerstats
output:
  - type: report
  - type: redis
    key: "gogstash-docker-%{host}"
    host:
      - "127.0.0.1:6379"
  • Configure for nginx.yml with gonx filter (example)
chsize: 1000
worker: 2

input:
  - type: redis
    host: redis.server:6379
    key:  filebeat-nginx
    connections: 1

filter:
  - type: gonx
    format: '$clientip - $auth [$time_local] "$full_request" $response $bytes "$referer" "$agent"'
    source: message
  - type: gonx
    format: '$verb $request HTTP/$httpversion'
    source: full_request
  - type: date
    format: ["02/Jan/2006:15:04:05 -0700"]
    source: time_local
  - type: remove_field
    fields: ["full_request", "time_local"]
  - type: add_field
    key: host
    value: "%{beat.hostname}"
  - type: geoip2
    db_path: "GeoLite2-City.mmdb"
    ip_field: clientip
    key: req_geo
  - type: typeconv
    conv_type: int64
    fields: ["bytes", "response"]

output:
  - type: elastic
    url: ["http://elastic.server:9200"]
    index: "log-nginx-%{[email protected]}"
    document_type: "%{type}"
  • Configure for beats.yml with grok filter (example)
chsize: 1000
worker: 2
event:
  sort_map_keys: false
  remove_field: ['@metadata']

input:
  - type: beats
    port: 5044
    reuseport: true
    host: 0.0.0.0
    ssl:  false

filter:
  - type: grok
    match: ["%{COMMONAPACHELOG}"]
    source: "message"
    patterns_path: "/etc/gogstash/grok-patterns"
  - type: date
    format: ["02/Jan/2006:15:04:05 -0700"]
    source: time_local
  - type: remove_field
    fields: ["full_request", "time_local"]
  - type: add_field
    key: host
    value: "%{beat.hostname}"
  - type: geoip2
    db_path: "GeoLite2-City.mmdb"
    ip_field: clientip
    key: req_geo
  - type: typeconv
    conv_type: int64
    fields: ["bytes", "response"]

output:
  - type: elastic
    url: ["http://elastic1:9200","http://elastic2:9200","http://elastic3:9200"]
    index: "filebeat-6.4.2-%{[email protected]}"
    document_type: "doc"
  • Run gogstash for nginx example (command line)
GOMAXPROCS=4 ./gogstash --CONFIG nginx.json
  • Run gogstash for dockerstats example (docker image)
docker run -it --rm \
	--name gogstash \
	--hostname gogstash \
	-e GOMAXPROCS=4 \
	-v "/var/run/docker.sock:/var/run/docker.sock" \
	-v "${PWD}/dockerstats.json:/gogstash/config.json:ro" \
	tsaikd/gogstash:0.1.8

Supported inputs

See input modules for more information

Supported filters

All filters support the following commmon functionality/configuration:

filter:
  - type: "whatever"

    # list of tags to add
    add_tag: ["addtag1", "addtag2"]

    # list of tags to remove
    remove_tag: ["removetag1", "removetag2"]

    # list of fields (key/value) to add
    add_field:
      - key: "field1"
        value: "value1"
      - key: "field2"
        value: "value2"
    # list of fields to remove
    remove_field: ["removefield1", "removefield2"]

See filter modules for more information

Supported outputs

See output modules for more information

Comments
  • Would gogstash silent fail in elastic output?

    Would gogstash silent fail in elastic output?

    I'm using gogstash 0.1.14 as an indexer from redis to elasticsearch

    There are some errors in my error logs:

    2018/06/19 08:25:47 output.go:52 [error] output failed: elastic: Error 429 (Too Many Requests): rejected execution of [email protected] on EsThreadPoolExecutor[name = data00/bulk, queue capacity = 200, [email protected]ed0e49[Running, pool size = 8, active threads = 8, queued tasks = 200, completed tasks = 27481615]] [type=es_rejected_execution_exception]
    

    I'm trying to find out the error came from: https://github.com/tsaikd/gogstash/blob/binary/config/output.go#L44

    However, I haven't see the event back to the event channel after it failed. Am I missing some thing or it will just lost event data after it failed?

    Thanks

    opened by tengattack 20
  • Add config workers

    Add config workers

    Add the workers config to increase CPU usage, since there only one concurrent thread for filter & output.

    In real, it can boost up 2x ~ 3x speed for index rate in elasticsearch.

    opened by tengattack 16
  • question: will Logstash like filters be a feature?

    question: will Logstash like filters be a feature?

    Logstash offers many filters -- like aggregate, grok, geoip, and so on. Are there plans to offer filters for gogstash ? Or is exec meant to replace filters ? It would be nice to see a Go replacement for Logstash and it's many features. Thanks.

    opened by cleesmith 10
  • Cannot be parsed using grok

    Cannot be parsed using grok

    The log cannot be parsed when I use the grok filter, but I can do it in the Grok Debugger, help cat config.json

    {
        "input": [
            {
                "type":"file",
                "path":"/var/log/nginx/access.log"
            }
        ],
        "debugch":true,
        "filter": [
            {
                "type":"grok",
                "source":"message",
                "match":["%{NGINXTEST}"],
                "patterns_path":"grok-patterns"
            }
        ],
        "output": [
            {
                "type": "stdout"
            }
        ]
    }
    

    cat grok-patterns

    NGINXTEST %{HOST:upstream_addr}:%{HOST:upstream_port} %{IPORHOST:http_host} %{NUMBER:request_time} %{NUMBER:response_time} %{IPORHOST:remote_addr} - %{NGUSER:remote_user} \[%{HTTPDATE:timestamp}\] "%{WORD:method} %{URIPATHPARAM:request} HTTP/%{NUMBER:httpversion}" %{NUMBER:status} (?:%{NUMBER:bytes}|-) (?:"(?:%{URI:referrer}|-)"|%{QS:referrer}) (?:%{QS:agent}) %{QS:xforwardedfor}
    

    Original log

    127.0.0.1:12345 test.monitor.com 0.001 0.001 127.0.0.1 - - [28/Apr/2019:18:26:10 +0800] "GET /view/test.jpeg HTTP/1.1" 200 9444 "http://test.monitor.com/view/" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_13_2) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/74.0.3729.108 Safari/537.36" "-" "-"
    

    Log Format

    '$upstream_addr $http_host $request_time $upstream_response_time $remote_addr - $remote_user [$time_local] "$request" '
                          '$status $body_bytes_sent "$http_referer" '
                          '"$http_user_agent" "$http_x_forwarded_for" "$request_body"';
    

    And Grok Gebugger image

    opened by wang1219 7
  • Implement gelf output

    Implement gelf output

    That implementation allow direct output to Graylog infrastructure.

    Multiple implementation of gelf_writer can be found over github. Mostly the one part of that PR is a classical version with a few adaptation to allow configuration of compression / chunkSize, ...

    opened by mderasse 6
  • Allow overriding

    Allow overriding "message" field, like all other fields.

    Allow and use overridden "message" field, like any other field. Otherwise things like grok filtering an overriding the "message" field do not work as the original message field keeps being used

    opened by horacimacias 6
  • output module

    output module "http" failed

    My output: output:

    • type: http urls: ["http://xxxxxxx/web_api/log_file_report"]

      http_status_codes: [200, 201, 202] http_error_codes: [501, 405, 404, 208, 505] retry_interval: 60 max_queue_size: 1 ignore_ssl: true

    Why Log output: 2022/08/22 16:54:29 output.go:99 [error] output module "http" failed: "http://xxxxxxx/web_api/log_file_report" retryable error 200 2022/08/22 16:55:28 simplequeue.go:129 [error] queue: httpsendone "http://xxxxxxx/web_api/log_file_report" retryable error 200

    opened by YangTao0 4
  • Added ip2location filter

    Added ip2location filter

    ip2location is another geo database, like geoip2 - but another provider. This filter is built using their own official library and is based on the code from the geoip2 filter.

    I also added some code for FileWatcher - but should this be placed somewhere else? My intention is to reuse this code other places (geoip2, ip2proxy) when the database is updated.

    To be able to test this filter yourself you need to create a free account on https://lite.ip2location.com

    opened by helgeolav 4
  • [WIP]  Output clickhouse

    [WIP] Output clickhouse

    Hi, this is still work in progress (features, docs and tests still missing) but basic functionality is there. Do you think you can have a look at it and let me know if I'm totally misguided or if you see this as a possible merge in the future?

    Thanks in advance

    opened by pachico 4
  • use a corporate-friendly open source license

    use a corporate-friendly open source license

    I'm considering using gogstash on a commercial product but the current LGPL license prevents me from doing this. Could you please consider switching to a more corporate-friendly open source license instead? e.g. MIT, Apache 2.0 and BSD tend to be fine

    opened by horacimacias 4
  • the inputsocket @timestamp problem

    the inputsocket @timestamp problem

    When input socket module is being used and the payload is a valid JSON, the @timestamp field is not being properly initialized to time.Now(). This results in the output LogEvent having a non-initialized Timestamp field.

    Currently the result of the output has a @timestamp being this:

    {
    	"@timestamp": "0001-01-01T00:00:00Z",
    	...
    }
    

    Using version 0.1.14

    opened by aantono 4
  • Improve error message for geoip filter

    Improve error message for geoip filter

    I see in logs string: gogstash_filter_geoip2_error, this is related to private subnets. Is it possible to improve output with error message for geoip filter?

    gogstash_filter_geoip2_private_skipped - for private subnets
    gogstash_filter_geoip2_error - use for all another errors in geoip filter.
    

    I would like to have more control over conditions during next steps.

    opened by KVInventoR 1
  • How does output to loki work?

    How does output to loki work?

    Hi all,

    I need help with configuration for this wonderful project. I have simple config:

    chsize: 1000
    
    input:
      - type: file
        path: /logging/test_input.log
        start_position: end
        sincedb_write_interval: 5
    
    filter:
      - type: grok
        match: ["%{IPTABLES_SRC}"]
        source: "message"
        patterns_path: "/iptables.grok"
    
      - type: geoip2
        db_path: "/GeoLite2-City.mmdb"
        ip_field: "src_ip"
        cache_size: 100000
        key: geoip
        flat_format: true
    
      - type: add_field
        key: 'geo2ip_city'
        value: '%{geoip.country_code}'
     
    output:
      - type: stdout
        codec: json
    
      - type: loki
        urls:
          - "http://mloki.fr.com.ua:3100/loki/api/v1/push"
          
    
     echo 'May 11 10:02:05 zabbix kernel: [1609112.875635] FW_F_IN_DROP: IN=ens18 OUT= MAC=3a:e9:5f:c7:41:78:d0:07:ca:8c:10:01:08:00 SRC=104.156.15.12 DST=19.0.20.1 LEN=40 TOS=0x00 PREC=0x00 TTL=243 ID=8530 PROTO=TCP SPT=58399 DPT=3080 WINDOW=1024 RES=0x00 SYN URGP=0' >> /logging/test_input.log
    

    Error:

    2022/05/11 20:07:03 outputloki.go:89 [warning] key: geoip error:Unable to Cast to string
    {"host":"gogstash","path":"/logging/test_input.log","@timestamp":"2022-05-11T20:07:03.018256841Z","message":"May 11 10:02:05 zabbix kernel: [1609112.875635] FW_F_IN_DROP: IN=ens18 OUT= MAC=3a:e9:5f:c7:41:78:d0:07:ca:8c:10:01:08:00 SRC=104.156.15.12 DST=19.0.20.1LEN=40 TOS=0x00 PREC=0x00 TTL=243 ID=8530 PROTO=TCP SPT=58399 DPT=3080 WINDOW=1024 RES=0x00 SYN URGP=0","offset":0,"src_ip":"104.156.15.12","geoip":{"longitude":-97.822,"timezone":"America/Chicago","continent_code":"NA","country_code":"US","country_name":"United States","ip":"104.156.15.12","latitude":37.751,"location":[-97.822,37.751]},"geo2ip_city":"US"}
    

    also, I tried to use output stdout without codec json, and result was without geoip data, looks like geoip can be injected only in json mode.

    Is there anything which can help me to resolve issue? I would like to send logs to loki with geoip data to build dashboard with worldmap and geoip data.

    opened by KVInventoR 8
  • Added discard filter

    Added discard filter

    In this PR I have added a discard filter. This filter allows me to

    1. Discard an event from further processing when I know that I will not need it. Today this has to be done by using a cond output, but you still need to send the event through all stages of processing before it is discarded.
    2. Discard the event in case we have back pressure issues. This can be handy if the output supports back pressure, the input does not and discarding the event is your best option in this case.

    I also rewrote the filter handling routine to allow for an event to start at a specific filter, instead of at the beginning of the list of filters. I think there are many good ways to use this. I am currently working on a filter where my goal is to identify and remove all events of a kind expect the last one. To do this I need to discard all but the last event that I need to inject for further processing from the next filter in the pipeline.

    Other good examples for the discard code can be logstash aggregate or logstash throttling.

    opened by helgeolav 7
  • Fields with dots in the name cannot be referenced

    Fields with dots in the name cannot be referenced

    I'm looking to process JSON logs with fields like this:

    {  "http.method": "GET" }
    

    ...and many more similar fields with . in the name.

    I can use the json filter to transform this into fields, but none of the other filters are able to manipulate these fields. For example with these filters:

    filter:
      - type: json
      - type: remove_field
        fields: ["http.method"]
    

    ...running the above JSON through yields:

    {
    	"@timestamp": "2020-08-21T02:26:06.158181Z",
    	"host": "MyHost",
    	"http.method": "GET",
    	"message": "{\"http.method\":\"GET\"}"
    }
    

    There's no way to remove the field! Other filters have a similar issue.

    The problem is that getPathValue always interprets its input as a path expression and not necessarily a literal key into the map. Reference: https://github.com/tsaikd/gogstash/blob/d4613914309c6655ff2234eb15a4fb013c0d2ee1/config/logevent/pathvalue.go#L87

    Logstash has field reference syntax to handle these different cases. I can understand that gogstash may not want such complexity, but it does feel like there should be a solution to my problem. Otherwise I think gogstash will be unusable for me.

    opened by edsrzf 3
  • Back pressure handling document

    Back pressure handling document

    Hi

    I would like to know about if output is not available (ex : elasticsearch full disk), what will happen? What is the buffering mechanism and is there any parameters (like max_disk_buffer_size, max_memory_buffer_size)?

    Thanks

    opened by JoHuang 3
Releases(0.1.21)
Owner
Tsai KD
Tsai KD
The open and composable observability and data visualization platform. Visualize metrics, logs, and traces from multiple sources like Prometheus, Loki, Elasticsearch, InfluxDB, Postgres and many more.

The open-source platform for monitoring and observability. Grafana allows you to query, visualize, alert on and understand your metrics no matter wher

Grafana Labs 51.3k Sep 30, 2022
Like Prometheus, but for logs.

Loki: like Prometheus, but for logs. Loki is a horizontally-scalable, highly-available, multi-tenant log aggregation system inspired by Prometheus. It

Grafana Labs 17.1k Sep 28, 2022
LogVoyage - logging SaaS written in GoLang

No longer maintained, sorry. Completely rewritten v2 is going to be released soon. Please follow http://github.com/logvoyage LogVoyage - fast and simp

null 93 Sep 26, 2022
A system and resource monitoring tool written in Golang!

Grofer A clean and modern system and resource monitor written purely in golang using termui and gopsutil! Currently compatible with Linux only. Curren

PES Open Source Community 222 Sep 27, 2022
Simple log parser written in Golang

Simple log parser written in Golang

Matteo Baiguini 0 Oct 31, 2021
The Simplest and worst logging library ever written

gologger A Simple Easy to use go logger library. Displays Colored log into console in any unix or windows platform. You can even store your logs in fi

Sadlil Rhythom 41 Sep 26, 2022
A Statsd implementation written in GO lang

statsgod Statsgod is a metric aggregation service inspired by the statsd project. Written in Golang, it increases performance and can be deployed with

Acquia 121 Oct 1, 2022
GoVector is a vector clock logging library written in Go.

GoVector is a vector clock logging library written in Go. The vector clock algorithm is used to order events in distributed systems in the absence of a centralized clock. GoVector implements the vector clock algorithm and provides feature-rich logging and encoding infrastructure.

Distributed clocks 164 Aug 13, 2022
ChangeTower is intended to help you watch changes in webpages and get notified of any changes written in Go

ChangeTower is intended to help you watch changes in webpages and get notified of any changes written in Go

The Cats 32 Jul 6, 2022
Port information web scraper written in Go.

Whatport is an open source tool that scrapes port information from SpeedGuide's Port Database Usage whatport [port(s)] (Seperate ports with a space)

Abdelouahab 7 Aug 18, 2022
An open source Pusher server implementation compatible with Pusher client libraries written in GO

Try browsing the code on Sourcegraph! IPÊ An open source Pusher server implementation compatible with Pusher client libraries written in Go. Why I wro

Claudemiro 357 Sep 29, 2022
A reusable logger module for basic logging, written in Go

logger A reusable logger module for basic logging, written in Go. Usage Client p

Praveen Ravichandran 1 Jan 8, 2022
Logger - Simple logger without written with std pkg

Go-Logger Simple usage is: package main

MaskedTrench 2 Jan 2, 2022
Stream logs through websockets, written in Go

Stream logs through websockets, written in Go

Praveen Ravichandran 1 Jan 8, 2022
A simple digital clock written in go to show time in hh : mm : ss format in console

Go console clock a simple digital clock written in go to show time in "hh : mm :

Mojtaba Khodami 0 Feb 3, 2022
Logger - Some implementations for Logs written in Go

Logger will be a tool for sending logs to different places in your software. Rig

Carlos Gaona 1 Feb 16, 2022
Excel binding to struct written in Go.(Only supports Go1.18+)

exl Excel binding to struct written in Go.(Only supports Go1.18+) usage Read Excel package main import ( "fmt" "github.com/go-the-way/exl" ) type

go-the-way 12 Sep 27, 2022
Simple and blazing fast lockfree logging library for golang

glg is simple golang logging library Requirement Go 1.11 Installation go get github.com/kpango/glg Example package main import ( "net/http" "time"

Yusuke Kato 171 Sep 26, 2022
Logging library for Golang

GLO Logging library for Golang Inspired by Monolog for PHP, severity levels are identical Install go get github.com/lajosbencz/glo Severity levels Deb

Lajos Bencz 15 Sep 26, 2022