Small utility to allow simpler, quicker testing of parsing files in crowdsec

Related tags

golang crowdsec
Overview

cs_parser_test

Small utility to allow simpler, quicker testing of parsing files in crowdsec

Usage

$ sudo cs_parser_test -t syslog /var/log/mail.log

NB No changes to the running instance are made.

Essentially you need to supply the same data you would enter into the acquis.yaml file. The above would be shown as

---
# postfix
filenames:
  - /var/log/mail.log
labels:
  type: syslog

One of the intents of wriing the app was to allow me to test different parsers without needing to contiually alter the running instance or the acquis.yaml file. So to test a parser for my postfix logs I can add the parser and then run the app with a different type specified.

By default this will now load the scenarios and show which ones would handle each line.

The output can get quite long, so careful use of the -n flag is advised!

$ sudo ./cs_parser_test -type postfix -n 1 /var/log/mail.log
Processing file /var/log/mail.log
Configuration from /etc/crowdsec/config.yaml

INFO[0000] Loading grok library /etc/crowdsec/patterns/ 
INFO[0000] Loading enrich plugins                       
INFO[0000] Loading parsers 11 stages                    
INFO[0000] Loaded 1 parser nodes                         file=/etc/crowdsec/parsers/s02-enrich/http-logs.yaml
INFO[0000] Loaded 1 parser nodes                         file=/etc/crowdsec/parsers/s01-parse/postscreen-logs.yaml
...
INFO[0000] Loaded 12 nodes, 3 stages                    
INFO[0000] Loading postoverflow Parsers                 
INFO[0000] Loaded 0 nodes, 0 stages                     
Scenario: crowdsecurity/http-backdoors-attempts
INFO[0000] Adding leaky bucket                           cfg= file=/etc/crowdsec/scenarios/http-backdoors-attempts.yaml name=crowdsecurity/http-backdoors-attempts
    loaded OK
/etc/crowdsec/scenarios/http-backdoors-attempts.yaml
  Scenario: crowdsecurity/http-xss-probbing
INFO[0000] Adding leaky bucket                           cfg= file=/etc/crowdsec/scenarios/http-xss-probing.yaml name=crowdsecurity/http-xss-probbing
    loaded OK
/etc/crowdsec/scenarios/http-xss-probing.yaml
  Scenario: crowdsecurity/iptables-scan-multi_ports
INFO[0000] Adding leaky bucket                           cfg= file=/etc/crowdsec/scenarios/iptables-scan-multi_ports.yaml name=crowdsecurity/iptables-scan-multi_ports
    loaded OK
...

Scanning file until 1 match is found...

Line 32:
  Processed? true
  Final Stage: s02-enrich
  Parsed Entries [evt.Parsed]:
    message             : Jul 12 06:26:10 xxxxxxx: warning: unknown[111.17.201.197]: SASL LOGIN authentication failed: UGFzc3dvcmQ6
    program             : postfix
    remote_addr         : 111.17.201.197
    message_failure     :  UGFzc3dvcmQ6
    remote_host         : unknown
Metadata [evt.Meta]:
    log_type_enh        : spam-attempt
    service             : postfix
    source_ip           : 111.17.201.197
    log_type            : postfix
    IsoCode             : CN
    ASNNumber           : 24444
    SourceRange         : 111.16.0.0/15
    source_hostname     : unknown
    IsInEU              : false
    ASNOrg              : Shandong Mobile Communication Company Limited

Processed by 1 scenario
    - zathras777/postfix-test

Scanned a total of 32 lines to find 1 matches

Command Options

$ ./cs_parser_test -h
Usage of ./cs_parser_test:
  -all
        show all line results (verbose)
  -c string
        configuration file to use (default "/etc/crowdsec/config.yaml")
  -n int
        how many lines to show (default: 0, unlimited)
  -parse
        parse ONLY. no scenarios loaded
  -type string
        type to assign (default "syslog")

Why?

The Crowdsec project has a great and very responsive development team but they are developing their product quickly and there have been a lot of large changes since I started using it. As I evolved my parsers I found it hard at times to figure out whether the change was correct or not. This small app is designed to allow me to develop the parsers and scenarios further while quickly test against a file.

Maybe it'll be of use to others?

Future

Likely the next change will be to add a way to use the app to create a sample log file to parse by saving all the lines that match into a new file. This should assist with debugging by providing a richer dataset.

Patches, corrections and improvements always welcome.

Owner
david reid
david reid
A super simple Lodash like utility library with essential functions that empowers the development in Go

A simple Utility library for Go Go does not provide many essential built in functions when it comes to the data structure such as slice and map. This

Rahul Baruri 81 Jul 17, 2021
Extremely flexible golang deep comparison, extends the go testing package, tests HTTP APIs and provides tests suite

go-testdeep Extremely flexible golang deep comparison, extends the go testing package. Latest news Synopsis Description Installation Functions Availab

Maxime Soulé 183 Jul 20, 2021
A Go (golang) library for parsing and verifying versions and version constraints.

go-version is a library for parsing versions and version constraints, and verifying versions against a set of constraints. go-version can sort a collection of versions properly, handles prerelease/beta versions, can increment versions, etc.

HashiCorp 941 Jul 19, 2021
基建KIT库

GKIT _____/\\\\\\\\\\\\__/\\\________/\\\__/\\\\\\\\\\\__/\\\\\\\\\\\\\\\_

null 66 Jul 20, 2021
bebop is a bebop parser written in Go, for generating Go code.

bebop is a bebop parser written in Go, for generating Go code. bebop can read .bop files and output .go files representing them: package main i

Patrick Stephen 24 Jul 14, 2021
a tool for creating exploited media files for discord

Discord-Exploits A program for creating exploited media files for discord written in Go. Usage discord-exploits is a command line utility, meaning you

schmenn 194 Jul 22, 2021
Molecule is a Go library for parsing protobufs in an efficient and zero-allocation manner

Molecule Molecule is a Go library for parsing protobufs in an efficient and zero-allocation manner. The API is loosely based on this excellent Go JSON

Richard Artoul 339 Jul 7, 2021
Robust & Easy to use struct mapper and utility methods for Go

go-model Robust & Easy to use model mapper and utility methods for Go struct. Typical methods increase productivity and make Go development more fun ?

Jeevanandam M. 310 Jul 14, 2021
gopkg is a universal utility collection for Go, it complements offerings such as Boost, Better std, Cloud tools.

gopkg is a universal utility collection for Go, it complements offerings such as Boost, Better std, Cloud tools. Table of Contents Introduction

Bytedance Inc. 169 Jul 24, 2021
⚖️ A tool for transpiling C to Go.

A tool for converting C to Go. The goals of this project are: To create a generic tool that can convert C to Go. To be cross platform (linux and mac)

Elliot Chance 1.7k Jul 12, 2021
A tool to check problems about meta files of Unity

A tool to check problems about meta files of Unity on Git repositories, and also the tool can do limited autofix for meta files of auto-generated files.

DeNA 38 Jul 14, 2021
A tool and library for using structural regular expressions.

Structural Regular Expressions sregx is a package and tool for using structural regular expressions as described by Rob Pike (link).

Zachary Yedidia 22 Jun 18, 2021
sigbypass4xx is a utility to automate well-know techniques used to bypass access control restrictions.

sigbypass4xx sigbypass4xx is a utility to automate well-know techniques used to bypass access control restrictions. Resources Usage Installation From

Signed Security 3 Jul 12, 2021
A collection of small Go utilities to make life easier.

The simplego package provides a collection of Go utilities for common tasks.

John Wang 16 Jul 13, 2021