Small utility to allow simpler, quicker testing of parsing files in crowdsec

Overview

cs_parser_test

Small utility to allow simpler, quicker testing of parsing files in crowdsec

Usage

$ sudo cs_parser_test -t syslog /var/log/mail.log

NB No changes to the running instance are made.

Essentially you need to supply the same data you would enter into the acquis.yaml file. The above would be shown as

---
# postfix
filenames:
  - /var/log/mail.log
labels:
  type: syslog

One of the intents of wriing the app was to allow me to test different parsers without needing to contiually alter the running instance or the acquis.yaml file. So to test a parser for my postfix logs I can add the parser and then run the app with a different type specified.

By default this will now load the scenarios and show which ones would handle each line.

The output can get quite long, so careful use of the -n flag is advised!

$ sudo ./cs_parser_test -type postfix -n 1 /var/log/mail.log
Processing file /var/log/mail.log
Configuration from /etc/crowdsec/config.yaml

INFO[0000] Loading grok library /etc/crowdsec/patterns/ 
INFO[0000] Loading enrich plugins                       
INFO[0000] Loading parsers 11 stages                    
INFO[0000] Loaded 1 parser nodes                         file=/etc/crowdsec/parsers/s02-enrich/http-logs.yaml
INFO[0000] Loaded 1 parser nodes                         file=/etc/crowdsec/parsers/s01-parse/postscreen-logs.yaml
...
INFO[0000] Loaded 12 nodes, 3 stages                    
INFO[0000] Loading postoverflow Parsers                 
INFO[0000] Loaded 0 nodes, 0 stages                     
Scenario: crowdsecurity/http-backdoors-attempts
INFO[0000] Adding leaky bucket                           cfg= file=/etc/crowdsec/scenarios/http-backdoors-attempts.yaml name=crowdsecurity/http-backdoors-attempts
    loaded OK
/etc/crowdsec/scenarios/http-backdoors-attempts.yaml
  Scenario: crowdsecurity/http-xss-probbing
INFO[0000] Adding leaky bucket                           cfg= file=/etc/crowdsec/scenarios/http-xss-probing.yaml name=crowdsecurity/http-xss-probbing
    loaded OK
/etc/crowdsec/scenarios/http-xss-probing.yaml
  Scenario: crowdsecurity/iptables-scan-multi_ports
INFO[0000] Adding leaky bucket                           cfg= file=/etc/crowdsec/scenarios/iptables-scan-multi_ports.yaml name=crowdsecurity/iptables-scan-multi_ports
    loaded OK
...

Scanning file until 1 match is found...

Line 32:
  Processed? true
  Final Stage: s02-enrich
  Parsed Entries [evt.Parsed]:
    message             : Jul 12 06:26:10 xxxxxxx: warning: unknown[111.17.201.197]: SASL LOGIN authentication failed: UGFzc3dvcmQ6
    program             : postfix
    remote_addr         : 111.17.201.197
    message_failure     :  UGFzc3dvcmQ6
    remote_host         : unknown
Metadata [evt.Meta]:
    log_type_enh        : spam-attempt
    service             : postfix
    source_ip           : 111.17.201.197
    log_type            : postfix
    IsoCode             : CN
    ASNNumber           : 24444
    SourceRange         : 111.16.0.0/15
    source_hostname     : unknown
    IsInEU              : false
    ASNOrg              : Shandong Mobile Communication Company Limited

Processed by 1 scenario
    - zathras777/postfix-test

Scanned a total of 32 lines to find 1 matches

Command Options

$ ./cs_parser_test -h
Usage of ./cs_parser_test:
  -all
        show all line results (verbose)
  -c string
        configuration file to use (default "/etc/crowdsec/config.yaml")
  -n int
        how many lines to show (default: 0, unlimited)
  -parse
        parse ONLY. no scenarios loaded
  -type string
        type to assign (default "syslog")

Why?

The Crowdsec project has a great and very responsive development team but they are developing their product quickly and there have been a lot of large changes since I started using it. As I evolved my parsers I found it hard at times to figure out whether the change was correct or not. This small app is designed to allow me to develop the parsers and scenarios further while quickly test against a file.

Maybe it'll be of use to others?

Future

Likely the next change will be to add a way to use the app to create a sample log file to parse by saving all the lines that match into a new file. This should assist with debugging by providing a richer dataset.

Patches, corrections and improvements always welcome.

You might also like...
 A library for parsing ANSI encoded strings
A library for parsing ANSI encoded strings

Go ANSI Parser converts strings with ANSI escape codes into a slice of structs that represent styled text

Gene parsing package for Axie Infinity

agp Package agp is a gene parsing package for Axie Infinity. The name agp stands for "Axie Gene Parser" which decodes the hex representation of an Axi

Simple library to handle ANSI functions and parsing of color formatting strings

Emerald A basic color library for use in my Go projects, built on top of mgutz/ansi. Package ansi is a small, fast library to create ANSI colored stri

This library provides an ASTERIX Frame(binary data) decoding/parsing(json,xml) capabilities for Go.

GoAsterix This library provides an ASTERIX Frame(binary data) decoding/parsing(json,xml) capabilities for Go. ASTERIX ASTERIX (All Purpose Structured

Go-path - A helper package that provides utilities for parsing and using ipfs paths

go-path is a helper package that provides utilities for parsing and using ipfs paths

A collection of small Go utilities to make life easier.

The simplego package provides a collection of Go utilities for common tasks.

MNA - stands for mobile number assignment - a small zero external dependency golang library that is used to identify mobile number assignment in tanzania

MNA - stands for mobile number assignment - a small zero external dependency golang library that is used to identify mobile number assignment in tanzania

go-linereader: A small library for streaming lines from an io.Reader.

go-linereader: A small library for streaming lines from an io.Reader.

Go Small Library

Go Small Library (gosl) Copyright Gon Y. Yi 2021 Goal General No import of any library whatsoever including standard libr

Owner
david reid
david reid
Split multiple Kubernetes files into smaller files with ease. Split multi-YAML files into individual files.

Split multiple Kubernetes files into smaller files with ease. Split multi-YAML files into individual files.

Patrick D'appollonio 204 Dec 29, 2022
Split multiple Kubernetes files into smaller files with ease. Split multi-YAML files into individual files.

kubectl-slice: split Kubernetes YAMLs into files kubectl-slice is a neat tool that allows you to split a single multi-YAML Kubernetes manifest into mu

Patrick D'appollonio 205 Jan 3, 2023
A small & fast dependency-free library for parsing micro expressions.

MicroExpr A small & fast dependency-free library for parsing micro expressions. This library was originally built for use in templating languages (e.g

Daniel G. Taylor 10 Nov 25, 2022
A small utility to extract your Telegram contacts as VCF file.

Telegram to VCF A small utility to extract your Telegram contacts as VCF file. Usage At first build this application or use the precompiled binaries o

Hirbod Behnam 4 Nov 16, 2022
This is a small utility that finds unused exported Go symbols (functions, methods ...) in Go

This is a small utility that finds unused exported Go symbols (functions, methods ...) in Go. For all other similar use cases

Bjørn Erik Pedersen 22 Nov 8, 2022
Small utility to cleanup entries with unexpected/erroneous ttl in redis

Small utility to cleanup entries with unexpected/erroneous ttl in redis

Selva 0 Apr 28, 2022
Small tool for splitting files found in a path into multiple groups

Small tool for splitting files found in a path into multiple groups. Usefull for parallelisation of whatever can be paralleled with multiple files.

Antonio Martinović 0 Jan 30, 2022
Golang source code parsing, usage like reflect package

gotype Golang source code parsing, usage like reflect package English 简体中文 Usage API Documentation Examples License Pouch is licensed under the MIT Li

null 52 Dec 9, 2022
Molecule is a Go library for parsing protobufs in an efficient and zero-allocation manner

Molecule Molecule is a Go library for parsing protobufs in an efficient and zero-allocation manner. The API is loosely based on this excellent Go JSON

Richard Artoul 376 Jan 5, 2023
A Go (golang) library for parsing and verifying versions and version constraints.

go-version is a library for parsing versions and version constraints, and verifying versions against a set of constraints. go-version can sort a collection of versions properly, handles prerelease/beta versions, can increment versions, etc.

HashiCorp 1.3k Jan 9, 2023