EarlyBird is a sensitive data detection tool capable of scanning source code repositories for clear text password violations, PII, outdated cryptography methods, key files and more.

Related tags

earlybird
Overview

Logo

EarlyBird is a sensitive data detection tool capable of scanning source code repositories for clear text password violations, PII, outdated cryptography methods, key files and more. It can be used to scan remote git repositories, local files or directories or as a pre-commit step.

Installation

Linux & Mac

Running the build.sh script will produce a binary for each OS, while the install.sh script will install Earlybird on your system. This will create a .go-earlybird directory in your home directory with all the configuration files. Finally installing go-earlybird as an executable in /usr/local/bin/.

./build.sh && ./install.sh

Windows

Running build.bat will produce your binaries while the install.bat script will create a 'go-earlybird' directory in C:\Users\[my user]\App Data\, and copy the required configurations there. This script will also install go-earlybird.exe as an executable in the App Data directory (which should be in your path).

build.bat && install.bat

Usage

To launch a basic EarlyBird scan against a directory:

$ go-earlybird --path=/path/to/directory
$ go-earlybird.exe --path=C:\path\to\directory

or to scan a remote git repo:

$ go-earlybird --git=https://github.com/americanexpress/earlybird

Click here for Detailed Usage instructions.

Documentation

Why Are We Doing This?

The MITRE Corporation provides a catalog of Common Weakness Enumerations (CWE), documenting issues that should be avoided. Some of the relevant CWEs that are handled by the use of EarlyBird include:


Contributing

We welcome your interest in the American Express Open Source Community on Github. Any Contributor to any Open Source Project managed by the American Express Open Source Community must accept and sign an Agreement indicating agreement to the terms below. Except for the rights granted in this Agreement to American Express and to recipients of software distributed by American Express, You reserve all right, title, and interest, if any, in and to your contributions. Please fill out the Agreement.

License

Any contributions made under this project will be governed by the Apache License 2.0.

Code of Conduct

This project adheres to the American Express Community Guidelines. By participating, you are expected to honor these guidelines.

Issues
  • Build fails on macOS

    Build fails on macOS

    When running build.sh, the build fails with the following log:

    Running Unit Tests
    go: downloading golang.org/x/net v0.0.0-20200324143707-d3edc9973b7e
    go: downloading github.com/gorilla/mux v1.7.4
    go: downloading github.com/gocarina/gocsv v0.0.0-20200330101823-46266ca37bd3
    go: downloading gopkg.in/src-d/go-git.v4 v4.13.1
    go: downloading github.com/dghubble/sling v1.3.0
    go: downloading github.com/howeyc/gopass v0.0.0-20190910152052-7cb4b85ec19c
    go: downloading github.com/google/go-github v17.0.0+incompatible
    go: downloading golang.org/x/text v0.3.2
    go: downloading golang.org/x/crypto v0.0.0-20190701094942-4def268fd1a4
    go: downloading github.com/google/go-querystring v1.0.0
    go: downloading github.com/sergi/go-diff v1.0.0
    go: downloading gopkg.in/src-d/go-billy.v4 v4.3.2
    go: downloading github.com/kevinburke/ssh_config v0.0.0-20190725054713-01f96b0aa0cd
    go: downloading github.com/xanzy/ssh-agent v0.2.1
    go: downloading github.com/mitchellh/go-homedir v1.1.0
    go: downloading github.com/emirpasic/gods v1.12.0
    go: downloading golang.org/x/sys v0.0.0-20200323222414-85ca7c5b95cd
    go: downloading github.com/jbenet/go-context v0.0.0-20150711004518-d14ea06fba99
    go: downloading github.com/src-d/gcfg v1.4.0
    go: downloading gopkg.in/warnings.v0 v0.1.2
    # github.com/americanexpress/earlybird/pkg/api
    pkg/api/api.go:72:19: conversion from Duration (int64) to string yields a string of one rune, not a string of digits (did you mean fmt.Sprint(x)?)
    pkg/api/api.go:141:19: conversion from Duration (int64) to string yields a string of one rune, not a string of digits (did you mean fmt.Sprint(x)?)
    # github.com/americanexpress/earlybird/pkg/core
    pkg/core/core.go:261:19: conversion from Duration (int64) to string yields a string of one rune, not a string of digits (did you mean fmt.Sprint(x)?)
    # github.com/americanexpress/earlybird/pkg/writers
    pkg/writers/jsonout_test.go:55:17: conversion from Duration (int64) to string yields a string of one rune, not a string of digits (did you mean fmt.Sprint(x)?)
    Unit Tests FAILED!
    FAIL	github.com/americanexpress/earlybird/pkg/api [build failed]
    ok  	github.com/americanexpress/earlybird/pkg/config	0.325s
    FAIL	github.com/americanexpress/earlybird/pkg/core [build failed]
    Failed to open ignore file open .ge_ignore: no such file or directory
    Failed to open ignore file open /Users/phil/.ge_ignore: no such file or directory
    Failed to open ignore file open .ge_ignore: no such file or directory
    --- FAIL: Test_isIgnoredFile (0.00s)
        --- FAIL: Test_isIgnoredFile/Check_if_file_is_ignored (0.00s)
            fileUtil_test.go:148: isIgnoredFile() = false, want true
    FAIL
    FAIL	github.com/americanexpress/earlybird/pkg/file	0.222s
    ok  	github.com/americanexpress/earlybird/pkg/git	0.487s
    ok  	github.com/americanexpress/earlybird/pkg/postprocess	0.209s
    ok  	github.com/americanexpress/earlybird/pkg/scan	0.254s
    ok  	github.com/americanexpress/earlybird/pkg/update	0.468s
    ok  	github.com/americanexpress/earlybird/pkg/utils	0.252s
    ok  	github.com/americanexpress/earlybird/pkg/wildcard	0.184s
    FAIL	github.com/americanexpress/earlybird/pkg/writers [build failed]
    FAIL
    

    This occurs on macOS version 10.15.5, and Go version go1.15.2 darwin/amd64.

    opened by Phuurl 6
  • Cloning of git repository leaves files in temporary directory

    Cloning of git repository leaves files in temporary directory

    After data is being downloaded using -git flag to temporary directory it is not being removed after data check. I think that the expected behavior should be that the data is removed from the temporary location after checking presence of sensitive data.

    The problem is that if I am running check against many code repositories temp directory grows significantly and I need to have additional monitoring activity to erase temporary data.

    Version 1.24.6

    enhancement question 
    opened by spaluchowski 4
  • meta: Comparison to gitleaks?

    meta: Comparison to gitleaks?

    👋 Cool project! I was curious how earlybird compares to other projects like gitleaks? I see earlybird can scan a bit more types of targets, but are the patterns both recognize the same? I've used gitleaks for a while and am curious to adopt both tools.

    Project: https://github.com/zricethezav/gitleaks

    opened by adamdecaf 3
  • Configuration in default path is mandatory to launch the tool

    Configuration in default path is mandatory to launch the tool

    Even if configuration parameter is set in command line the default configuration path is still trying to load. And if this path is not present earlybird fails to launch with error: "Failed to load Earlybird configopen /var/lib/jenkins/.go-earlybird/earlybird.json: no such file or directory"

    Expected: If configuration path is provided in command line, the default configuration path should not be checked.

    Version: 1.24.6

    bug 
    opened by spaluchowski 3
  • Adds automated release pipeline via GitHub Actions

    Adds automated release pipeline via GitHub Actions

    Automated way of compiling and publishing new releases with GitHub Actions with every push to main.

    Closes #11

    opened by Phuurl 2
  • Corrects script exec error

    Corrects script exec error

    Moves #! to fix the exec issue as mentioned in #1.

    Closes #1

    opened by Phuurl 2
  • Fix readme typo

    Fix readme typo

    null

    opened by erjanmx 1
  • Lack of GitHub releases / precompiled binaries

    Lack of GitHub releases / precompiled binaries

    GitHub releases would allow users to download prebuilt versions of the application without having to compile it. This is especially useful for automated CI pipelines, where having to setup a Go environment and compile EarlyBird can add unnecessary overhead.

    Please publish compiled binaries of EarlyBird via GitHub releases to allow them to be downloaded as part of CI pipelines. This could be implemented in a CI/CD pipeline of your choice - I'll open a PR to do it in GitHub Actions since that's probably the easiest and doesn't require any external services.

    opened by Phuurl 1
  • Wrong verbose information about scanned files

    Wrong verbose information about scanned files

    When I scan folder with 4 files I see in log:

    Reading file Reading file /mnt/c/git-repo/earlybird/verify/.ge_ignore Reading file /mnt/c/git-repo/earlybird/verify/checkfile.properties Reading file /mnt/c/git-repo/earlybird/verify/checkfile2.properties ***** Total issues found ***** 0 TOTAL ISSUES

    4 files scanned in 18.7711ms

    The first entry is "Reading file" without the data of the file in folder.

    bug 
    opened by spaluchowski 1
  • Error: invalid memory address or nil pointer dereference

    Error: invalid memory address or nil pointer dereference

    I have just built the binaries from the source code (both linux/amd64 and windows/amd64 behave the same way). When executing it I get as a result:

    panic: runtime error: invalid memory address or nil pointer dereference [signal SIGSEGV: segmentation violation code=0x1 addr=0x18 pc=0xa12032]

    goroutine 1 [running]: github.com/americanexpress/earlybird/pkg/core.(*EarlybirdCfg).GetRuleModulesMap.func1(0xc000026f60, 0x19, 0x0, 0x0, 0xc47f00, 0xc00010ed80, 0xc000107be8, 0x4108ad) /var/lib/jenkins/workspace/Earlybird-build-binaries/earlybird/pkg/core/core.go:148 +0x32 path/filepath.Walk(0xc000026f60, 0x19, 0xc000107c30, 0xc000026f60, 0x19) /usr/lib/golang/src/path/filepath/path.go:404 +0x6b github.com/americanexpress/earlybird/pkg/core.(*EarlybirdCfg).GetRuleModulesMap(0x1044e80, 0x10445e0, 0xc00002c810) /var/lib/jenkins/workspace/Earlybird-build-binaries/earlybird/pkg/core/core.go:147 +0xef github.com/americanexpress/earlybird/pkg/core.(*EarlybirdCfg).ConfigInit(0x1044e80) /var/lib/jenkins/workspace/Earlybird-build-binaries/earlybird/pkg/core/core.go:176 +0x2ef main.main() /var/lib/jenkins/workspace/Earlybird-build-binaries/earlybird/go-earlybird.go:47 +0x35d

    question 
    opened by spaluchowski 3
  • Incorrect path to file in case of keys detection

    Incorrect path to file in case of keys detection

    Problem

    Files containing keys are not reported with full path to file, only file name is printed out in JSON report (haven't tested other types of report).

    Example

    "caption": "Key database file",
    "filename": "keystore.jks",
    

    instead of:

    "caption": "Key database file",
    "filename": "src/resources/keystore.jks",
    

    Additional information

    Files containing other type of issues point directly to the right place:

    "caption": "Potential password in file",
    "filename": "war/src/broken/resources/runtime.properties",
    

    Contradictory not all key files are reported that way:

    "caption": "Potential repository key in file",
    "filename": "frontend/properties.toml",
    

    Problem applies to rules like:

    "caption": "Keychain database file",
    "caption": "Potential cryptographic key bundle",
    "caption": "Key database file",
    

    and similar.

    bug 
    opened by spaluchowski 0
  • Embedding default configurations in released binary

    Embedding default configurations in released binary

    Hey there 👋 Love the project and thanks for all your work here!

    Suggestion

    For all files in the config/ directory, I think this is a perfect use case to embed your configurations in the binary utilizing go:embed.

    This would essentially alleviate the requirement for users to install the repository locally to setup the go-earlybird project using your shell scripts because the binary would already have the configs packaged in via embed.FS rather than reading the user's local filesystem for the baseline configs. This then opens up the door for you to distribute an easily packagable go get command, brew install or whatever package manager user's would like because everything they need is right there in the executable.

    This could be the default and then allow users to provide additional configs if they choose to by reading the ~/.go-earlybird/foo directory. As mentioned here in your utils.go

    Specifics

    Referenced Earlybird Config Directory Additional documentation on embed.FS

    So right here where you are reading in the config with an os.Open, if you switch this to use the new embed.FS, you'll get this end result.

    Current

    //LoadConfig parses json configuration file into structure
    func LoadConfig(cfg interface{}, path string) (err error) {
    
    	jsonFile, err := os.Open(path)
    	// if we os.Open returns an error then handle it
    	if err != nil {
    		return err
    	}
    ...
    

    Proposed Change

    + //go:embed config/
    +var embeddedConfig embed.FS
    
    //LoadConfig parses json configuration file into structure
    func LoadConfig(cfg interface{}, path string) (err error) {
    
    +	jsonFile, err := embeddedConfig.Open(path)
    	// if we embeddedConfig.Open returns an error then handle it
    	if err != nil {
    		return err
    	}
    ...
    
    enhancement 
    opened by bhayes-zd 1
  • Base directory should be ignored during ignore matching

    Base directory should be ignored during ignore matching

    This is rather an enhancement than a bug, but makes things more clear

    As of now whole path is being checked against ignorefile BUT Imagine the situation that:

    1. Ignore file contains entry "/test/"
    2. User uses his /var/lib/test/projects/ directory to download his projects into Then when Earlybird is executed with parameter: -path " /var/lib/test/projects/" it will ignore all project files, nothing will be scanned

    Proposed remediation: Base directory path should be removed from matching against ignored patterns

    enhancement 
    opened by spaluchowski 1
  • Ignorefile case sensitivity is broken

    Ignorefile case sensitivity is broken

    .ge_ignore file case sensitivity works in a weird way. At least on Windows machine. Entries in ignore file have to be put in lower case to match something.

    The problem is that when in ignore file is entry: '*.txt' it matches any *.txt files (i.e. Readme.txt, readme.TXT) which is good behavior. But when I put entry "*.TXT" it does not match anything (at least not expecting readme.TXT). It showed when I tried to exclude some path which contained word "Libs" (i.e. "Libs/sweet.lib"). When I tried with all combinations of "*/Libs/*", "*Libs*", "Libs*" nothing worked. Only "*/libs/*" was matching.

    This is not a huge problem, but it is a very counter intuitive behavior.

    Version: 1.24.6

    good first issue 
    opened by spaluchowski 0
Owner
American Express
American Express
Easy to use cryptographic framework for data protection: secure messaging with forward secrecy and secure data storage. Has unified APIs across 14 platforms.

Themis provides strong, usable cryptography for busy people General purpose cryptographic library for storage and messaging for iOS (Swift, Obj-C), An

Cossack Labs 1.3k Sep 9, 2021
Cossack Labs 793 Sep 8, 2021
🌘🦊 DalFox(Finder Of XSS) / Parameter Analysis and XSS Scanning tool based on golang

Finder Of XSS, and Dal(달) is the Korean pronunciation of moon. What is DalFox ?? ?? DalFox is a fast, powerful parameter analysis and XSS scanner, bas

HAHWUL 1.2k Sep 12, 2021
A tool to check for vulnerabilities in your Golang dependencies, powered by Sonatype OSS Index

Nancy nancy is a tool to check for vulnerabilities in your Golang dependencies, powered by Sonatype OSS Index, and as well, works with Nexus IQ Server

Sonatype Community 317 Sep 10, 2021
Container Signing

cosign Container Signing, Verification and Storage in an OCI registry. Cosign aims to make signatures invisible infrastructure. Info Cosign is develop

sigstore 955 Sep 13, 2021
MX1014 is a flexible, lightweight and fast port scanner.

MX1014 MX1014 是一个遵循 “短平快” 原则的灵活、轻便和快速端口扫描器 此工具仅限于安全研究和教学,用户承担因使用此工具而导致的所有法律和相关责任! 作者不承担任何法律和相关责任! Version 1.1.1 - 版本修改日志 Features 兼容 nmap 的端口和目标语法 支持各

L 22 Aug 3, 2021
Ah shhgit! Find secrets in your code. Secrets detection for your GitHub, GitLab and Bitbucket repositories: www.shhgit.com

shhgit helps secure forward-thinking development, operations, and security teams by finding secrets across their code before it leads to a security br

Paul 3.3k Sep 12, 2021
gosec - Golang Security Checker

Inspects source code for security problems by scanning the Go AST.

Secure Go 5.4k Sep 15, 2021
A fully self-contained Nmap like parallel port scanning module in pure Golang that supports SYN-ACK (Silent Scans)

gomap What is gomap? Gomap is a fully self-contained nmap like module for Golang. Unlike other projects which provide nmap C bindings or rely on other

jtimperio 26 Sep 7, 2021
ServerScan一款使用Golang开发的高并发网络扫描、服务探测工具。

ServerScan ███████╗███████╗██████╗ ██╗ ██╗███████╗██████╗ ███████╗ ██████╗ █████╗ ███╗ ██╗ ██╔════╝██╔════╝██╔══██╗██║ ██║██╔════╝██╔══██╗

Trim 912 Sep 7, 2021
Nuclei is a fast tool for configurable targeted vulnerability scanning based on templates offering massive extensibility and ease of use.

Fast and customisable vulnerability scanner based on simple YAML based DSL. How • Install • For Security Engineers • For Developers • Documentation •

ProjectDiscovery 5.2k Sep 12, 2021
Password generator written in Go

go-generate-password Password generator written in Go. Use as a library or as a CLI. Usage CLI go-generate-password can be used on the cli, just insta

Miles Croxford 27 Aug 25, 2021
crowdsec 3.7k Sep 11, 2021
DockerSlim (docker-slim): Don't change anything in your Docker container image and minify it by up to 30x (and for compiled languages even more) making it secure too! (free and open source)

Minify and Secure Docker containers (free and open source!) Don't change anything in your Docker container image and minify it by up to 30x making it

docker-slim 10.6k Sep 13, 2021
Validate the Strength of a Password in Go

go-password-validator Simple password validator using raw entropy values. Hit the project with a star if you find it useful ⭐ Supported by Qvault This

Lane Wagner 290 Sep 4, 2021
go实现iOS重签名模块(iOS超级签名、蒲公英ios内测分发原理)

这是什么 一个用go实现的iOS重签名模块,即市面上的iOS超级签名、蒲公英ios内测分发原理 使用本模块可以进行基本的IPA安装包重签名分发 实现功能:苹果开发者账号管理、IPA安装包管理 前提 1.生成ios.csr和ios.key文件 openssl genrsa -out

寻寻觅觅 247 Sep 11, 2021
Secure software enclave for storage of sensitive information in memory.

MemGuard Software enclave for storage of sensitive information in memory. This package attempts to reduce the likelihood of sensitive data being expos

Awn 2k Sep 14, 2021
🔑 A decentralized key derivation protocol for simple passphrase.

Throttled Identity Protocol (TIP) is a decentralized key derivation protocol, which allows people to obtain a strong secret key through a very simple passphrase, e.g. a six-digit PIN.

Mixin Network 24 Aug 24, 2021