Ah shhgit! Find secrets in your code. Secrets detection for your GitHub, GitLab and Bitbucket repositories: www.shhgit.com

Overview

shhgit helps secure forward-thinking development, operations, and security teams by finding secrets across their code before it leads to a security breach.

Go

NEW: Interested in using shhgit to secure your enterprise?

Accidentally leaking secrets — usernames and passwords, API tokens, or private keys — in a public code repository is a developers and security teams worst nightmare. Fraudsters constantly scan public code repositories for these secrets to gain a foothold in to systems. Code is more connected than ever so often these secrets provide access to private and sensitive data — cloud infrastructures, database servers, payment gateways, and file storage systems to name a few.

shhgit can constantly scan your code repositories to find and alert you of these secrets.

Installation

You have two options. I'd recommend the first as it will give you access to the shhgit live web interface. Use the second option if you just want the command line interface.

via Docker

  1. Clone this repository: git clone https://github.com/eth0izzle/shhgit.git
  2. Build via Docker compose: docker-compose build
  3. Edit your config.yaml file (i.e. adding your GitHub tokens)
  4. Bring up the stack: docker-compose up
  5. Open up http://localhost:8080/

via Go get

Note: this method does not include the shhgit web interface

  1. Install Go for your platform.
  2. go get github.com/eth0izzle/shhgit will download and build shhgit automatically. Or you can clone this repository and run go build -v -i.
  3. Edit your config.yaml file and see usage below.

Usage

shhgit can work in two ways: consuming the public APIs of GitHub, Gist, GitLab and BitBucket or by processing files in a local directory.

By default, shhgit will run in the former 'public mode'. For GitHub and Gist, you will need to obtain and provide an access token (see this guide; it doesn't require any scopes or permissions. And then place it under github_access_tokens in config.yaml). GitLab and BitBucket do not require any API tokens.

You can also forgo the signatures and use shhgit with your own custom search query, e.g. to find all AWS keys you could use shhgit --search-query AWS_ACCESS_KEY_ID=AKIA. And to run in local mode (and perhaps integrate in to your CI pipelines) you can pass the --local flag (see usage below).

Options

--clone-repository-timeout
        Maximum time it should take to clone a repository in seconds (default 10)
--config-path
        Searches for config.yaml from given directory. If not set, tries to find if from shhgit binary's and current directory
--csv-path
        Specify a path if you want to write found secrets to a CSV. Leave blank to disable
--debug
        Print debugging information
--entropy-threshold
        Finds high entropy strings in files. Higher threshold = more secret secrets, lower threshold = more false positives. Set to 0 to disable entropy checks (default 5.0)
--local
        Specify local directory (absolute path) which to scan. Scans only given directory recursively. No need to have Github tokens with local run.
--maximum-file-size
        Maximum file size to process in KB (default 512)
--maximum-repository-size
        Maximum repository size to download and process in KB) (default 5120)
--minimum-stars
        Only clone repositories with this many stars or higher. Set to 0 to ignore star count (default 0)
--path-checks
        Set to false to disable file name/path signature checking, i.e. just match regex patterns (default true)
--process-gists
        Watch and process Gists in real time. Set to false to disable (default true)
--search-query
        Specify a search string to ignore signatures and filter on files containing this string (regex compatible)
--silent
        Suppress all output except for errors
--temp-directory
        Directory to store repositories/matches (default "%temp%\shhgit")
--threads
        Number of concurrent threads to use (default number of logical CPUs)

Config

The config.yaml file has 7 elements. A default is provided.

github_access_tokens: # provide at least one token
  - 'token one'
  - 'token two'
webhook: '' # URL to a POST webhook.
webhook_payload: '' # Payload to POST to the webhook URL
blacklisted_strings: [] # list of strings to ignore
blacklisted_extensions: [] # list of extensions to ignore
blacklisted_paths: [] # list of paths to ignore
blacklisted_entropy_extensions: [] # additional extensions to ignore for entropy checks
signatures: # list of signatures to check
  - part: '' # either filename, extension, path or contents
    match: '' # simple text comparison (if no regex element)
    regex: '' # regex pattern (if no match element)
    name: '' # name of the signature

Signatures

shhgit comes with 150 signatures. You can remove or add more by editing the config.yaml file.

1Password password manager database file, Amazon MWS Auth Token, Apache htpasswd file, Apple Keychain database file, Artifactory, AWS Access Key ID, AWS Access Key ID Value, AWS Account ID, AWS CLI credentials file, AWS cred file info, AWS Secret Access Key, AWS Session Token, Azure service configuration schema file, Carrierwave configuration file, Chef Knife configuration file, Chef private key, CodeClimate, Configuration file for auto-login process, Contains a private key, Contains a private key, cPanel backup ProFTPd credentials file, Day One journal file, DBeaver SQL database manager configuration file, DigitalOcean doctl command-line client configuration file, Django configuration file, Docker configuration file, Docker registry authentication file, Environment configuration file, esmtp configuration, Facebook access token, Facebook Client ID, Facebook Secret Key, FileZilla FTP configuration file, FileZilla FTP recent servers file, Firefox saved passwords DB, git-credential-store helper credentials file, Git configuration file, GitHub Hub command-line client configuration file, Github Key, GNOME Keyring database file, GnuCash database file, Google (GCM) Service account, Google Cloud API Key, Google OAuth Access Token, Google OAuth Key, Heroku API key, Heroku config file, Hexchat/XChat IRC client server list configuration file, High entropy string, HockeyApp, Irssi IRC client configuration file, Java keystore file, Jenkins publish over SSH plugin file, Jetbrains IDE Config, KDE Wallet Manager database file, KeePass password manager database file, Linkedin Client ID, LinkedIn Secret Key, Little Snitch firewall configuration file, Log file, MailChimp API Key, MailGun API Key, Microsoft BitLocker recovery key file, Microsoft BitLocker Trusted Platform Module password file, Microsoft SQL database file, Microsoft SQL server compact database file, Mongoid config file, Mutt e-mail client configuration file, MySQL client command history file, MySQL dump w/ bcrypt hashes, netrc with SMTP credentials, Network traffic capture file, NPM configuration file, NuGet API Key, OmniAuth configuration file, OpenVPN client configuration file, Outlook team, Password Safe database file, PayPal/Braintree Access Token, PHP configuration file, Picatic API key, Pidgin chat client account configuration file, Pidgin OTR private key, PostgreSQL client command history file, PostgreSQL password file, Potential cryptographic private key, Potential Jenkins credentials file, Potential jrnl journal file, Potential Linux passwd file, Potential Linux shadow file, Potential MediaWiki configuration file, Potential private key (.asc), Potential private key (.p21), Potential private key (.pem), Potential private key (.pfx), Potential private key (.pkcs12), Potential PuTTYgen private key, Potential Ruby On Rails database configuration file, Private SSH key (.dsa), Private SSH key (.ecdsa), Private SSH key (.ed25519), Private SSH key (.rsa), Public ssh key, Python bytecode file, Recon-ng web reconnaissance framework API key database, remote-sync for Atom, Remote Desktop connection file, Robomongo MongoDB manager configuration file, Rubygems credentials file, Ruby IRB console history file, Ruby on Rails master key, Ruby on Rails secrets, Ruby On Rails secret token configuration file, S3cmd configuration file, Salesforce credentials, Sauce Token, Sequel Pro MySQL database manager bookmark file, sftp-deployment for Atom, sftp-deployment for Atom, SFTP connection configuration file, Shell command alias configuration file, Shell command history file, Shell configuration file (.bashrc, .zshrc, .cshrc), Shell configuration file (.exports), Shell configuration file (.extra), Shell configuration file (.functions), Shell profile configuration file, Slack Token, Slack Webhook, SonarQube Docs API Key, SQL Data dump file, SQL dump file, SQLite3 database file, SQLite database file, Square Access Token, Square OAuth Secret, SSH configuration file, SSH Password, Stripe API key, T command-line Twitter client configuration file, Terraform variable config file, Tugboat DigitalOcean management tool configuration, Tunnelblick VPN configuration file, Twilo API Key, Twitter Client ID, Twitter Secret Key, Username and password in URI, Ventrilo server configuration file, vscode-sftp for VSCode, Windows BitLocker full volume encrypted data file, WP-Config

Contributing

  1. Fork it, baby!
  2. Create your feature branch: git checkout -b my-new-feature
  3. Commit your changes: git commit -am 'Add some feature'
  4. Push to the branch: git push origin my-new-feature
  5. Submit a pull request.

Disclaimer

I take no responsibility for how you use this tool. Don't be a dick.

License

MIT. See LICENSE

Issues
  • Dockerfile

    Dockerfile

    opened by fnxpt 11
  • Include shhgit-live front-end interface

    Include shhgit-live front-end interface

    From https://shhgit.darkport.co.uk/

    TODO: clean up the code and do some further testing across platforms.

    enhancement 
    opened by eth0izzle 10
  • installation and setup problems:which webhook url to use and payload, go get(cloning) not working

    installation and setup problems:which webhook url to use and payload, go get(cloning) not working

    shhgit 1 shhgit 2

    i have a number of pproblems but all can be solved through adding EXAMPLE instances of the apps installation process and usage. in installing in the powershell cli i used the cloning method and run the go command stated and nothing happened.When i cloned and clicked ok It showed the cant find cmdlet error. i tried a couple of things stated in the pictures below with no progress. i also had questions concerning which webhook is in use (assuming things went successful after cloning) is it a slack webhook url ? and if so what about the webhook payload? also in using the app would the commands start with shhgit then a command eg: shhgit --entropy-threshold or shhgit --maximum-file-size and what about when combining them as i tried just to test and ask you guys and like how it shows in the last picture the maximum file size color shows that it is not being processed as a command like how entropy-threshold is (by looking at the yellow color) lastly would i have to add a repository of an org/ user after those commands or in a config file or does it simply look at all repos in github. If so what would i have to do to narrow down to certain repositories? Thank you for your time ADDING EXAMPLES would be HELPFUL to as script kiddies who value this tool. as i've searched for information about this tool elsewhere with no luck.

    opened by phill96 8
  • Added support for running shhgit locally

    Added support for running shhgit locally

    Added support to provide configuration file path as an parameter (helps if running within Docker)

    Updated local run documentation

    opened by Hi-Fi 5
  • custom webhooks and other webhook modifications

    custom webhooks and other webhook modifications

    Currently, there is no implementation that is specific to Slack Webhooks, so this PR makes it more obvious that it's just POSTing to a URL with the pre-determined JSON template of {"text": "..."}.

    Also, INFO felt too low for a notification-worthy log level on slack/mattermost/etc.

    The resulting notification had ANSI color characters, making it hard to read, so those were stripped as well. ( #30 )

    • raises loglevel threshold for webhook notifications
    • remove ANSI color codes from HTTP payloads for cleaner notifications
    • modify docs/code to reflect generic capabilities of webhooks
    opened by audibleblink 5
  • Format code and fix file permissions

    Format code and fix file permissions

    Just ran gofmt/goimports over the entire codebase.

    opened by muesli 5
  • errors parsing go.mod

    errors parsing go.mod

    I'm getting this error :/ Could you help me remove this ?

    Vicky:shhgit dv$ docker build --tag fnxpt/shhgit:latest .
    \Sending build context to Docker daemon  157.2kB
    Step 1/9 : FROM golang:alpine AS builder
     ---> dda4232b2bd5
    Step 2/9 : WORKDIR /go/src
     ---> Using cache
     ---> 50bf36eb4bc4
    Step 3/9 : ADD . .
     ---> Using cache
     ---> 1f7e4ac77b67
    Step 4/9 : RUN export CGO_ENABLED=0 && go install && go build -o /
     ---> Running in e1c84a3a3020
    go: errors parsing go.mod:
    /go/src/go.mod:3: usage: go 1.23
    The command '/bin/sh -c export CGO_ENABLED=0 && go install && go build -o /' returned a non-zero code: 1
    Vicky:shhgit dv$ 
    
    opened by DJ621 4
  • Usage Instructions for Gitlab

    Usage Instructions for Gitlab

    Hi We have about 200 gitlab repos- could you please help us with usage instructions with gitlab.

    opened by vikas1389 4
  • workflows: add simple docker action

    workflows: add simple docker action

    This fixes #53

    opened by RiRa12621 4
  • Azure DevOps support

    Azure DevOps support

    Your project is so cool.

    My org is using Azure DevOps and it would be great if we could use this project.

    It is though unclear what has to be done in order to integrate, hence also big challenge for me to help out. If there is any documentation or help here I would gladly try to dig in.

    opened by TheLeftMoose 3
  • Is there a unit test to assert the regular expressions are working in config.yaml?

    Is there a unit test to assert the regular expressions are working in config.yaml?

    If I add a new regex, is there a test that needs to be updated too?

    How are all the rules in config.yaml tested?

    opened by dustinsand 0
  • How to set more threads in Docker?

    How to set more threads in Docker?

    Hello i setup the tool and look amazing. Im trying to setup the docker with the --threads 3, i put 3 github tokens and inside docker-compose.yml i set this

    entrypoint: ["/app/shhgit", "--threads 3 --live=http://shhgit-www/push"]

    But when i compose up i get this error, any advice to set flags for the docker?

    Attaching to shhgit.www, shhgit.app
    shhgit.app    | flag provided but not defined: -threads 3
    shhgit.app    | Usage of /app/shhgit:
    shhgit.app    |   -clone-repository-timeout uint
    shhgit.app    |         Maximum time it should take to clone a repository in seconds. Increase this if you have a slower connection (default 10)
    shhgit.app    |   -config-path string
    shhgit.app    |         Searches for config.yaml from given directory. If not set, tries to find if from shhgit binary's and current directory
    shhgit.app    |   -csv-path string
    shhgit.app    |         CSV file path to log found secrets to. Leave blank to disable
    shhgit.app    |   -debug
    shhgit.app    |         Print debugging information
    shhgit.app    |   -entropy-threshold float
    shhgit.app    |         Set to 0 to disable entropy checks (default 5)
    shhgit.app    |   -live string
    shhgit.app    |         Your shhgit live endpoint
    shhgit.app    |   -local string
    shhgit.app    |         Specify local directory (absolute path) which to scan. Scans only given directory recursively. No need to have GitHub tokens with local run.
    shhgit.app    |   -maximum-file-size uint
    shhgit.app    |         Maximum file size to process in KB (default 256)
    shhgit.app    |   -maximum-repository-size uint
    shhgit.app    |         Maximum repository size to process in KB (default 5120)
    shhgit.app    |   -minimum-stars uint
    shhgit.app    |         Only process repositories with this many stars. Default 0 will ignore star count
    shhgit.app    |   -path-checks
    shhgit.app    |         Set to false to disable checking of filepaths, i.e. just match regex patterns of file contents (default true)
    shhgit.app    |   -process-gists
    shhgit.app    |         Will watch and process Gists. Set to false to disable. (default true)
    shhgit.app    |   -search-query string
    shhgit.app    |         Specify a search string to ignore signatures and filter on files containing this string (regex compatible)
    shhgit.app    |   -silent
    shhgit.app    |         Suppress all output except for errors
    shhgit.app    |   -temp-directory string
    shhgit.app    |         Directory to process and store repositories/matches (default "/tmp/shhgit")
    shhgit.app    |   -threads int
    shhgit.app    |         Number of concurrent threads (default number of logical CPUs)
    shhgit.app exited with code 2
    

    Thanks!

    opened by elch0laj 0
  • For Bug Bounty hunting - Filter by company

    For Bug Bounty hunting - Filter by company

    Hello @eth0izzle, you posted in this reddit thread: REDDIT in regards to the fact that with some tweaking of the signatures, shhgit would make for a great addition to bug bounty hunting workflows, and directed us to Bug bounty targets data. I've tried tweaking with the signatures and also grepping the results in order to filter for a specific company but it's no easy task doing this. I was wondering if a filter by company could be implemented or if you have other ideas for us bug bounty hunters to use shhgit in order to automate the secret discovery process. A guide would really help. Thank you!

    opened by elxandre 0
  • After 5000 request goes to sleep

    After 5000 request goes to sleep

    Hi,

    The tool is working fine. The only problem for me is after exhausting 5000 requests it goes to sleep. Now I understand that there is a limit for requests set by github but is there any way to bypass that?

    Any help would be greatly appreciated.

    Thanks!

    opened by Flash7797 0
  • Live Error: Error connecting to shhgit. Reload to retry?

    Live Error: Error connecting to shhgit. Reload to retry?

    The webpage loads and I'm getting matches in the console, but I keep getting that error on the website and it doesn't show any matches.

    opened by CryptoDevol 3
  • ERROR: Service 'shhgit-www' failed to build : no matching manifest for windows/amd64 10.0.19042 in the manifest list entries

    ERROR: Service 'shhgit-www' failed to build : no matching manifest for windows/amd64 10.0.19042 in the manifest list entries

    B:!---Tools\shhgit-master λ docker-compose build Building shhgit-www Step 1/7 : FROM debian:buster-slim AS builder buster-slim: Pulling from library/debian ERROR: Service 'shhgit-www' failed to build : no matching manifest for windows/amd64 10.0.19042 in the manifest list entries

    opened by CryptoDevol 0
  • Could you please a new tag for releases?

    Could you please a new tag for releases?

    Hello, I maintain the shhgit package in the Arch User Repository and currently the package is not working because the latest available release for shhgit is v0.2 which is almost 2 years old. This makes packaging shhgit harder as v0.2 uses an older Go version.

    I´d greatly appreciate it if you could add a more updated tag. 😄 Thanks.

    opened by da-edra 0
  • Add support for new GitHub authentication token format

    Add support for new GitHub authentication token format

    https://github.blog/changelog/2021-03-31-authentication-token-format-updates-are-generally-available/

    opened by x-way 0
  • Support for Azure Devops Cloud and On-Premise

    Support for Azure Devops Cloud and On-Premise

    I would like to request support for Azure Devops Cloud and On-Premise.

    Here is the link to the list repository documentation and get repository documentation https://docs.microsoft.com/en-us/rest/api/azure/devops/git/repositories/list?view=azure-devops-server-rest-5.0 https://docs.microsoft.com/en-us/rest/api/azure/devops/git/repositories/get%20repository?view=azure-devops-server-rest-5.0

    opened by cliftonz1 0
  • Links don't point to correct location

    Links don't point to correct location

    The links are are erroneously including the git hash in the url:

    As an example, you can see the hash in the URL, when linking to the file: https://github.com/<REDACTED/<REDACTED>/blob/master/2a078102<REDACTED>658bdb65a8c881/start/server/store.sqlite

    I think this should either be /blob/master/$file or /blob/2a078102<REDACTED>658bdb65a8c881/$file.

    image

    image

    Running current master 65351a789931d3a6e7be37e983d7c861103fdec6

    opened by Daviey 1
Owner
Paul
I make stuff, break stuff, and protect stuff from getting hacked. @darkp0rt
Paul
Cossack Labs 807 Oct 15, 2021
A containerd runc shim for replacing environment variables with external secrets

ext-secrets-runc-shim A containerd, runc-based, shim for replacing environment variables with secrets from arbitrary external engines. Quickstart Inst

Pelotech 5 Jul 29, 2021
How to systematically secure anything: a repository about security engineering

How to Secure Anything Security engineering is the discipline of building secure systems. Its lessons are not just applicable to computer security. In

Veeral Patel 6.4k Oct 17, 2021
Not Yet Another Password Manager written in Go using libsodium

secrets Secure and simple passwords manager written in Go. It aims to be NYAPM (Not Yet Another Password Manager), but tries to be different from othe

Jarmo Pertman 25 Oct 9, 2021
A tool for secrets management, encryption as a service, and privileged access management

Vault Please note: We take Vault's security and our users' trust very seriously. If you believe you have found a security issue in Vault, please respo

HashiCorp 22k Oct 18, 2021
crowdsec 3.9k Oct 22, 2021
Take a list of domains and scan for endpoints, secrets, api keys, file extensions, tokens and more...

Take a list of domains and scan for endpoints, secrets, api keys, file extensions, tokens and more... Coded with ?? by edoardottt. Share on Twitter! P

gilfoyle97 301 Oct 16, 2021
High-Performance Shortlink ( Short URL ) app creator in Golang. For privacy reasons, you may prefer to host your own short URL app and this is the one to use.

About The Project Shortlink App in Golang Multiple Node based Architecture to create and scale at ease Highly performant key-value storage system Cent

null 116 Oct 12, 2021
null 669 Oct 24, 2021
Telling tales on you for leaking secrets!

Squealer Telling tales on you for leaking secrets! Squealer scans a local git repository for secrets that are being leaked deep within the commit hist

Owen Rumney 110 Oct 1, 2021
ServerScan一款使用Golang开发的高并发网络扫描、服务探测工具。

ServerScan ███████╗███████╗██████╗ ██╗ ██╗███████╗██████╗ ███████╗ ██████╗ █████╗ ███╗ ██╗ ██╔════╝██╔════╝██╔══██╗██║ ██║██╔════╝██╔══██╗

Trim 947 Oct 23, 2021
EarlyBird is a sensitive data detection tool capable of scanning source code repositories for clear text password violations, PII, outdated cryptography methods, key files and more.

EarlyBird is a sensitive data detection tool capable of scanning source code repositories for clear text password violations, PII, outdated cryptograp

American Express 432 Oct 15, 2021
DockerSlim (docker-slim): Don't change anything in your Docker container image and minify it by up to 30x (and for compiled languages even more) making it secure too! (free and open source)

Minify and Secure Docker containers (free and open source!) Don't change anything in your Docker container image and minify it by up to 30x making it

docker-slim 10.9k Oct 24, 2021
Build Go applications for IOS

go-build-for-ios Build Go applications for IOS This repository contains a PoC that lets you build any Go application for IOS platform. Cross-compilati

Marcin Tojek 19 Mar 1, 2021
Idiomatic nmap library for go developers

nmap This library aims at providing idiomatic nmap bindings for go developers, in order to make it easier to write security audit tools using golang.

Brendan Le Glaunec 498 Oct 21, 2021
Carbon Black Harbor Adapter is a scanner to scan images in Harbor Registry with the help of Carbon Black Cloud.

carbon-black-adapter-for-harbor Overview Carbon Black adapter for Harbor integrates your Harbor Registry with the Carbon Black Cloud. It leverages Har

VMware 1 Oct 22, 2021
🌘🦊 DalFox(Finder Of XSS) / Parameter Analysis and XSS Scanning tool based on golang

Finder Of XSS, and Dal(달) is the Korean pronunciation of moon. What is DalFox ?? ?? DalFox is a fast, powerful parameter analysis and XSS scanner, bas

HAHWUL 1.4k Oct 23, 2021
A simple, modern and secure encryption tool (and Go library) with small explicit keys, no config options, and UNIX-style composability.

age age is a simple, modern and secure file encryption tool, format, and library. It features small explicit keys, no config options, and UNIX-style c

Filippo Valsorda 8.8k Oct 23, 2021
Tracee: Linux Runtime Security and Forensics using eBPF

Tracee is a Runtime Security and forensics tool for Linux. It is using Linux eBPF technology to trace your system and applications at runtime, and analyze collected events to detect suspicious behavioral patterns.

Aqua Security 1.3k Oct 18, 2021