Ah shhgit! Find secrets in your code. Secrets detection for your GitHub, GitLab and Bitbucket repositories: www.shhgit.com

Overview

shhgit helps secure forward-thinking development, operations, and security teams by finding secrets across their code before it leads to a security breach.

Go

NEW: Interested in using shhgit to secure your enterprise?

Accidentally leaking secrets — usernames and passwords, API tokens, or private keys — in a public code repository is a developers and security teams worst nightmare. Fraudsters constantly scan public code repositories for these secrets to gain a foothold in to systems. Code is more connected than ever so often these secrets provide access to private and sensitive data — cloud infrastructures, database servers, payment gateways, and file storage systems to name a few.

shhgit can constantly scan your code repositories to find and alert you of these secrets.

Installation

You have two options. I'd recommend the first as it will give you access to the shhgit live web interface. Use the second option if you just want the command line interface.

via Docker

  1. Clone this repository: git clone https://github.com/eth0izzle/shhgit.git
  2. Build via Docker compose: docker-compose build
  3. Edit your config.yaml file (i.e. adding your GitHub tokens)
  4. Bring up the stack: docker-compose up
  5. Open up http://localhost:8080/

via Go get

Note: this method does not include the shhgit web interface

  1. Install Go for your platform.
  2. go get github.com/eth0izzle/shhgit will download and build shhgit automatically. Or you can clone this repository and run go build -v -i.
  3. Edit your config.yaml file and see usage below.

Usage

shhgit can work in two ways: consuming the public APIs of GitHub, Gist, GitLab and BitBucket or by processing files in a local directory.

By default, shhgit will run in the former 'public mode'. For GitHub and Gist, you will need to obtain and provide an access token (see this guide; it doesn't require any scopes or permissions. And then place it under github_access_tokens in config.yaml). GitLab and BitBucket do not require any API tokens.

You can also forgo the signatures and use shhgit with your own custom search query, e.g. to find all AWS keys you could use shhgit --search-query AWS_ACCESS_KEY_ID=AKIA. And to run in local mode (and perhaps integrate in to your CI pipelines) you can pass the --local flag (see usage below).

Options

--clone-repository-timeout
        Maximum time it should take to clone a repository in seconds (default 10)
--config-path
        Searches for config.yaml from given directory. If not set, tries to find if from shhgit binary's and current directory
--csv-path
        Specify a path if you want to write found secrets to a CSV. Leave blank to disable
--debug
        Print debugging information
--entropy-threshold
        Finds high entropy strings in files. Higher threshold = more secret secrets, lower threshold = more false positives. Set to 0 to disable entropy checks (default 5.0)
--local
        Specify local directory (absolute path) which to scan. Scans only given directory recursively. No need to have Github tokens with local run.
--maximum-file-size
        Maximum file size to process in KB (default 512)
--maximum-repository-size
        Maximum repository size to download and process in KB) (default 5120)
--minimum-stars
        Only clone repositories with this many stars or higher. Set to 0 to ignore star count (default 0)
--path-checks
        Set to false to disable file name/path signature checking, i.e. just match regex patterns (default true)
--process-gists
        Watch and process Gists in real time. Set to false to disable (default true)
--search-query
        Specify a search string to ignore signatures and filter on files containing this string (regex compatible)
--silent
        Suppress all output except for errors
--temp-directory
        Directory to store repositories/matches (default "%temp%\shhgit")
--threads
        Number of concurrent threads to use (default number of logical CPUs)

Config

The config.yaml file has 7 elements. A default is provided.

github_access_tokens: # provide at least one token
  - 'token one'
  - 'token two'
webhook: '' # URL to a POST webhook.
webhook_payload: '' # Payload to POST to the webhook URL
blacklisted_strings: [] # list of strings to ignore
blacklisted_extensions: [] # list of extensions to ignore
blacklisted_paths: [] # list of paths to ignore
blacklisted_entropy_extensions: [] # additional extensions to ignore for entropy checks
signatures: # list of signatures to check
  - part: '' # either filename, extension, path or contents
    match: '' # simple text comparison (if no regex element)
    regex: '' # regex pattern (if no match element)
    name: '' # name of the signature

Signatures

shhgit comes with 150 signatures. You can remove or add more by editing the config.yaml file.

1Password password manager database file, Amazon MWS Auth Token, Apache htpasswd file, Apple Keychain database file, Artifactory, AWS Access Key ID, AWS Access Key ID Value, AWS Account ID, AWS CLI credentials file, AWS cred file info, AWS Secret Access Key, AWS Session Token, Azure service configuration schema file, Carrierwave configuration file, Chef Knife configuration file, Chef private key, CodeClimate, Configuration file for auto-login process, Contains a private key, Contains a private key, cPanel backup ProFTPd credentials file, Day One journal file, DBeaver SQL database manager configuration file, DigitalOcean doctl command-line client configuration file, Django configuration file, Docker configuration file, Docker registry authentication file, Environment configuration file, esmtp configuration, Facebook access token, Facebook Client ID, Facebook Secret Key, FileZilla FTP configuration file, FileZilla FTP recent servers file, Firefox saved passwords DB, git-credential-store helper credentials file, Git configuration file, GitHub Hub command-line client configuration file, Github Key, GNOME Keyring database file, GnuCash database file, Google (GCM) Service account, Google Cloud API Key, Google OAuth Access Token, Google OAuth Key, Heroku API key, Heroku config file, Hexchat/XChat IRC client server list configuration file, High entropy string, HockeyApp, Irssi IRC client configuration file, Java keystore file, Jenkins publish over SSH plugin file, Jetbrains IDE Config, KDE Wallet Manager database file, KeePass password manager database file, Linkedin Client ID, LinkedIn Secret Key, Little Snitch firewall configuration file, Log file, MailChimp API Key, MailGun API Key, Microsoft BitLocker recovery key file, Microsoft BitLocker Trusted Platform Module password file, Microsoft SQL database file, Microsoft SQL server compact database file, Mongoid config file, Mutt e-mail client configuration file, MySQL client command history file, MySQL dump w/ bcrypt hashes, netrc with SMTP credentials, Network traffic capture file, NPM configuration file, NuGet API Key, OmniAuth configuration file, OpenVPN client configuration file, Outlook team, Password Safe database file, PayPal/Braintree Access Token, PHP configuration file, Picatic API key, Pidgin chat client account configuration file, Pidgin OTR private key, PostgreSQL client command history file, PostgreSQL password file, Potential cryptographic private key, Potential Jenkins credentials file, Potential jrnl journal file, Potential Linux passwd file, Potential Linux shadow file, Potential MediaWiki configuration file, Potential private key (.asc), Potential private key (.p21), Potential private key (.pem), Potential private key (.pfx), Potential private key (.pkcs12), Potential PuTTYgen private key, Potential Ruby On Rails database configuration file, Private SSH key (.dsa), Private SSH key (.ecdsa), Private SSH key (.ed25519), Private SSH key (.rsa), Public ssh key, Python bytecode file, Recon-ng web reconnaissance framework API key database, remote-sync for Atom, Remote Desktop connection file, Robomongo MongoDB manager configuration file, Rubygems credentials file, Ruby IRB console history file, Ruby on Rails master key, Ruby on Rails secrets, Ruby On Rails secret token configuration file, S3cmd configuration file, Salesforce credentials, Sauce Token, Sequel Pro MySQL database manager bookmark file, sftp-deployment for Atom, sftp-deployment for Atom, SFTP connection configuration file, Shell command alias configuration file, Shell command history file, Shell configuration file (.bashrc, .zshrc, .cshrc), Shell configuration file (.exports), Shell configuration file (.extra), Shell configuration file (.functions), Shell profile configuration file, Slack Token, Slack Webhook, SonarQube Docs API Key, SQL Data dump file, SQL dump file, SQLite3 database file, SQLite database file, Square Access Token, Square OAuth Secret, SSH configuration file, SSH Password, Stripe API key, T command-line Twitter client configuration file, Terraform variable config file, Tugboat DigitalOcean management tool configuration, Tunnelblick VPN configuration file, Twilo API Key, Twitter Client ID, Twitter Secret Key, Username and password in URI, Ventrilo server configuration file, vscode-sftp for VSCode, Windows BitLocker full volume encrypted data file, WP-Config

Contributing

  1. Fork it, baby!
  2. Create your feature branch: git checkout -b my-new-feature
  3. Commit your changes: git commit -am 'Add some feature'
  4. Push to the branch: git push origin my-new-feature
  5. Submit a pull request.

Disclaimer

I take no responsibility for how you use this tool. Don't be a dick.

License

MIT. See LICENSE

Issues
  • installation and setup problems:which webhook url to use and payload, go get(cloning) not working

    installation and setup problems:which webhook url to use and payload, go get(cloning) not working

    shhgit 1 shhgit 2

    i have a number of pproblems but all can be solved through adding EXAMPLE instances of the apps installation process and usage. in installing in the powershell cli i used the cloning method and run the go command stated and nothing happened.When i cloned and clicked ok It showed the cant find cmdlet error. i tried a couple of things stated in the pictures below with no progress. i also had questions concerning which webhook is in use (assuming things went successful after cloning) is it a slack webhook url ? and if so what about the webhook payload? also in using the app would the commands start with shhgit then a command eg: shhgit --entropy-threshold or shhgit --maximum-file-size and what about when combining them as i tried just to test and ask you guys and like how it shows in the last picture the maximum file size color shows that it is not being processed as a command like how entropy-threshold is (by looking at the yellow color) lastly would i have to add a repository of an org/ user after those commands or in a config file or does it simply look at all repos in github. If so what would i have to do to narrow down to certain repositories? Thank you for your time ADDING EXAMPLES would be HELPFUL to as script kiddies who value this tool. as i've searched for information about this tool elsewhere with no luck.

    opened by phill96 8
  • custom webhooks and other webhook modifications

    custom webhooks and other webhook modifications

    Currently, there is no implementation that is specific to Slack Webhooks, so this PR makes it more obvious that it's just POSTing to a URL with the pre-determined JSON template of {"text": "..."}.

    Also, INFO felt too low for a notification-worthy log level on slack/mattermost/etc.

    The resulting notification had ANSI color characters, making it hard to read, so those were stripped as well. ( #30 )

    • raises loglevel threshold for webhook notifications
    • remove ANSI color codes from HTTP payloads for cleaner notifications
    • modify docs/code to reflect generic capabilities of webhooks
    opened by audibleblink 5
  • errors parsing go.mod

    errors parsing go.mod

    I'm getting this error :/ Could you help me remove this ?

    Vicky:shhgit dv$ docker build --tag fnxpt/shhgit:latest .
    \Sending build context to Docker daemon  157.2kB
    Step 1/9 : FROM golang:alpine AS builder
     ---> dda4232b2bd5
    Step 2/9 : WORKDIR /go/src
     ---> Using cache
     ---> 50bf36eb4bc4
    Step 3/9 : ADD . .
     ---> Using cache
     ---> 1f7e4ac77b67
    Step 4/9 : RUN export CGO_ENABLED=0 && go install && go build -o /
     ---> Running in e1c84a3a3020
    go: errors parsing go.mod:
    /go/src/go.mod:3: usage: go 1.23
    The command '/bin/sh -c export CGO_ENABLED=0 && go install && go build -o /' returned a non-zero code: 1
    Vicky:shhgit dv$ 
    
    opened by DJ621 4
  • Azure DevOps support

    Azure DevOps support

    Your project is so cool.

    My org is using Azure DevOps and it would be great if we could use this project.

    It is though unclear what has to be done in order to integrate, hence also big challenge for me to help out. If there is any documentation or help here I would gladly try to dig in.

    opened by TheLeftMoose 3
  • Support for GitHub Enterprise implemented here

    Support for GitHub Enterprise implemented here

    This is pretty rough right now but it gets the job done.

    https://github.com/0xtavian/shhgit

    Things changed: Added basic auth functionally for Cloning repos which is required by GHE https://github.com/0xtavian/shhgit/blob/master/core/git.go

    Implemented baseURL here https://github.com/0xtavian/shhgit/blob/master/core/session.go#L65

    opened by 0xtavian 3
  • shhgit stuck when more than one token were added to config.yaml

    shhgit stuck when more than one token were added to config.yaml

    CLI print:

    [email protected]:~/shhgit# shhgit 
    
          _     _           _ _   
         | |   | |         (_) |  
      ___| |__ | |__   __ _ _| |_ 
     / __| '_ \| '_ \ / _` | | __|
     \__ \ | | | | | | (_| | | |_ 
     |___/_| |_|_| |_|\__, |_|\__|
                       __/ |      
        v0.4          |___/
            Paul Price (@darkp0rt) - www.darkport.co.uk
    
    [*] Loaded 150 signatures. Using 1 worker threads. Temp work dir: /tmp/shhgit
    
    [?] Token ghp_uDbEXk[..] has 51/60 calls remaining.
    -^C
    [email protected]:~/shhgit# shhgit 
    
    
    
    
    
    ^C
    [email protected]:~/shhgit# 
    
    opened by fngoo 2
  • Watch GitHub issues for secrets

    Watch GitHub issues for secrets

    I've found many secrets in GitHub issue comments, i.e. people copy pasting their code asking for help without redacting the secrets/keys - you can even view comment history if they were later removed.

    We can listen to the IssueCommentEvent to get a stream of real time comments (https://docs.github.com/en/developers/webhooks-and-events/github-event-types#issuecommentevent) and process the comment key within the payload as if it were code (we would need to skip file path + extension checks).

    enhancement 
    opened by eth0izzle 2
  • [BUG] checkSignatures() is breaking local repos

    [BUG] checkSignatures() is breaking local repos

    https://github.com/eth0izzle/shhgit/blob/3fb0d7df259a645d5027df5b021cebaa6c5a373f/main.go#L140

    Is causing issues when running this locally because it removes unexpected folders like .git, potentially breaking people's local work

    opened by RiRa12621 2
  • Config Yaml file into container

    Config Yaml file into container

    The goal was to simplify the usage and build of a Dockerised shhgit image. The workflow is now simpler :

    • The user git clone
    • The user modify his config.yaml (if not for local repository)
    • The user build the image with the config.yaml file within it
    opened by DloomPlz 2
  • Support of Bitbucket on Private server

    Support of Bitbucket on Private server

    I was wondering if there is a possibility to support bitbucket for enterprise usage. For instance, if a company has their private bitbucket server and would like to monitor the secrets. How difficult is it to tweak shhgit to do the job for them?

    opened by ahmedsherif 2
  • Sleeping forever

    Sleeping forever

    I've let the shhgit run for a while and seems that once all the tokens are exhausted it never wakes up again. I had a look at the code and I think the problem is this: client.RateLimitedUntil is a Duration, it should be decreased or reset to zero once the sleep occurs, so the recursive function that does the sleep can actually escape returning the client. But the code that re/sets the remaining duration (client.RateLimitedUntil) is done after the recursive sleeping function.

    image github.go:94 github.go:29

    I have a very little knowledge about Go so I might be very wrong there but my shhgit did indeed sleep forever. I did repair it in my fork, exchanging the RateLimitedUntil from time.Duration to time.Time (directly using resp.Rate.Reset.Time value) and in the session.GetClient() checking if the date is after time.Now(), if not then sleep the time.Until duration. The question is if the repair makes sense. #7

    EDIT: The image is from GetGists but the same applies to GetRepositories

    opened by wereii 2
  • Error docker install

    Error docker install "docker-compose up"

    Step 8/9 returns the following error

    Step 8/9 : COPY /shhgit /app COPY failed: file not found in build context or excluded by .dockerignore: stat shhgit: file does not exist ERROR: Service 'shhgit-app' failed to build : Build failed Any ideas?

    Installing in UBUNTU 20.04

    opened by RonPulent 1
  • NotGitBleed shhgit support

    NotGitBleed shhgit support

    Although shhgit scans file systems and git repos as far as I am aware it doesn't currently scan commit metadata for passwords.

    Recently this has been published: https://www.notgitbleed.com/

    A lot of Github users of large open source projects accidentally commit their GitHub credentials even when tools such as shhgit are being used at an alarming rate.

    Since this work has been published we have worked with GitHub to mitigate this on GitHub and they have built a scanning tool: https://github.blog/changelog/2022-04-11-secret-scanning-detects-and-revokes-leaked-passwords/

    It would be great to confirm that shhgit doesn't currently scan git commit metadata and to find out if this is something you can support in future.

    opened by carolosf 0
  • Remove globals from core to improve usage as a library

    Remove globals from core to improve usage as a library

    This PR addresses some embedding issues I experienced with the core library:

    • Global sessions made it difficult to process multiple repositories simultaneous
    • Global flags made it equally difficult to have multiple configurations simultaneously
    • Hardcoded config.yaml name felt awkward (For instance, I prefer sshgit.yaml).

    Please let me know if there are any improvements you would like me to make. Thanks for the excellent tool & library!

    opened by tstromberg 0
  • Support extension and filename matching for files larger than MaximumFileSize

    Support extension and filename matching for files larger than MaximumFileSize

    With the current implementation files larger than MaximumFileSize are completely ignored. But if a file is large, we can still process the filename an extension signatures.

    This is really useful for detecting large files like SQL dumps archives etc which are ignored with the current implementation.

    opened by nikos-glikis 0
  • Can't build in Docker

    Can't build in Docker

    Hi, I can run the Go build fine, but wanted to run the Docker build as a comparison but keep running into the following error:

    #5 65.63 dpkg-deb: building package 'nginx-dbg' in '../nginx-dbg_1.20.2-1~buster_arm64.deb'.
    #5 65.63 dpkg-deb: building package 'nginx' in '../nginx_1.20.2-1~buster_arm64.deb'.
    #5 67.84  dpkg-genbuildinfo --build=binary
    #5 67.96  dpkg-genchanges --build=binary >../nginx_1.20.2-1~buster_arm64.changes
    #5 68.06 dpkg-genchanges: info: binary-only upload (no source code included)
    #5 68.06  dpkg-source --after-build .
    #5 68.11 dpkg-buildpackage: info: binary-only upload (no source included)
    #5 68.11 mv: cannot stat 'nginx_1*~buster_amd64.deb': No such file or directory
    ------
    
    failed to solve: rpc error: code = Unknown desc = executor failed running [/bin/sh -c apt-get update -y         && apt-get install --no-install-recommends -y wget git unzip lsb-release gnupg2 dpkg-dev ca-certificates         && echo "deb-src http://nginx.org/packages/`lsb_release -is | tr '[:upper:]' '[:lower:]'` `lsb_release -cs` nginx" | tee /etc/apt/sources.list.d/nginx.list         && wget http://nginx.org/keys/nginx_signing.key && apt-key add nginx_signing.key && rm nginx_signing.key         && cd /tmp         && apt-get update         && apt-get source nginx         && apt-get build-dep nginx --no-install-recommends -y         && git clone https://github.com/wandenberg/nginx-push-stream-module.git nginx-push-stream-module         && cd nginx-1*         && sed -i "[email protected][email protected]_ssl_module --add-module=/tmp/nginx-push-stream-module @g" debian/rules         && dpkg-buildpackage -uc -us -b         && cd ..         && mv nginx_1*~buster_amd64.deb nginx.deb]: exit code: 1
    

    Any suggestions here? I'm building on an M1 Mac for details.

    opened by chenboy3 4
Owner
Paul
I make stuff, break stuff, and protect stuff from getting hacked. @darkp0rt
Paul
Secretsmanager - Secrets management that allows you to store your secrets encrypted in git

I created secretsmanager to store some secrets within a repository. The secrets are encrypted at rest, with readable keys and editable JSON, so you can rename a key or delete it by hand. The cli tool handles the bare minumum of requirements.

Tit Petric 20 May 6, 2022
Find secrets and passwords in container images and file systems

Find secrets and passwords in container images and file systems

null 1.4k Aug 14, 2022
Proto-find is a tool for researchers that lets you find client side prototype pollution vulnerability.

proto-find proto-find is a tool for researchers that lets you find client side prototype pollution vulnerability. How it works proto-find open URL in

null 49 Jul 17, 2022
🔎 Help find Trojan Source vulnerability in code 👀 . Useful for code review in project with multiple collaborators

TrojanSourceFinder TrojanSourceFinder helps developers detect "Trojan Source" vulnerability in source code. Trojan Source vulnerability allows an atta

Ariary 44 Jun 13, 2022
Cossack Labs 1k Aug 14, 2022
Log4j detector and reporting server for scalable detection of vulnerable running processes.

Log4j Detector A client and reporting server to identify systems vulnerable to Log4j at scale. This work is based on Stripe's Remediation Tools, but w

Praetorian 8 Apr 8, 2022
Lightweight static analysis for many languages. Find bug variants with patterns that look like source code.

Lightweight static analysis for many languages. Find bugs and enforce code standards. Semgrep is a fast, open-source, static analysis tool that finds

r2c 6.9k Aug 10, 2022
Volana - Shell command obfuscation to avoid detection systems

volana (moon in malagasy) { Use it ; ??(hide from); ??(detected by) } Shell comm

Ariary 38 Jun 20, 2022
A CVE-2021-22205 Gitlab RCE POC written in Golang

Golang-CVE-2021-22205-POC A bare bones CVE-2021-22205 Gitlab RCE POC written in Golang which affects Gitlab CE/EE < 13.10.3 Gitlab CE/EE < 13.9.6 Gitl

Matt 3 Jul 4, 2022
Finds an identifiable hash value for each version of GitLab vulnerable to CVE-2021-22205

Finds an identifiable hash value for each version of GitLab vulnerable to CVE-2021-22205

GitLab Red Team 1 Jun 30, 2022
Git watchdog will scan your public repository and find out the vulnerabilities

Dependencies Docker Go 1.17 MySQL 8.0.25 Bootstrap Run chmod +x start.sh if start.sh script does not have privileged to run Run ./start.sh --bootstrap

Quang Nguyen 2 Dec 30, 2021
WhiteSource Log4j Detect is a free CLI tool that quickly scans your projects to find vulnerable Log4j versions

Log4jDetect WhiteSource Log4j Detect is a free CLI tool that quickly scans your projects to find vulnerable Log4j versions containing the following kn

WhiteSource 138 Aug 13, 2022
A collection of offensive Go packages inspired by different Go repositories.

OffensiveGolang OffensiveGolang is a collection of offensive Go packs inspired by different repositories. Ideas have been taken from OffensiveGoLang a

MrTux 27 Aug 10, 2022
Take a list of domains and scan for endpoints, secrets, api keys, file extensions, tokens and more...

Take a list of domains and scan for endpoints, secrets, api keys, file extensions, tokens and more... Coded with ?? by edoardottt. Share on Twitter! P

gilfoyle97 531 Aug 16, 2022
A tool for secrets management, encryption as a service, and privileged access management

Vault Please note: We take Vault's security and our users' trust very seriously. If you believe you have found a security issue in Vault, please respo

HashiCorp 25.3k Aug 15, 2022
QR secrets is a cryptographically secure mechanism to store secret data with the highest levels of security and store it on physical paper.

QR Secrets QR secrets is a cryptographically secure mechanism to store secret data with the highest levels of security. Incorporating; AES256-GCM-HKDF

Go Compile 0 Jan 12, 2022
🍷 Find exploits and vulnerabilities in the most important databases.

?? Dionisio Dionisio is a tool that can automate the search for exploits and vulnerabilities. Written in Go and open source, Dionisio has an advanced

Y G Λ O 0 Mar 14, 2022
Telling tales on you for leaking secrets!

Squealer Telling tales on you for leaking secrets! Squealer scans a local git repository for secrets that are being leaked deep within the commit hist

Owen Rumney 120 Jun 30, 2022
Friends don't let friends leak secrets on their terminal window

senv - safer env Friends don't let friends leak secrets in terminal windows. ?? Print your environment to the terminal without worry.

null 103 Aug 4, 2022