A passive reconnaissance tool for known URLs discovery

Overview

sigurlfind3r

release maintenance open issues closed issues license twitter

A passive reconnaissance tool for known URLs discovery - it gathers a list of URLs passively using various online sources.

Resource

Features

  • Collect known URLs:
  • Reduce noise:
    • Regex filter URLs.
    • Removes duplicate pages in the sense of URL patterns that are probably repetitive and points to the same web template.
  • Output to stdout for piping or save to file.

Installation

From Binary

You can download the pre-built binary for your platform from this repository's releases page, extract, then move it to your $PATHand you're ready to go.

From Source

sigurlfind3r requires go1.14+ to install successfully. Run the following command to get the repo

GO111MODULE=on go get -u -v github.com/signedsecurity/sigurlfind3r/cmd/sigurlfind3r

From Github

git clone https://github.com/signedsecurity/sigurlfind3r.git && \
cd sigurlfind3r/cmd/sigurlfind3r/ && \
go build; mv sigurlfind3r /usr/local/bin/ && \
sigurlfind3r -h

Post Installation

sigurlfind3r will work after installation. However, to configure sigurlfind3r to work with certain services - currently github - you will need to have setup API keys. The API keys are stored in the $HOME/.config/sigurlfind3r/conf.yaml file - created upon first run - and uses the YAML format. Multiple API keys can be specified for each of these services.

Example:

version: 1.5.0
sources:
    - commoncrawl
    - github
    - otx
    - urlscan
    - wayback
    - waybackrobots
keys:
    github:
        - d23a554bbc1aabb208c9acfbd2dd41ce7fc9db39
        - asdsd54bbc1aabb208c9acfbd2dd41ce7fc9db39

Usage

DiSCLAIMER: fetching urls from github is a bit slow.

sigurlfind3r -h

This will display help for the tool.

     _                  _  __ _           _ _____
 ___(_) __ _ _   _ _ __| |/ _(_)_ __   __| |___ / _ __
/ __| |/ _` | | | | '__| | |_| | '_ \ / _` | |_ \| '__|
\__ \ | (_| | |_| | |  | |  _| | | | | (_| |___) | |
|___/_|\__, |\__,_|_|  |_|_| |_|_| |_|\__,_|____/|_| 1.5.0
       |___/

USAGE:
  sigurlfind3r [OPTIONS]

OPTIONS:
   -d, --domain            domain to fetch urls for
  -eS, --exclude-sources   comma(,) separated list of sources to exclude
   -f, --filter            URL filtering regex
  -iS, --include-subs      include subdomains' urls
  -lS, --list-sources      list all the available sources
  -nC, --no-color          no color mode
   -s  --silent            silent mode: output urls only
  -uS, --use-sources       comma(,) separated list of sources to use
   -o, --output            output file

Examples

Basic

sigurlfind3r -d tesla.com

Regex filter URLs

sigurlfind3r -d tesla.com -f ".(jpg|jpeg|gif|png|ico|css|eot|tif|tiff|ttf|woff|woff2)"

Include Subdomains' URLs

sigurlfind3r -d tesla.com -iS

Contribution

Issues and Pull Requests are welcome!

Issues
  • Output save in File

    Output save in File

    Hello , Really great tool but I noticed that there is no option for saving output in a file that causes a lot of problem for me. It would be really really best if you kindly add this feature to this tool for saving output in a file.

    enhancement 
    opened by awais922609 2
  • [BUG]

    [BUG]

    Describe the bug panic: runtime error: index out of range [0] with length 0

    Steps To Reproduce ./sigurlfind3r -d example.com -iS Output [ INF ] fetching urls for

    panic: runtime error: index out of range [0] with length 0

    goroutine 18 [running]: github.com/signedsecurity/sigurlfind3r/pkg/sigurlfind3r/scraping/sources/commoncrawl.(*Source).Run.func1() /Users/emilaltynbaev/Documents/bugbounty/tools/sigurlfind3r/pkg/sigurlfind3r/scraping/sources/commoncrawl/commoncrawl.go:82 +0x7bc created by github.com/signedsecurity/sigurlfind3r/pkg/sigurlfind3r/scraping/sources/commoncrawl.(*Source).Run /Users/emilaltynbaev/Documents/bugbounty/tools/sigurlfind3r/pkg/sigurlfind3r/scraping/sources/commoncrawl/commoncrawl.go:63 +0xf2

    Additional context MacOS 12.1

    bug 
    opened by emil246 1
  • feat: Extract URLs from wayback robots.txt files

    feat: Extract URLs from wayback robots.txt files

    This feature adds a new source, waybackrobots. It extracts endpoints marked as disallowed from robots.txt found on your target domain and snapshotted by the Wayback Machine.

    opened by enenumxela 0
  • Question

    Question

    I see this alot but figured i would ask you, why would you use import "github.com/signedsecurity/sigurlfind3r/pkg/sigurlfind3r/session"

    when you can just

    import crawler_utils "path_to_File_name_of_package"
    
    opened by ArkAngeL43 0
  • [BUG]

    [BUG]

    Description Hi, thanks for the amazing tool. When I playing around using 'commoncrawl' as the source I found it sometimes can print out urls but sometimes can't. So I modified the code to print out the errors alongside then I found commoncrawl returned tons of 503 and timeout

    Steps To Reproduce ./sigurlfind3r -d tesla.com --include-subs -uS commoncrawl

    Screenshots Screenshot_sigurlfind3r_1 Screenshot_sigurlfind3r_2

    Additional context I tested over my home network as well as a VPS located on Los Angeles. I changed a bit of the code to avoid parallel processing and increased the timeout to 60s from 10s. Then it worked as expected. I haven't found any official documented rate limit of commoncrawl though, but reliability is the top concern in my opinion, especially in terms of automation. Just for your reference and thanks again for your work :)

    bug 
    opened by dirtybull 1
Releases(v1.9.0)
A tool to find redirection chains in multiple URLs

UnChain A tool to find redirection chains in multiple URLs Introduction UnChain automates process of finding and following `30X` redirects by extracti

RedCode Labs 79 Aug 2, 2022
A tool to filter URLs by parameter count or size

GoFilter A tool to filter URLs by parameter count or size. This tool requires unique sorted URL list. For example: cat hosts.txt | sort -u > sorted &&

Ayberk ESER 7 Sep 10, 2021
Otx - otx tool can scrap to find sensitive information and vulnerable endpoint urls.

otx Description This tool is base on AlienVault Open Threat Exchange (OTX)? and this tool can help you to extract all the urls endpoints which can be

ShaneKhant 4 Feb 9, 2022
efaceconv - Code generation tool for high performance conversion from interface{} to immutable type without allocations.

efaceconv High performance conversion from interface{} to immutable types without additional allocations This is tool for go generate and common lib (

Ivan 50 May 14, 2022
GoWrap is a command line tool for generating decorators for Go interfaces

GoWrap GoWrap is a command line tool that generates decorators for Go interface types using simple templates. With GoWrap you can easily add metrics,

Max Chechel 631 Jul 30, 2022
Perforator is a tool for recording performance metrics over subregions of a program using the Linux "perf" interface.

Perforator Perforator is a tool for recording performance metrics over subregions of a program (e.g., functions) using the Linux "perf" interface.

Zachary Yedidia 33 May 5, 2022
a tool for creating exploited media files for discord

Discord-Exploits A program for creating exploited media files for discord written in Go. Usage discord-exploits is a command line utility, meaning you

schmenn 219 Dec 29, 2021
A full-featured license tool to check and fix license headers and resolve dependencies' licenses.

SkyWalking Eyes A full-featured license tool to check and fix license headers and resolve dependencies' licenses. Usage You can use License-Eye in Git

The Apache Software Foundation 147 Aug 9, 2022
A tool and library for using structural regular expressions.

Structural Regular Expressions sregx is a package and tool for using structural regular expressions as described by Rob Pike (link).

Zachary Yedidia 37 Jul 22, 2022
TUI grep tool respect for IntelliJ

ilse TUI grep tool respect for IntelliJ Requirements ripgrep for fast grep bat for beautiful preview Features support HeadMatch(FirstMatch), WordMatch

momochi29 34 Apr 22, 2022
A tool to check problems about meta files of Unity

A tool to check problems about meta files of Unity on Git repositories, and also the tool can do limited autofix for meta files of auto-generated files.

DeNA 60 Jun 24, 2022
⚖️ A tool for transpiling C to Go.

A tool for converting C to Go. The goals of this project are: To create a generic tool that can convert C to Go. To be cross platform (linux and mac)

Elliot Chance 1.8k Aug 6, 2022
Go tool to modify struct field tags

Go tool to modify/update field tags in structs. gomodifytags makes it easy to update, add or delete the tags in a struct field. You can easily add new tags, update existing tags (such as appending a new key, i.e: db, xml, etc..) or remove existing tags

Fatih Arslan 1.8k Aug 4, 2022
🤖🤝A tool to test and analyze storage and retrieval deal capability on the Filecoin network.

Dealbot A tool to test and analyze storage and retrieval deal capability on the Filecoin network. Getting Started Clone the repo and build: git clone

Filecoin 28 Jul 26, 2022
A simple tool to auto remove unused imports in rust

rust_strip A simple tool to remove unused imports in RUST. Call Cargo build/test Process the warning logs of unused imports Then replace the involved

sundyli 6 Apr 24, 2022
A simple business indicator tool that uses a sliding window to detect whether the indicator exceeds the threshold

melon A simple business indicator tool that uses a sliding window to detect whether the indicator exceeds the threshold Usage //create the metric //th

believe 4 Jul 11, 2021
A Golang tool to whitelist ASN's based on organization name

A Golang tool to whitelist ASN's based on organization name. This works by providing a list of ASN org names. This tool uses goPacket to monitor incoming traffic, capturing the IP's and checking the IP to see if it is a part of a whitelisted ASN. If it is not, it blocks that connection and future connections using iptables.

JP 14 Jul 23, 2022
Chaos Engineering tool for introducing failure into syscalls

Syscall monkey Chaos Engineering tool for tampering with syscalls.

null 7 Jun 11, 2022
dontgo403 is a tool to bypass 40X errors.

dontgo403 is a tool to bypass 40X errors. Installation git clone https://github.com/devploit/dontgo403; cd dontgo403; go get; go build Options ./dontg

devploit 243 Jul 28, 2022