urlhunter is a recon tool that allows searching on URLs that are exposed via shortener services such as bit.ly and goo.gl.

Overview
				o  	  Utku Sen's
				 \_/\o   
				( Oo)                    \|/
				(_=-)  .===O-  ~~U~R~L~~ -O-
				/   \_/U'        hunter  /|\
				||  |_/
				\\  |    utkusen.com
				{K ||	twitter.com/utkusen

urlhunter is a recon tool that allows searching on URLs that are exposed via shortener services such as bit.ly and goo.gl. The project is written in Go.

How?

A group named URLTeam (kudos to them) are brute forcing the URL shortener services and publishing matched results on a daily basis. urlhunter downloads their collections and lets you analyze them.

Installation

From Binary

You can download the pre-built binaries from the releases page and run. For example:

tar xzvf urlhunter_0.1.0_Linux_amd64.tar.gz

./urlhunter --help

From Source

  1. Install Go on your system

  2. Run: go get -u github.com/utkusen/urlhunter

Note For The Windows Users: urlhunter uses XZ Utils which is pre-installed on Linux and macOS systems. For Windows systems, you need to download it from https://tukaani.org/xz/

Usage

urlhunter requires 3 parameters to run: -keywords, -date and -o.

For example: urlhunter -keywords keywords.txt -date 2020-11-20 -o out.txt

-keywords

You need to specify the txt file that contains keywords to search on URLs. Keywords must be written line by line. You have three different ways to specify keywords:

Single Keyword: urlhunter will search the given keyword as a substring. For example:

acme.com keyword will both match https://acme.com/blabla and https://another.com/?referrer=acme.com

Multiple Keywords: urlhunter will search the given keywords with an AND logic. Which means, a URL must include all the provided keywords. Keywords must be separated with , character. For example:

acme.com,admin will match https://acme.com/secret/adminpanel but won't match https://acme.com/somethingelse

Regex Values: urlhunter will search for the given regex value. In the keyword file, the line that contains a regular expression formula must start with regex string. The format is: regex REGEXFORMULA. For example:

regex 1\d{10} will match https://example.com/index.php?id=12938454312 but won't match https://example.com/index.php?id=abc223

-date

urlhunter downloads the archive files of the given date(s). You have three different ways to specify the date:

Latest: urlhunter will download the latest archive. -date latest

Single Date: urlhunter will download the archive of the given date. Date format is YYYY-MM-DD.

For example: -date 2020-11-20

Date Range: urlhunter will download all the archives between given start and end dates.

For example: -date 2020-11-10:2020-11-20

-o

You can specify the output file with -o parameter. For example -o out.txt

Demonstration Video

Watch the video

The Speed Problem

Archive.org throttles the speed when downloading files. Therefore, downloading an archive takes more time than usual. As a workaround, you can download the archives via Torrent and put them under the archive/ folder which is located in the same directory with the urlhunter's binary. The directory tree will look like:

|-urlhunter
|---urlhunter(binary)
|---archive
|-----urlteam_2020-11-20-11-17-04
|-----urlteam_2020-11-17-11-17-04

Example Use Cases

urlhunter might be useful for cyber intelligence and bug bounty purposes. For example:

docs.google.com/a/acme.com drive.google.com/a/acme.com keywords allow you to find public Google Docs&Drive share links of Acme company.

acme.com,password_reset_token keyword may allow you to find the working password reset tokens of acme.com

trello.com allows you to find public Trello addresses.

Thanks

Special thanks to Samet Bekmezci(@sametbekmezci) who gave me the idea of this tool.

Donation

Bitcoin Address: 1MVGYtuYAbZ1MubRFb7dxhQH35w7RQBcVe

Comments
  • urlteam_2020-12-28-03-17-02 Archive already exists!

    urlteam_2020-12-28-03-17-02 Archive already exists!

    I have download this repo. I have download xz-5.2.5-windows.zip. Install files to c:\Windows\System32. I have goo message from: > xz --help First time run was downloaded the archives to archives\urlteam_2020-12-28-03-17-02
    But then i have errors: go run main.go -date latest -keywords keywords.txt -o out.txt

    Search starting for: latest
    urlteam_2020-12-28-03-17-02 Archive already exists!
    panic: runtime error: index out of range [1] with length 1
    
    goroutine 1 [running]:
    main.searchFile(0xc0001a5380, 0x37, 0xc0001adbc0, 0x14, 0xc000010118, 0x7)
            ../urlhunter/main.go:201 +0xc5c
    main.getArchive(0xc000300000, 0x172e4, 0x1fe00, 0xc0000100a8, 0x6, 0xc0000100f0, 0xc, 0xc000010118, 0x7)
            ../urlhunter/main.go:194 +0xb2f
    main.main()
            ../urlhunter/main.go:94 +0x387
    exit status 2
    
    opened by DamianMorozov 11
  • Run urlhunter Timeout

    Run urlhunter Timeout

    when i run the command "urlhunter -keywords keywords.txt -date latest -o out.txt", it went wrong。 "panic: Get "https://archive.org/services/search/v1/scrape?debug=false&xvar=production&total_only=false&count=10000&fields=identifier%2Citem_size&q=Urlteam%20Release": dial tcp 162.125.80.6:443: i/o timeout"

    opened by susengo 4
  • Output Shortlinks and refactor searching code

    Output Shortlinks and refactor searching code

    This PR updates the searching logic to:

    1. Refactors it to make it a little simpler, reducing the branching conditions
    2. Use a Beaconfile parsing library to more robustly parse the dumpfiles
    3. Output the shortlink that mapped to a given long link.

    The new output file from this PR will now be a CSV that looks like this:

    shortlink,longlink
    

    Closes https://github.com/utkusen/urlhunter/issues/14

    opened by rzhade3 2
  • Return shortlink referring to a given longlink

    Return shortlink referring to a given longlink

    Currently, we're only searching for/ returning the long link for a given resource. However, the archives also contain information about the shortlink within the file (the format of the file is in the Beacon Link Dump format:

    b9YiMs|https://www.google.com
    

    It'd be helpful to also return the shortlink which refers to the longlink. This could be implemented by scanning the archive to find the entire line that contains a resource instead of just the specific string that was matched.

    opened by rzhade3 1
  • Make URLHunter platform agnostic

    Make URLHunter platform agnostic

    This PR changes a couple of call sites where URLHunter was assuming a certain path separator to use the platform agnostic os.PathSeparator and filepath.Join() instead. This should let Windows users also use this tool.

    It should address https://github.com/utkusen/urlhunter/issues/12

    opened by rzhade3 1
  • XZ executable file not found

    XZ executable file not found

    image

    Dear Creator, Love this tool, and can't wait to see its full capabilities. I'm faced with the following errors even when copying the files to the path folders

    opened by terminatordoink 1
  • Containerize URLHunter

    Containerize URLHunter

    Would you be open to containerizing this service, as well as possibly publishing it to DockerHub (or GitHub Container Registry)? It would really help to automate running this service.

    I did see that someone submitted this PR: https://github.com/utkusen/urlhunter/pull/2, which you closed.

    opened by rzhade3 1
  • Search Different Dump Types

    Search Different Dump Types

    Thanks for the super useful tool @utkusen 🙇

    I was trying to use this tool for archives that went further back than 2019, but it kept bugging out since there weren't any Google shortlinks from back then.

    This PR adds the ability to download dumps of arbitrary types (lsgd6, tinyurl7, etc), by downloading the Metadata XML file from the archive, which gives all of the dump types that are available in that archive.

    opened by rzhade3 1
  • Install via Go Get fails

    Install via Go Get fails

    go/src/golang.org/x/term/term_unix_linux.go:9:7: ioctlReadTermios redeclared in this block
            previous declaration at go/src/golang.org/x/term/term_unix_aix.go:9:26
    go/src/golang.org/x/term/term_unix_linux.go:10:7: ioctlWriteTermios redeclared in this block
            previous declaration at go/src/golang.org/x/term/term_unix_aix.go:10:27
    

    I'm using Ubuntu.

    opened by RiversideRocks 1
  • Several improvements

    Several improvements

    It's commonplace for CLI programs to accept short and long forms of their arguments. urlhunter did not do this, so I updated it to be easier/faster to use. Now we can use:

    • -k, --keyword
    • -d, --date
    • -o, --output

    I also corrected the documentation to match this update, and added archive/ to the .gitignore. I've standardized the output messages as well: Previously, they were scattered all over the place. I've now centralized them using crash(), warning() and info(). Each function has its own color. As a side effect, I've also made the color coding make more sense: only errors should be red, yet this program used the color red for non-critical events very often. I've corrected that.

    I have also added the -a, --archive option so we can specify a path to the archives. I didn't document this option in the README though

    opened by ItsIgnacioPortal 0
  • Flagset Refactor

    Flagset Refactor

    Hello @utkusen, thanks for this useful tool. My PR contains two issues:

    Folder Structure

    urlhunter needs a best practice folder structure. Thanks to community we have a nice guide here. https://github.com/golang-standards/project-layout

    I started with main.go but we also need to move all functions to appropriate folders.

    So for a start, I moved main.go to ./cmd/urlhunter/main.go The project can be started with go run ./cmd/urlhunter/

    Flag/Arguments Usage

    Because this is a command line tool, you need a better flag usage. I created options.go in config folder and moved your flag usage logic to there. Also changed and added variations of your flags. Now it has a better explanation now as in the screenshot. You should update your README I think. You can use this flag/options structure in all your projects. ;). Screen Shot 2020-11-22 at 22 40 30

    Hope you like it.

    opened by yakuter 0
  • Using a multiline keywords file breaks the output

    Using a multiline keywords file breaks the output

    When keywords.txt has a single word as its content:

    code
    
    C:\Users\REDACTED\Desktop>main.exe -k C:\Users\REDACTED\Documents\GitHub\urlhunter\keywords.txt -d 2022-01-01:2022-01-04 -o test.txt -a C:\Users\REDACTED\Documents\GitHub\urlhunter\archives
    
            o         Utku Sen's
             \_/\o
            ( Oo)                    \|/
            (_=-)  .===O-  ~~U~R~L~~ -O-
            /   \_/U'        hunter  /|\
            ||  |_/
            \\  |    utkusen.com
            {K ||   twitter.com/utkusen
    
    
    Search starting for: 2022-01-01
    [+]: Couldn't find an archive with that date.
    Search starting for: 2022-01-02
    [+]: urlteam_2022-01-02-11-17-02 already exists locally. Skipping download..
    [+]: Searching: "code" in C:\Users\REDACTED\Documents\GitHub\urlhunter\archives\urlteam_2022-01-02-11-17-02\goo-gl\______.txt
    ^C
    

    When keywords.txt has multiple lines as its content:

    code
    auth
    token
    
    C:\Users\REDACTED\Desktop>main.exe -k C:\Users\REDACTED\Documents\GitHub\urlhunter\keywords.txt -d 2022-01-01:2022-01-04 -o test.txt -a C:\Users\REDACTED\Documents\GitHub\urlhunter\archives
    
            o         Utku Sen's
             \_/\o
            ( Oo)                    \|/
            (_=-)  .===O-  ~~U~R~L~~ -O-
            /   \_/U'        hunter  /|\
            ||  |_/
            \\  |    utkusen.com
            {K ||   twitter.com/utkusen
    
    
    Search starting for: 2022-01-01
    [+]: Couldn't find an archive with that date.
    Search starting for: 2022-01-02
    [+]: urlteam_2022-01-02-11-17-02 already exists locally. Skipping download..
    " in C:\Users\REDACTED\Documents\GitHub\urlhunter\archives\urlteam_2022-01-02-11-17-02\goo-gl\______.txt
    ^C
    
    opened by ItsIgnacioPortal 1
Releases(v0.1.2)
Owner
Utku Sen
Utku Sen
An easy, fluent way of building URLs via chainable methods written in Go(lang)

Burl - a fluent URL builder Burl provides an easy, fluent way of building URLs via chainable methods. How to install go get github.com/mr-cryka/burl Q

Daniel Krajka 0 Feb 5, 2022
this allows you to get the real link of without get tracked bit.ly

check the real url from a url shortener (bit.ly) Also you can use it as an API example with deno const rawResponse = await fetch("https://anti-url-s

pai 10 Feb 19, 2022
A web application attack surface mapping tool. It takes in a list of urls then performs numerous probes

sigurlscann3r A web application attack surface mapping tool. It takes in a list of urls then performs numerous probes Resources Features Installation

Signed Security 10 Sep 24, 2022
A code generator that turns plain old Go services into RPC-enabled (micro)services with robust HTTP APIs.

Frodo Frodo is a code generator and runtime library that helps you write RPC-enabled (micro) services and APIs. It parses the interfaces/structs/comme

Rob Signorelli 21 Oct 25, 2022
A URL shortener using http://is.gd/ and the Go programming language (http://golang.org/)

goisgd A simple command line URL shortener using http://is.gd/. Getting the Code go get github.com/NickPresta/GoURLShortener Usage Import this librar

Nick Presta 22 Apr 6, 2022
a url shortener made using go and redis

__ | |--.--.--.----.-----. .-----.-----. | <| | | _|-- __|__| _ | _ | |__|__|_____|__| |_____|__|___

André Kelpe 76 Dec 10, 2020
A barebones URL Shortener implementation in Go using Gin and MySQL. Also features a basic frontend.

URL Shortener in Go This is a barebones URL Shortener implementation in Go using the Gin web framework and MySQL. Also features a basic frontend. Loca

Shreyas Gupta 6 Dec 22, 2021
urlsh is URL shortener application built on Go language.

GOlang URL shortener service with UI, API, Cache, Hits Counter and forwarder using postgres and redis in backend, bulma in frontend

Jitendra Adhikari 122 Nov 23, 2022
Go URL shortener

Golang Short URL To build the short link engine by Go. Development Environment: Ubuntu 20.04 Go 1.17.1 Requirement Console app, receive “URL” from any

Siong-Ui Te (戴上為) 0 Nov 4, 2021
A simple URL shortener application.

url-shortener A simple URL shortener application. Features The service supports 2 data stores (in-memory and PostgreSQL). The -db parameter is used to

Demid Fedorov 0 Dec 9, 2021
Yandex practicum go url shortener service

go-musthave-shortener-tpl Шаблон репозитория для практического трека «Go в веб-р

Dindon 0 Feb 26, 2022
Url Shortener Api For Golang

Url Shortener Api Using Go Endpoints Get and redirect to long url GET: "/:id" Cr

Lutfi Andriyanto 1 May 29, 2022
Simple URL shortener written in Golang

Alviss Introduction Simple URL shortener project, written in Golang. Setup and r

Amir Iravanimanesh 22 Aug 25, 2022
A sample url shortener app to test Keploy integration capabilities

Example URL Shortener A sample url shortener app to test Keploy integration capabilities Installation git clone https://github.com/keploy/example-url-

null 6 Oct 22, 2022
API from AnonURL website, an anonymous URL shortener

API from AnonURL website, an anonymous URL shortener

null 0 Jan 23, 2022
Cfurlshort - Url Shortener service with golang

Url Shortener This respository contains the code for my url shortener service. D

null 0 Jan 26, 2022
Goshort - Just another simple url shortener to use with curl

qnd.be Just another simple url shortener to use with curl. Kinda like 0x0.st. Ju

rune 0 Jan 29, 2022
OMUS - One More URL Shortener

OMUS OMUS - One More URL Shortener; For now - planned only API. Functions: Gener

Vladislav 2 Feb 12, 2022
Short url backend - Go musthave shortener tpl

go-musthave-shortener-tpl Шаблон репозитория для практического трека «Go в веб-р

Dmitriy 0 Feb 15, 2022