httpx is a fast and multi-purpose HTTP toolkit allows to run multiple probers using retryablehttp library, it is designed to maintain the result reliability with increased threads.

Overview

httpx

FeaturesInstallationUsageRunning httpxNotesJoin Discord

httpx is a fast and multi-purpose HTTP toolkit allow to run multiple probers using retryablehttp library, it is designed to maintain the result reliability with increased threads.

Features

httpx

  • Simple and modular code base making it easy to contribute.
  • Fast And fully configurable flags to probe mutiple elements.
  • Supports multiple HTTP based probings.
  • Smart auto fallback from https to http as default.
  • Supports hosts, URLs and CIDR as input.
  • Handles edge cases doing retries, backoffs etc for handling WAFs.

Supported probes:-

Probes Default check Probes Default check
URL true IP true
Title true CNAME true
Status Code true Raw HTTP false
Content Length true HTTP2 false
TLS Certificate true HTTP 1.1 Pipeline false
CSP Header true Virtual host false
Location Header true CDN false
Web Server true Path false
Web Socket true Ports false
Response Time true Request method false

Installation Instructions

httpx requires go1.14+ to install successfully. Run the following command to get the repo -

▶ GO111MODULE=on go get -v github.com/projectdiscovery/httpx/cmd/httpx

Usage

httpx -h

This will display help for the tool. Here are all the switches it supports.

👉 httpx help menu 👈
Usage of ./httpx:

  -H value
      Custom Header
  -allow value
      Allowlist ip/cidr
  -body string
      Request Body
  -cdn
      Check if domain's ip belongs to known CDN (akamai, cloudflare, ..)
  -cname
      Output first cname
  -content-length
      Extracts content length
  -content-type
      Extracts content-type
  -csp-probe
      Send HTTP probes on the extracted CSP domains
  -debug
      Debug mode
  -deny value
      Denylist ip/cidr
  -extract-regex string
      Extract Regex
  -fc string
      Filter status code
  -filter-regex string
      Filter Regex
  -filter-string string
      Filter String
  -fl string
      Filter content length
  -follow-host-redirects
      Only follow redirects on the same host
  -follow-redirects
      Follow Redirects
  -http-proxy string
      HTTP Proxy, eg http://127.0.0.1:8080
  -http2
      HTTP2 probe
  -include-chain
      Show Raw HTTP Chain In Output (-json only)
  -include-response
      Show Raw HTTP Response In Output (-json only)
  -ip
      Output target ip
  -json
      JSON Output
  -l string
      File containing domains
  -location
      Extracts location header
  -match-regex string
      Match Regex
  -match-string string
      Match string
  -max-response-body-size int
      Maximum response body size (default 2147483647)
  -mc string
      Match status code
  -method
      Output method
  -ml string
      Match content length
  -no-color
      No Color
  -no-fallback
      If HTTPS on port 443 is successful on default configuration, probes also port 80 for HTTP
  -o string
      File to write output to (optional)
  -path string
      Request path/file (example '/api')
  -paths string
      Command separated paths or file containing one path per line (example '/api/v1,/apiv2')
  -pipeline
      HTTP1.1 Pipeline
  -ports value
      ports range (nmap syntax: eg 1,2-10,11)
  -random-agent
      Use randomly selected HTTP User-Agent header value
  -request string
      File containing raw request
  -response-in-json
      Show Raw HTTP Response In Output (-json only) (deprecated)
  -response-time
      Output the response time
  -retries int
      Number of retries
  -silent
      Silent mode
  -sr
      Save response to file (default 'output')
  -srd string
      Save response directory (default "output")
  -stats
      Enable statistic on keypress (terminal may become unresponsive till the end)
  -status-code
      Extracts status code
  -store-chain
      Save chain to file (default 'output')
  -tech-detect
      Perform wappalyzer based technology detection
  -threads int
      Number of threads (default 50)
  -timeout int
      Timeout in seconds (default 5)
  -title
      Extracts title
  -tls-grab
      Perform TLS data grabbing
  -tls-probe
      Send HTTP probes on the extracted TLS domains
  -unsafe
      Send raw requests skipping golang normalization
  -verbose
      Verbose Mode
  -version
      Show version of httpx
  -vhost
      Check for VHOSTs
  -vhost-input
      Get a list of vhosts as input
  -web-server
      Extracts server header
  -websocket
      Prints out if the server exposes a websocket
  -x string
      Request Methods, use ALL to check all verbs ()

Running httpX

Running httpx with stdin

This will run the tool against all the hosts and subdomains in hosts.txt and returns URLs running HTTP webserver.

▶ cat hosts.txt | httpx 

    __    __  __       _  __
   / /_  / /_/ /_____ | |/ /
  / __ \/ __/ __/ __ \|   / 
 / / / / /_/ /_/ /_/ /   |  
/_/ /_/\__/\__/ .___/_/|_|   v1.0  
             /_/            

    projectdiscovery.io

[WRN] Use with caution. You are responsible for your actions
[WRN] Developers assume no liability and are not responsible for any misuse or damage.

https://mta-sts.managed.hackerone.com
https://mta-sts.hackerone.com
https://mta-sts.forwarding.hackerone.com
https://docs.hackerone.com
https://www.hackerone.com
https://resources.hackerone.com
https://api.hackerone.com
https://support.hackerone.com

Running httpx with file input

This will run the tool against all the hosts and subdomains in hosts.txt and returns URLs running HTTP webserver.

▶ httpx -l hosts.txt -silent

https://docs.hackerone.com
https://mta-sts.hackerone.com
https://mta-sts.managed.hackerone.com
https://mta-sts.forwarding.hackerone.com
https://www.hackerone.com
https://resources.hackerone.com
https://api.hackerone.com
https://support.hackerone.com

Running httpx with CIDR input

echo 173.0.84.0/24 | httpx -silent

https://173.0.84.29
https://173.0.84.43
https://173.0.84.31
https://173.0.84.44
https://173.0.84.12
https://173.0.84.4
https://173.0.84.36
https://173.0.84.45
https://173.0.84.14
https://173.0.84.25
https://173.0.84.46
https://173.0.84.24
https://173.0.84.32
https://173.0.84.9
https://173.0.84.13
https://173.0.84.6
https://173.0.84.16
https://173.0.84.34

Running httpx with subfinder

▶ subfinder -d hackerone.com | httpx -title -tech-detect -status-code -title -follow-redirects

    __    __  __       _  __
   / /_  / /_/ /_____ | |/ /
  / __ \/ __/ __/ __ \|   /
 / / / / /_/ /_/ /_/ /   |
/_/ /_/\__/\__/ .___/_/|_|
             /_/              v1.0.6

    projectdiscovery.io

Use with caution. You are responsible for your actions
Developers assume no liability and are not responsible for any misuse or damage.
https://mta-sts.managed.hackerone.com [404] [Page not found · GitHub Pages] [Varnish,GitHub Pages,Ruby on Rails]
https://mta-sts.hackerone.com [404] [Page not found · GitHub Pages] [Varnish,GitHub Pages,Ruby on Rails]
https://mta-sts.forwarding.hackerone.com [404] [Page not found · GitHub Pages] [GitHub Pages,Ruby on Rails,Varnish]
https://docs.hackerone.com [200] [HackerOne Platform Documentation] [Ruby on Rails,jsDelivr,Gatsby,React,webpack,Varnish,GitHub Pages]
https://support.hackerone.com [301,302,301,200] [HackerOne] [Cloudflare,Ruby on Rails,Ruby]
https://resources.hackerone.com [301,301,404] [Sorry, no Folders found.]

📋 Notes

  • As default, httpx checks for HTTPS probe and fall-back to HTTP only if HTTPS is not reachable.
  • For printing both HTTP/HTTPS results, no-fallback flag can be used.
  • Custom scheme for ports can be defined, for example -ports http:443,http:80,https:8443
  • vhost, http2, pipeline, ports, csp-probe, tls-probe and path are unique flag with different probes.
  • Unique flags should be used for specific use cases instead of running them as default with other flags.
  • When using json flag, all the information (default probes) included in the JSON output.

Thanks

httpx is made with 🖤 by the projectdiscovery team. Community contributions have made the project what it is. See the Thanks.md file for more details. Do also check out these similar awesome projects that may fit in your workflow:

Probing feature is inspired by @tomnomnom/httprobe work ❤️

Issues
  • [BUG] CIDR in combination with -ports

    [BUG] CIDR in combination with -ports

    To replicate the issue of missing valid host with CIDR input

    with prips:-

    > prips 1.1.1.0/24 | httpx -title -content-length -status-code -ports 80,443 -silent | grep 1.1.1.1:80
    
    http://1.1.1.1:80 [301] [186] [301 Moved Permanently]
    

    with httpx internal CIDR handler: -

    > echo 1.1.1.0/24 | httpx -title -content-length -status-code -ports 80,443 -silent | grep 1.1.1.1:80
    

    To replicate the duplication issue with CIDR input

    > echo 1.1.1.0/24 | httpx -title -content-length -status-code -ports 80,443 -silent | sort | grep http://1.1.1.24:80
    http://1.1.1.24:80 [403] [16] []
    
    echo 1.1.1.0/24 | httpx -title -content-length -status-code -ports 80,443 -silent | sort | grep http://1.1.1.24:80
    
    http://1.1.1.24:80 [403] [16] []
    http://1.1.1.24:80 [403] [16] []
    http://1.1.1.24:80 [403] [16] []
    

    I have opened ports 80 and 443 on 192.168.8.1

    echo 192.168.8.1 | ./httpx  -title -content-length -status-code -ports 80,443
    
        __    __  __       _  __
       / /_  / /_/ /_____ | |/ /
      / __ \/ __/ __/ __ \|   / 
     / / / / /_/ /_/ /_/ /   |  
    /_/ /_/\__/\__/ .___/_/|_|  
                 /_/              v1           
    
    		projectdiscovery.io
    
    [WRN] Use with caution. You are responsible for your actions
    [WRN] Developers assume no liability and are not responsible for any misuse or damage.
    https://192.168.8.1:443 [307] [13] []
    http://192.168.8.1:80 [307] [13] []
    

    But when I run CIDR scan, the httpx does not return any open port on 192.168.8.1

    echo 192.168.8.0/24 | ./httpx  -title -content-length -status-code -ports 80,443
    
        __    __  __       _  __
       / /_  / /_/ /_____ | |/ /
      / __ \/ __/ __/ __ \|   / 
     / / / / /_/ /_/ /_/ /   |  
    /_/ /_/\__/\__/ .___/_/|_|  
                 /_/              v1           
    
    		projectdiscovery.io
    
    [WRN] Use with caution. You are responsible for your actions
    [WRN] Developers assume no liability and are not responsible for any misuse or damage.
    
    Status: Completed 
    opened by marcopolo157 16
  • Add probe-all-ips feature

    Add probe-all-ips feature

    Added flag to support probe all ips #245.

    Example output: For default output, I just implicitly enable IP address output:

    $ echo hackerone.com | ./httpx -sc -scan-all-ips
    https://hackerone.com [302] [104.16.99.52]
    https://hackerone.com [302] [104.16.100.52]
    

    JSON (as described in the original issue):

    {
      "timestamp": "2021-10-23T19:10:52.740432132+03:00",
      "scheme": "https",
      "port": "443",
      "path": "/",
      "body-sha256": "bf697861898d0a7fabf4886f0eb238a440f45622e062ef40ac266e5575796347",
      "header-sha256": "883abfaf1ec8e7c9270389ceb45e0c4b8a16264c933174b93b14766b6f66c85c",
      "a": [
        "104.16.100.52",
        "104.16.99.52"
      ],
      "url": "https://hackerone.com:443",
      "input": "hackerone.com",
      "location": "https://www.hackerone.com/",
      "webserver": "cloudflare",
      "content-type": "text/html",
      "method": "GET",
      "host": "104.16.100.52",
      "content-length": 92,
      "status-code": 302,
      "csp": {
        "domains": [
          "a5s.hackerone-ext-content.com",
          "b5s.hackerone-ext-content.com",
          "errors.hackerone.net",
          "profile-photos.hackerone-user-content.com",
          "www.youtube-nocookie.com",
          "www.google-analytics.com",
          "cover-photos.hackerone-user-content.com",
          "hackathon-photos.hackerone-user-content.com",
          "hackerone-us-west-2-production-attachments.s3.us-west-2.amazonaws.com",
          "https://errors.hackerone.net/api/30/csp-report/?sentry_key=374aea95847f4040a69f9c8d49a3a59d"
        ]
      },
      "response-time": "963.997528ms",
      "failed": false
    }
    {
      "timestamp": "2021-10-23T19:10:52.774901606+03:00",
      "scheme": "https",
      "port": "443",
      "path": "/",
      "body-sha256": "bf697861898d0a7fabf4886f0eb238a440f45622e062ef40ac266e5575796347",
      "header-sha256": "e129249144cc96d0bcf0273eb03e42f002169966d76179ffbd5d359264e5ea35",
      "a": [
        "104.16.100.52",
        "104.16.99.52"
      ],
      "url": "https://hackerone.com:443",
      "input": "hackerone.com",
      "location": "https://www.hackerone.com/",
      "webserver": "cloudflare",
      "content-type": "text/html",
      "method": "GET",
      "host": "104.16.99.52",
      "content-length": 92,
      "status-code": 302,
      "csp": {
        "domains": [
          "profile-photos.hackerone-user-content.com",
          "https://errors.hackerone.net/api/30/csp-report/?sentry_key=374aea95847f4040a69f9c8d49a3a59d",
          "www.youtube-nocookie.com",
          "b5s.hackerone-ext-content.com",
          "www.google-analytics.com",
          "errors.hackerone.net",
          "cover-photos.hackerone-user-content.com",
          "hackathon-photos.hackerone-user-content.com",
          "hackerone-us-west-2-production-attachments.s3.us-west-2.amazonaws.com",
          "a5s.hackerone-ext-content.com"
        ]
      },
      "response-time": "1.005377776s",
      "failed": false
    }
    
    Status: Completed Type: Enhancement 
    opened by zerodivisi0n 14
  • Add default port for URL to solve automatic use of HTTP

    Add default port for URL to solve automatic use of HTTP

    The results are as expected

    echo "www.baidu.com"|httpx   -title -content-length -status-code #auto use https
    https://www.baidu.com [302] [161] [302 Found]
    
    echo "http://www.baidu.com:80"|httpx   -title -content-length -status-code #use  http
    http://www.baidu.com:80 [200] [300813] [百度一下,你就知道]
    

    Results did not meet expectations

    echo "http://www.baidu.com"|httpx   -title -content-length -status-code #auto jump https
    https://www.baidu.com [302] [161] [302 Found]
    

    Modified program

    echo "http://www.baidu.com"|httpx   -title -content-length -status-code #use http
    http://www.baidu.com:80 [200] [300813] [百度一下,你就知道]
    
    Status: Completed Type: Enhancement 
    opened by Becivells 10
  • cname probe not work

    cname probe not work

    In pr #108 seems that cname probe has been added, but when I try it locally it seems that not work,

    Use the test case in the original pr:

    https://www.hackerone.com
    https://api.hackerone.com
    https://mta-sts.hackerone.com
    https://mta-sts.managed.hackerone.com
    https://support.hackerone.com
    https://mta-sts.forwarding.hackerone.com
    https://docs.hackerone.com
    https://resources.hackerone.com
    
    ./httpx -l domains.txt -cname
    
        __    __  __       _  __
       / /_  / /_/ /_____ | |/ /
      / __ \/ __/ __/ __ \|   /
     / / / / /_/ /_/ /_/ /   |
    /_/ /_/\__/\__/ .___/_/|_|
                 /_/              v1.0.5
    
    		projectdiscovery.io
    
    Use with caution. You are responsible for your actions
    Developers assume no liability and are not responsible for any misuse or damage.
    https://mta-sts.forwarding.hackerone.com
    http://mta-sts.managed.hackerone.com
    http://mta-sts.hackerone.com
    http://api.hackerone.com
    http://docs.hackerone.com
    http://www.hackerone.com
    http://resources.hackerone.com
    http://support.hackerone.com
    

    When I dig into source code I find that in internal/runner/runner.go:

    dnsData, err := hp.Dialer.GetDNSData(domain)
    

    and in fast dialer GetDNSData function:

    		data, err = d.dnsclient.Resolve(hostname)
    		if err != nil && d.options.EnableFallback {
    			data, err = d.dnsclient.ResolveWithSyscall(hostname)
    		}
    		if err != nil {
    			return nil, err
    		}
    		if data == nil {
    			return nil, errors.New("could not resolve host")
    		}
    		b, _ := data.Marshal()
    		err = d.hm.Set(hostname, b)
    		if err != nil {
    			return nil, err
    		}
    		return data, nil
    

    It call retryabledns's Resolve function which only query for the A record:

    func (c *Client) Resolve(host string) (*DNSData, error) {
    	return c.Query(host, dns.TypeA)
    }
    

    So I think it is the root cause? I think the fastdialer need to add an function for A|CNAME type record.

    Status: Completed Type: Bug 
    opened by iflody 10
  • Json parsing error while validating the json output in JSONLint

    Json parsing error while validating the json output in JSONLint

    while beautifying the output of json in any online website i encounter an error saying

    Error: Parse error on line 22: ...": "500.794337ms"} { "ips": ["13.227.1 ----------------------^ Expecting 'EOF', '}', ',', ']', got '{'

    Type: Question 
    opened by sA74n 10
  • Add httprobe like prefer-https option to avoid duplicacy

    Add httprobe like prefer-https option to avoid duplicacy

    If a server is listening on both 80 and 443 ports and the response is identical, there should be an option to avoid saving http://target.com in output file. httprobe from tomnomnom has the cool feature. You may check it out for better understanding of what I am saying.

    Type: Question 
    opened by AdmiralGaust 10
  • Installation issue with go modules

    Installation issue with go modules

    Description

    Hello Team

    Thank you very much for making this tool for us it's been very helpful for us as I was poking around some features and flag I noticed that there is an issue with the mc flag it actually renders an error when you try to match a 200 or 302 status code

    Test

    Let's say I want to check the js file that att.com has in the archive I came with this oneliner echo "att.com" | ~/go/bin/gau | grep '\.js$' | httpx -mc 200

    Expected behavior

    Checking results that has 200 status code and display

    Issue

    issue

    What I tried

    I tried to copy paste the same command you are using in the example section  httpx -status-code -mc 200,302

    Ended up with the same message issue2

    Config

    Ubuntu 18.04 Golang1.14 GO111MODULE=on httpx version 0.0.7

    Priority: High Type: Bug 
    opened by Sicks3c 10
  • add TLS SNI and let HTTPS have priority

    add TLS SNI and let HTTPS have priority

    it's possible to let users specify TLS SNI like this one https://github.com/ffuf/ffuf/issues/440 and make HTTPS have the priority.

    Add a new -sni flag that will be used when establishing TLS connections:

    echo hackerone.com | httpx -sni www.hackerone.com
    
    Priority: Low Status: Completed Type: Enhancement 
    opened by 0xAwali 9
  • panic: runtime error: invalid memory address or nil pointer dereference

    panic: runtime error: invalid memory address or nil pointer dereference

    httpx version:[INF] Current Version: v1.1.4

    Current Behavior:httpx -l target.txt -path "///////../../../../../../etc/passwd"

    Expected Behavior:

    image

    Steps To Reproduce:

    Run 'httpx -l target.txt -path "///////../../../../../../etc/passwd"' See error image

    panic: runtime error: invalid memory address or nil pointer dereference
    [signal SIGSEGV: segmentation violation code=0x1 addr=0x58 pc=0x9002d3]
    
    goroutine 14 [running]:
    github.com/projectdiscovery/httpx/runner.(*Runner).analyze(_, _, {_, _}, {_, _}, {_, _}, {0xc000eea780, 0x1b}, ...)
            /home/runner/work/httpx/httpx/runner/runner.go:874 +0xe93
    github.com/projectdiscovery/httpx/runner.(*Runner).process.func1({0xc000eea780, 0x1b}, {0x9fe1ef, 0x3}, {0xa042ab, 0xa})
            /home/runner/work/httpx/httpx/runner/runner.go:651 +0x125
    created by github.com/projectdiscovery/httpx/runner.(*Runner).process
            /home/runner/work/httpx/httpx/runner/runner.go:649 +0x965
    

    Anything else:

    Priority: Medium Status: Completed Type: Bug 
    opened by z7701858 8
  • httpx specifices :80 and :443 in the Host header, causing false negatives

    httpx specifices :80 and :443 in the Host header, causing false negatives

    httpx version:

    Current Version: v1.1.4

    Current Behavior:

    httpx sends http/https probes with the port specified within the http Host header even for ports :80 and :443 which causes false negatives when server / reverse proxy targeted validated a strict vhost is used, without port information.

    Expected Behavior:

    Implement one of:

    • Probe with AND without :PORT for the most common http ports such as 80, 443, 8080, 8440
    • Add an option to have all specified ports scanned both with vhost:port and vhost within the Host http header

    Steps To Reproduce:

    # curl -k https://127.0.0.1/
    # ncat -lnvp 443 --ssl
    
    Ncat: Version 7.80 ( https://nmap.org/ncat )
    Ncat: Generating a temporary 2048-bit RSA key. Use --ssl-key and --ssl-cert to use a permanent one.
    Ncat: SHA-1 fingerprint: 44DD 4CEF 4E03 1541 B6FB 79EC 61C8 99CD AFE7 C098
    Ncat: Listening on :::443
    Ncat: Listening on 0.0.0.0:443
    Ncat: Connection from 127.0.0.1.
    Ncat: Connection from 127.0.0.1:48334.
    GET / HTTP/1.1
    Host: 127.0.0.1
    User-Agent: curl/7.68.0
    Accept: */*
    
    # echo 127.0.0.1 | httpx -ports 443 -no-fallback-scheme (same behavior without -no-fallback-scheme)
    # ncat -lnvp 443 --ssl
    Ncat: Version 7.80 ( https://nmap.org/ncat )
    Ncat: Generating a temporary 2048-bit RSA key. Use --ssl-key and --ssl-cert to use a permanent one.
    Ncat: SHA-1 fingerprint: 925C B498 3CF6 4D33 EF6C FDB4 3820 92FF BADB 001B
    Ncat: Listening on :::443
    Ncat: Listening on 0.0.0.0:443
    Ncat: Connection from 127.0.0.1.
    Ncat: Connection from 127.0.0.1:48336.
    GET / HTTP/1.1
    Host: 127.0.0.1:443
    User-Agent: Mozilla/5.0 (Macintosh; Intel Mac OS X 10_13_4) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/74.0.3729.169 Safari/537.36
    Accept-Charset: utf-8
    Accept-Encoding: gzip
    Connection: close
    

    Anything else:

    I've had something like 10% false negative over 20k+ hosts, so I think this is worth patching :rose: Thanks so much for all the awesome tools and automation mindset! ;)

    Priority: Medium Status: Completed Type: Enhancement 
    opened by laluka 8
  • http-proxy still ignored, user-agent trimming issue

    http-proxy still ignored, user-agent trimming issue

    Hello,

    I'm using httpx v0.8, the command: cat urls.txt | httpx -http-proxy http://*.*.*.*:3128 -mc 200,302 -H 'User-Agent: Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:79.0) Gecko/20100101 Firefox/79.0' -H 'Cookie: ...' -o result.txt

    For some reason httpx still fetches given URL list with my default IP which results in 403 errors from the server. What is interesting is that when I use ffuf with the same proxy setting and those 2 same headers, everything works as expected.

    I tried to check the source code of cmd/httpx.go and find the issue myself, but I don't have go knowledge at all so after a few comparisons of how it's done in ffuf and in httpx I didn't spot the reason why proxy still ignored in httpx.

    Status: Completed Type: Bug 
    opened by Haones 8
  • Extend the scope or add scope control for matchers and extractors

    Extend the scope or add scope control for matchers and extractors

    Please describe your feature request:

    Request is to have a flag that extracts the www-authenticate type from headers.

    Describe the use case of this feature:

    In an effort to extract authentication methods in bulk, I frequently collect headers and parse out the www-authenticate header. It would be great if this extracted type (basic, digest Bearer) and any additional info (realm etc).

    Type: Enhancement 
    opened by fail-open 1
  • https randomly failing with proxies

    https randomly failing with proxies

    httpx version:

    master/main and dev

    Current Behavior:

    https urls are failing silently in certain cases while using proxy

    Expected Behavior:

    https protocol working correctly with proxies

    Steps To Reproduce:

    $ echo https://192.168.1.1 | go run . -proxy http://localhost:8080 -v
    ...
    [DBG] Failed 'https://192.168.1.1': remote error: tls: user canceled
    
    Screenshot 2022-08-09 at 11 25 47 Type: Bug Investigation 
    opened by Mzack9999 0
  • out of memory error

    out of memory error

    httpx version:

    v1.2.3, also tested with the latest one v1.2.4

    Current Behavior:

    I'm trying to scan working http/s servers from huge list and after some time "out of memory" error is thrown, My server have 64gb of ram and it's not nearly using it fully while active scanning.

    Error

    fatal error: runtime: out of memory
    
    runtime stack:
    runtime.throw({0xb7eae2?, 0x400000000?})
            /usr/local/go/src/runtime/panic.go:992 +0x71
    runtime.sysMap(0xc4d5000000, 0x7f01cfffed38?, 0x7f01cfffeda0?)
            /usr/local/go/src/runtime/mem_linux.go:189 +0x11b
    runtime.(*mheap).grow(0x11c4e80, 0x200000?)
            /usr/local/go/src/runtime/mheap.go:1404 +0x225
    runtime.(*mheap).allocSpan(0x11c4e80, 0x200000, 0x0, 0x1)
            /usr/local/go/src/runtime/mheap.go:1170 +0x171
    runtime.(*mheap).alloc.func1()
            /usr/local/go/src/runtime/mheap.go:912 +0x65
    runtime.systemstack()
            /usr/local/go/src/runtime/asm_amd64.s:469 +0x49
    
    goroutine 320007664 [running]:
    runtime.systemstack_switch()
            /usr/local/go/src/runtime/asm_amd64.s:436 fp=0xc00e959df8 sp=0xc00e959df0 pc=0x4637a0
    runtime.(*mheap).alloc(0x400000000?, 0x200000?, 0x80?)
            /usr/local/go/src/runtime/mheap.go:906 +0x65 fp=0xc00e959e40 sp=0xc00e959df8 pc=0x4281a5
    runtime.(*mcache).allocLarge(0xc00c102600?, 0x3fffffe00, 0x1)
            /usr/local/go/src/runtime/mcache.go:213 +0x85 fp=0xc00e959e90 sp=0xc00e959e40 pc=0x4186a5
    runtime.mallocgc(0x3fffffe00, 0xa72360, 0x1)
            /usr/local/go/src/runtime/malloc.go:1096 +0x5a5 fp=0xc00e959f08 sp=0xc00e959e90 pc=0x40ec65
    runtime.makeslice(0xc48e16a000?, 0x7e00?, 0x7e00?)
            /usr/local/go/src/runtime/slice.go:103 +0x52 fp=0xc00e959f30 sp=0xc00e959f08 pc=0x44c9f2
    bytes.makeSlice(0x3fffffe00)
            /usr/local/go/src/bytes/buffer.go:229 +0x65 fp=0xc00e959f80 sp=0xc00e959f30 pc=0x4f0345
    bytes.(*Buffer).grow(0xc0014e67b0, 0x200)
            /usr/local/go/src/bytes/buffer.go:142 +0x11f fp=0xc00e959fc8 sp=0xc00e959f80 pc=0x4efc9f
    bytes.(*Buffer).ReadFrom(0xc0014e67b0, {0x7f02101f1820, 0xc013a2e080})
            /usr/local/go/src/bytes/buffer.go:202 +0x45 fp=0xc00e95a020 sp=0xc00e959fc8 pc=0x4f0125
    net/http/httputil.drainBody({0xccf830, 0xc013a2e080})
            /usr/local/go/src/net/http/httputil/dump.go:31 +0x7e fp=0xc00e95a088 sp=0xc00e95a020 pc=0x9d0f1e
    net/http/httputil.DumpResponse(0xc00fcd3950, 0x1)
            /usr/local/go/src/net/http/httputil/dump.go:325 +0x95 fp=0xc00e95a0e0 sp=0xc00e95a088 pc=0x9d2bd5
    github.com/projectdiscovery/httputil.DumpResponseHeadersAndRaw(0xc00fcd3950)
            /root/work/pkg/mod/github.com/projectdiscovery/[email protected]/httputil.go:45 +0x10c fp=0xc00e95a1f8 sp=0xc00e95a0e0 pc=0x9d3c6c
    github.com/projectdiscovery/httpx/common/httpx.(*HTTPX).Do(0xc0000ae370, 0xc0141e5530, {{0x0?, 0xb50268?}})
            /root/work/pkg/mod/github.com/projectdiscovery/[email protected]/common/httpx/httpx.go:185 +0x23d fp=0xc00e95a3c8 sp=0xc00e95a1f8 pc=0x9e05fd
    github.com/projectdiscovery/httpx/runner.(*Runner).analyze(_, _, {_, _}, {{0xc00003f7ef, 0x10}, {0x0, 0x0}, {0x0, 0x0}}, ...)
            /root/work/pkg/mod/github.com/projectdiscovery/[email protected]/runner/runner.go:974 +0xc65 fp=0xc00e95b718 sp=0xc00e95a3c8 pc=0xa1a9c5
    github.com/projectdiscovery/httpx/runner.(*Runner).process.func1({{0xc00003f7ef, 0x10}, {0x0, 0x0}, {0x0, 0x0}}, {0xb50268?, 0x3?}, {0xc01583f410, 0x4})
            /root/work/pkg/mod/github.com/projectdiscovery/[email protected]/runner/runner.go:766 +0x13b fp=0xc00e95bf80 sp=0xc00e95b718 pc=0xa1923b
    github.com/projectdiscovery/httpx/runner.(*Runner).process.func3()
            /root/work/pkg/mod/github.com/projectdiscovery/[email protected]/runner/runner.go:792 +0x5c fp=0xc00e95bfe0 sp=0xc00e95bf80 pc=0xa190dc
    runtime.goexit()
            /usr/local/go/src/runtime/asm_amd64.s:1571 +0x1 fp=0xc00e95bfe8 sp=0xc00e95bfe0 pc=0x4659c1
    created by github.com/projectdiscovery/httpx/runner.(*Runner).process
            /root/work/pkg/mod/github.com/projectdiscovery/[email protected]/runner/runner.go:764 +0xa3f
    
    Type: Bug Investigation 
    opened by paul1339 1
  • Favicon flag improvement

    Favicon flag improvement

    Please describe your feature request:

    Currently, -favicon flag needs to be used separately in the workflow to calculate favicon hashes of hosts to probe, as such httpx needs to be executed twice and separately, and that can be optionally achieved in the same execution.

    This changes nothing in terms of favicon calculation or logic execution but is more of an improvement merging results of 2 execution into one to make the feature more practical + usable.

    Current behavior:

    echo hackerone.com | httpx -silent -sc -ct
    https://hackerone.com [302] [text/html]
    
    echo hackerone.com | httpx -silent -favicon
    https://hackerone.com/favicon.ico [595148549]
    

    Updated behavior:

    echo hackerone.com | httpx -silent -sc -ct -favicon
    https://hackerone.com [302] [text/html] [595148549]
    

    Describe the use case of this feature:

    More streamlined workflow for users who are also interested in collecting favicon information along with host probing.

    Priority: Medium Status: Completed Type: Enhancement 
    opened by ehsandeep 0
  • SNI override is being in cross domain redirects

    SNI override is being in cross domain redirects

    https://github.com/projectdiscovery/nuclei/issues/2284

    echo https://redirect-localhost.free.beeceptor.com| httpx -sni redirect-localhost.free.beeceptor.com -fr
    
    Priority: Low Status: On Hold Type: Bug 
    opened by brenocss 1
Releases(v1.2.4)
Owner
ProjectDiscovery
Security Through Intelligent Automation
ProjectDiscovery
Simple HTTP/HTTPS proxy - designed to be distributed as a self-contained binary that can be dropped in anywhere and run.

Simple Proxy This is a simple HTTP/HTTPS proxy - designed to be distributed as a self-contained binary that can be dropped in anywhere and run. Code b

Jamie Thompson 13 May 9, 2022
kcp is a prototype of a Kubernetes API server that is not a Kubernetes cluster - a place to create, update, and maintain Kube-like APis with controllers above or without clusters.

kcp is a minimal Kubernetes API server How minimal exactly? kcp doesn't know about Pods or Nodes, let alone Deployments, Services, LoadBalancers, etc.

Prototype of Future Kubernetes Ideas 1.6k Aug 9, 2022
Lightweight rest api that allows users to run Powershell commands over HTTP.

Powershell Proxy Lightweight rest api that allows users to run Powershell commands over HTTP. Requests require a valid JWT and responses are returned

Thomas Sampson 4 Mar 18, 2022
GoScan is a port-scanner made entirely in Go-lang. The purpose of the tool is to be fast, dynamic and simple so that a professional in the CyberSecurity area can make an optimized list of ports

?? GoScan GoScan is a port-scanner made entirely in Go-lang. The purpose of the tool is to be fast, dynamic and simple so that a professional in the C

Ottoni 4 Jul 19, 2022
mt-multiserver-proxy is a reverse proxy designed for linking multiple Minetest servers together

mt-multiserver-proxy mt-multiserver-proxy is a reverse proxy designed for linking multiple Minetest servers together. It is the successor to multiserv

null 11 Jul 31, 2022
Use Consul to do service discovery, use gRPC +kafka to do message produce and consume. Use redis to store result.

目录 gRPC/consul/kafka简介 gRPC+kafka的Demo gRPC+kafka整体示意图 限流器 基于redis计数器生成唯一ID kafka生产消费 kafka生产消费示意图 本文kafka生产消费过程 基于pprof的性能分析Demo 使用pprof统计CPU/HEAP数据的

null 43 Jul 9, 2022
🍔 Product-storage service, work on gRPC. Client sends the URL to download products, and requests the result.

?? Product-storage service, work on gRPC. Client sends the URL to download products, and requests the result. The server transfer request to a third-party resource for .csv-file uploading and saves the products to own database.

Pavel V 9 Dec 16, 2021
A simulation to see what's the result among normal people、rich-second generation、hard-working people

A simulation to see what's the result of competion among normal people、rich-second generation and hard-working people. 假设: 一个社会集体中有部分富二代,部分努力的人,多数是普通人

Myrainhua 0 Feb 20, 2022
Undirect - Get redirect link result with golang

Undirect Usage Request http://localhost:5000/?url=https%3A%2F%2Fokjk.co%2FSqOYfk

Nya Candy 1 Jan 12, 2022
GraspleResultNotifier - Grasple result checker with golang

Grasple result checker Environment variables TELEGRAM_GRASPLE_BOT_TOKEN: The tok

Kees Blok 1 Jan 25, 2022
Mackerel plugin to post bigquery's query result

mackerel-plugin-bigquery-query-result-importer Synopsis % mackerel-plugin-bigque

Yasuhisa Yoshida 0 Feb 5, 2022
Chat app that allows you to group multiple channels into one view.

hashchat Backend service Getting Started Essential steps to get your backend service deployed A helloworld example has been shipped with the template

Commit App Playground 0 Dec 13, 2021
Socketlogger allows multiple processes to log to a shared log file between processes.

socketlogger socketlogger is a language agnostic logging utility designed to allow mulitple applications, running on one or multiple computers, to wri

Ryan Johnson 1 Dec 7, 2021
TFTP and HTTP server specifically designed to serve iPXE ROMs and scripts.

pixie TFTP and HTTP server specifically designed to serve iPXE ROMs and scripts. pixie comes embedded with the following ROMs provided by the iPXE pro

Adrian L Lange 14 Jun 30, 2022
Echo-server - An HTTP echo server designed for testing applications and proxies

echo-server An HTTP echo server designed for testing applications and proxies. R

Erik Cavalcanti 5 Jul 14, 2022
Hetty is an HTTP toolkit for security research.

Hetty is an HTTP toolkit for security research. It aims to become an open source alternative to commercial software like Burp Suite Pro, with powerful

David Stotijn 4.6k Aug 6, 2022
Fast HTTP package for Go. Tuned for high performance. Zero memory allocations in hot paths. Up to 10x faster than net/http

fasthttp Fast HTTP implementation for Go. Currently fasthttp is successfully used by VertaMedia in a production serving up to 200K rps from more than

Aliaksandr Valialkin 18.2k Aug 15, 2022
Guilherme Biff Zarelli 3 Jun 6, 2022
A library designed for hosting Minecraft: Java Edition listeners.

expresso A library designed for hosting Minecraft: Java Edition listeners. Features Hosting listeners. All handshake, status, and login state packets.

Tal 3 Jul 29, 2022