Describe the bug If you provide url with port (not every web app stay on 80 and 443 port =) ) - cariddi cant parse it

like as

echo http://ya.ru:80 | cariddi
The URL provided is not built in a proper way: http://ya.ru:80

Describe the bug After a while we (for some reason) get a slice out of bounds errors

goroutine 25981 [running]:
github.com/edoardottt/cariddi/pkg/crawler.New.func15(0xc072bd1380)
	/opt/hacking/repository/cariddi/pkg/crawler/colly.go:215 +0x950
github.com/gocolly/colly.(*Collector).handleOnResponse(0xc072bd1380?, 0xc072bd1380)
	/home/cyb3rjerry/go/pkg/mod/github.com/gocolly/[email protected]/colly.go:936 +0x1be
github.com/gocolly/colly.(*Collector).fetch(0xc00018d520, {0x0?, 0x0?}, {0x8c1eaa, 0x3}, 0x1, {0x0?, 0x0}, 0x0?, 0xc0304a96e0, ...)
	/home/cyb3rjerry/go/pkg/mod/github.com/gocolly/[email protected]/colly.go:621 +0x69b
created by github.com/gocolly/colly.(*Collector).scrape
	/home/cyb3rjerry/go/pkg/mod/github.com/gocolly/[email protected]/colly.go:532 +0x645

To Reproduce Steps to reproduce the behavior:

1. run echo "https://mapbox.com/" | cariddi -s -intensive

Desktop (please complete the following information): Linux cyb3rjerry 5.19.0-23-generic 24-Ubuntu SMP PREEMPT_DYNAMIC Fri Oct 14 15:39:57 UTC 2022 x86_64 x86_64 x86_64 GNU/Linux

Hey there, this fix should solve issue #79 ! To solve it I'm using (*Collector).OnRequest method to print the URL that is actively being visited. Colly's doc also shows this method being used for this specific use case so I'd be tempted to say this is the best way of doing this.

I've also slipped in some simplifications of some for .. range loops that were proposed by go-staticcheck

Let me know what you think of it!

internal/unsafeheader

compile: version "go1.17.6" does not match go tool version "go1.18.2"

internal/cpu

compile: version "go1.17.6" does not match go tool version "go1.18.2"

internal/abi

how to fix it

thx.

Describe the bug Panic while compiling some regex during a find the secrets (-s) run. It also happens with the -e flag as well.

panic: regexp: Compile(`*`): error parsing regexp: missing argument to repetition operator: `*`

goroutine 1 [running]:
regexp.MustCompile(0x14fb20a, 0x1, 0x0)
	/usr/local/Cellar/go/1.16.4/libexec/src/regexp/regexp.go:311 +0x157
github.com/edoardottt/cariddi/crawler.Crawler(0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x2, 0x14, 0x1, 0x0, ...)
	/Users/sean/repos/cariddi/crawler/colly.go:54 +0x1ce
main.main()
	/Users/sean/repos/cariddi/main.go:91 +0x3cf

To Reproduce Steps to reproduce the behavior:

1. Create urls file with a valid url in it
2. Run the following command: cat urls|./cariddi -d 2 -s
3. See stack trace shortly after launching

Expected behavior Cariddi should process the provided site and find any/all secrets

Desktop (please complete the following information):

• OS: Mac OS
• Version: 11.4 (Bug Sur)

Describe the bug Hey there, so after running the scanner on an endpoint, I noticed that out of the blue I get a domain formatted in a bad way error. However, even when enabling -debug, I get no additional info as to what URL has killed the scan. Is this intended?

To Reproduce Steps to reproduce the behavior:

1. Run echo "https://xxxxxx.com" | cariddi -s -sf regex.txt -intensive -ot results.txt
2. Wait
3. Observe domain formatted in a bad way error

Expected behavior Would it not be better to simply skip that URL rather than killing the scan via os.Exit(1)?

Screenshots

Desktop (please complete the following information):

• OS: Linux XXXXXXXX 5.19.0-23-generic 24-Ubuntu SMP PREEMPT_DYNAMIC Fri Oct 14 15:39:57 UTC 2022 x86_64 x86_64 x86_64 GNU/Linux
• Version: Cariddi v1.1.9

Additional context

Let me know if you'd appreciate a hand in any way!

The URL provided is not built in a proper way: www.edoardoottavianelli.it

I tried with the sample I saw on the video but still not working. I have all packages and libraries, GO included so I don't know why is not working. Have a nice Sunday!

*EDIT It works using powershell but not terminal. Also some endpoints don't work, maybe because they are private e.g.: https://api.nike.com

Describe the bug When crawling https://mapbox.com/ we notice that "docs.mapbox.com" gets crawled

To Reproduce Steps to reproduce the behavior:

1. Run echo "https://mapbox.com/" | cariddi -s -intensive -i docs
2. Look at output

Expected behavior docs.* not to be crawler

Desktop (please complete the following information):

• OS: Linux WKS-001772 5.19.0-23-generic 24-Ubuntu SMP PREEMPT_DYNAMIC Fri Oct 14 15:39:57 UTC 2022 x86_64 x86_64 x86_64 GNU/Linux
• Browser [e.g. chrome, safari]
• Version: v1.1.9

Hello! 👋 By this PR I would like to start adding tests to the project, if you don't mind :)

utils package 0 -> 17% 🎉 ok github.com/edoardottt/cariddi/utils 0.209s coverage: 17.6% of statements

This if a draft PR, just for example, I'll be updating it.

And some questions

1. Do you want to use any libraries for tests? like https://github.com/stretchr/testify
2. Don't you mind if I make several changes within this PR? as in the first commit - there are some changes for reducing allocations, and simplifying code

Bug description When cariddi runs without -intensive flag set it works fine, all the urls crawled on the target(s) belong to the input domain(s).
This is the method used for normal behaviour (line 108). As you can notice on line 108 there is a colly option to restrict the crawler to crawl only urls belonging to the inputted domain(s).

Instead, when you use the -intensive flag set, this is the method used: from line 114 to 119.
The problem with this method is that there could be some false positives, like facebook.com?q=c.target.com.
I'm trying to figure out how pick only target urls even if cariddi is running in intensive mode.

Just to be clear:

• The normal behaviour (without -intensive) is that if you pass as input target.com, cariddi will crawl only (and strictly) urls belonging to target.com.
• If the -intensive mode is set, cariddi should crawl all the urls belonging to *.target.com.

Additional context Go-colly is the module used for crawling.

After review the code I've noticed that these two files don't follow the enforced ignored list that's passed via -i

@edoardottt Since it's only two calls at the very beginning do you feel like we should enfored the check or we should leave it as is? As the creator I feel like you would have the best insight on what behavior should/should not be enforced :)

https://github.com/edoardottt/cariddi/blob/f52de6c99d7689876aff55662d70f216407be6c4/pkg/crawler/colly.go#L263-L286

Hello Developers,,

The Tool is great after many scans I've discover and be sure that the tool not crawling and catch all parameters in pages specially the "parameters" in the "Filter Categories" most of this "Filters" are with POST requests

Here a live example for the Filter Categories when you check the filter the parameter will be added so u can see it in the URL

i try to scan with many mode -ext -e -c all of them didnt catch the POST parameters

?manufacturer=1-batella&c=1-baby-food&label=1-new&s[flavour]=mango&price_from=2&price_to=3

Step 1 https://i.ibb.co/0KNTH20/1.png

Step 2 https://i.ibb.co/y69q7x7/2.png

Step 3 https://i.ibb.co/YpD7vWJ/3.png

Hope my explain is clear and I hope the developer's find a solution to fix the crawl techniques to make the tool Crawl like this POST requests to make the tool extract more "parameters"

Best Regards,, and keep this tool UP!

Describe the bug CTRL+C once initiates the "smooth" exit correctly. However, pressing it a second time should "force quit" as it currently simply hangs

To Reproduce Steps to reproduce the behavior:

1. Run any long scan
2. Press CTRL+C twice
3. See error

Expected behavior Force quit on second CTRL+C

Screenshots

Desktop (please complete the following information): Linux cyb3rjerry 5.19.0-23-generic 24-Ubuntu SMP PREEMPT_DYNAMIC Fri Oct 14 15:39:57 UTC 2022 x86_64 x86_64 x86_64 GNU/Linux