Polite, slim and concurrent web crawler.


gocrawl GoDoc build status

gocrawl is a polite, slim and concurrent web crawler written in Go.

For a simpler yet more flexible web crawler written in a more idiomatic Go style, you may want to take a look at fetchbot, a package that builds on the experience of gocrawl.



  • Full control over the URLs to visit, inspect and query (using a pre-initialized goquery document)
  • Crawl delays applied per host
  • Obedience to robots.txt rules (using the robotstxt.go library)
  • Concurrent execution using goroutines
  • Configurable logging
  • Open, customizable design providing hooks into the execution logic

Installation and dependencies

gocrawl depends on the following userland libraries:

It requires Go1.1+ because of its indirect dependency on golang.org/x/net/html. To install:

go get github.com/PuerkitoBio/gocrawl

To install a previous version, you have to git clone https://github.com/PuerkitoBio/gocrawl into your $GOPATH/src/github.com/PuerkitoBio/gocrawl/ directory, and then run (for example) git checkout v0.3.2 to checkout a specific version, and go install to build and install the Go package.


  • 2019-07-22 : Use pre-compiled matchers for goquery (thanks @mikefaraponov). Tag v1.0.1.
  • 2016-11-20 : Fix log message so that it prints enqueued URLs (thanks @oherych). Tag as v1.0.0.
  • 2016-05-24 : Set the *URLContext.SourceURL() and *URLContext.NormalizedSourceURL() to the original URL on redirections (see #55). Thanks to github user @tmatsuo.
  • 2016-02-24 : Always use Options.UserAgent to make requests, use Options.RobotUserAgent only for robots.txt policy matching. Lint and vet the code a bit, better godoc documentation.
  • 2014-11-06 : Change import paths of net/html to golang.org/x/net/html (see https://groups.google.com/forum/#!topic/golang-nuts/eD8dh3T9yyA).
  • v0.4.1 : now go-getable, since goquery is go-getable too.
  • v0.4.0 : BREAKING CHANGES major refactor, API changes: * Use an *URLContext structure as first argument to all Extender interface functions that are called in the context of an URL, instead of a simple *url.URL pointer that was sometimes normalized, sometimes not. * Remove the EnqueueOrigin enumeration flag. It wasn't even used by gocrawl, and it is a kind of state associated with the URL, so this feature is now generalized with the next bullet... * Add a State for each URL, so that the crawling process can associate arbitrary data with a given URL (for example, the ID of a record in the database). Fixes issue #14. * More idiomatic use of errors (ErrXxx variables, Run() now returns an error, removing the need for the EndReason enum). * Much simplified Filter() function. It now only returns a bool indicating if it should be visited or not. The HEAD request override feature is provided by the *URLContext structure, and can be set anywhere. The priority feature was unimplemented and has been removed from the return values, if it gets implemented it will probably be via the *URLContext structure too. * Start, Run, Visit, Visited and the EnqueueChan all work with the empty interface type for the URL data. While this is an inconvenience for compile-time checks, it allows for more flexibility regarding the state feature. Instead of always forcing a map[string]interface{} type even when no state is needed, gocrawl supports various types. * Some other internal changes, better tests.
  • v0.3,2 : Fix the high CPU usage when waiting for a crawl delay.
  • v0.3.1 : Export the HttpClient variable used by the default Fetch() implementation (see issue #9).
  • v0.3.0 : BEHAVIOR CHANGE filter done with normalized URL, fetch done with original, non-normalized URL (see issue #10).
  • v0.2.0 : BREAKING CHANGES rework extension/hooks.
  • v0.1.0 : Initial release.


From example_test.go:

// Only enqueue the root and paths beginning with an "a"
var rxOk = regexp.MustCompile(`http://duckduckgo\.com(/a.*)?$`)

// Create the Extender implementation, based on the gocrawl-provided DefaultExtender,
// because we don't want/need to override all methods.
type ExampleExtender struct {
    gocrawl.DefaultExtender // Will use the default implementation of all but Visit and Filter

// Override Visit for our need.
func (x *ExampleExtender) Visit(ctx *gocrawl.URLContext, res *http.Response, doc *goquery.Document) (interface{}, bool) {
    // Use the goquery document or res.Body to manipulate the data
    // ...

    // Return nil and true - let gocrawl find the links
    return nil, true

// Override Filter for our need.
func (x *ExampleExtender) Filter(ctx *gocrawl.URLContext, isVisited bool) bool {
    return !isVisited && rxOk.MatchString(ctx.NormalizedURL().String())

func ExampleCrawl() {
    // Set custom options
    opts := gocrawl.NewOptions(new(ExampleExtender))

    // should always set your robot name so that it looks for the most
    // specific rules possible in robots.txt.
    opts.RobotUserAgent = "Example"
    // and reflect that in the user-agent string used to make requests,
    // ideally with a link so site owners can contact you if there's an issue
    opts.UserAgent = "Mozilla/5.0 (compatible; Example/1.0; +http://example.com)"

    opts.CrawlDelay = 1 * time.Second
    opts.LogFlags = gocrawl.LogAll

    // Play nice with ddgo when running the test!
    opts.MaxVisits = 2

    // Create crawler and start at root of duckduckgo
    c := gocrawl.NewCrawlerWithOptions(opts)

    // Remove "x" before Output: to activate the example (will run on go test)

    // xOutput: voluntarily fail to see log output


Gocrawl can be described as a minimalist web crawler (hence the "slim" tag, at ~1000 sloc), providing the basic engine upon which to build a full-fledged indexing machine with caching, persistence and staleness detection logic, or to use as is for quick and easy crawling. Gocrawl itself does not attempt to detect staleness of a page, nor does it implement a caching mechanism. If an URL is enqueued to be processed, it will make a request to fetch it (provided it is allowed by robots.txt - hence the "polite" tag). And there is no prioritization among the URLs to process, it assumes that all enqueued URLs must be visited at some point, and that the order in which they are is unimportant.

However, it does provide plenty of hooks and customizations. Instead of trying to do everything and impose a way to do it, it offers ways to manipulate and adapt it to anyone's needs.

As usual, the complete godoc reference can be found here.

Design rationale

The major use-case behind gocrawl is to crawl some web pages while respecting the constraints of robots.txt policies and while applying a good web citizen crawl delay between each request to a given host. Hence the following design decisions:

  • Each host spawns its own worker (goroutine) : This makes sense since it must first read its robots.txt data, and only then proceed sequentially, one request at a time, with the specified delay between each fetch. There are no constraints inter-host, so each separate worker can crawl independently.
  • The visitor function is called on the worker goroutine : Again, this is ok because the crawl delay is likely bigger than the time required to parse the document, so this processing will usually not penalize the performance.
  • Edge cases with no crawl delay are supported, but not optimized : In the rare but possible event when a crawl with no delay is needed (e.g.: on your own servers, or with permission outside busy hours, etc.), gocrawl accepts a null (zero) delay, but doesn't provide optimizations. That is, there is no "special path" in the code where visitor functions are de-coupled from the worker, or where multiple workers can be launched concurrently on the same host. (In fact, if this case is your only use-case, I would recommend not to use a library at all - since there is little value in it -, and simply use Go's standard libs and fetch at will with as many goroutines as are necessary.)
  • Extender interface provides the means to write a drop-in, fully encapsulated behaviour : An implementation of Extender can radically enhance the core library, with caching, persistence, different fetching strategies, etc. This is why the Extender.Start() method is somewhat redundant with the Crawler.Run() method, Run allows calling the crawler as a library, while Start makes it possible to encapsulate the logic required to select the seed URLs into the Extender. The same goes for Extender.End() and the return value of Run.

Although it could probably be used to crawl a massive amount of web pages (after all, this is fetch, visit, enqueue, repeat ad nauseam!), most realistic (and um... tested!) use-cases should be based on a well-known, well-defined limited bucket of seeds. Distributed crawling is your friend, should you need to move past this reasonable use.


The Crawler type controls the whole execution. It spawns worker goroutines and manages the URL queue. There are two helper constructors:

  • NewCrawler(Extender) : Creates a crawler with the specified Extender instance.
  • *NewCrawlerWithOptions(Options) : Creates a crawler with a pre-initialized *Options instance.

The one and only public function is Run(seeds interface{}) error which take a seeds argument (the base URLs used to start crawling) that can be expressed a number of different ways. It ends when there are no more URLs waiting to be visited, or when the Options.MaxVisit number is reached. It returns an error, which is ErrMaxVisits if this setting is what caused the crawling to stop.

The various types that can be used to pass the seeds are the following (the same types apply for the empty interfaces in `Extender.Start(interface{}) interface{}`, `Extender.Visit(*URLContext, *http.Response, *goquery.Document) (interface{}, bool)` and in `Extender.Visited(*URLContext, interface{})`, as well as the type of the `EnqueueChan` field):
  • string : a single URL expressed as a string
  • []string : a slice of URLs expressed as strings
  • *url.URL : a pointer to a parsed URL object
  • []*url.URL : a slice of pointers to parsed URL objects
  • map[string]interface{} : a map of URLs expressed as strings (for the key) and their associated state data
  • map[*url.URL]interface{} : a map of URLs expressed as parsed pointers to URL objects (for the key) and their associated state data

For convenience, the types gocrawl.S and gocrawl.U are provided as equivalent to the map of strings and map of URLs, respectively (so that, for example, the code can look like gocrawl.S{"http://site.com": "some state data"}).


The Options type is detailed in the next section, and it offers a single constructor, NewOptions(Extender), which returns an initialized options object with defaults and the specified Extender implementation.

Hooks and customizations

The Options type provides the hooks and customizations offered by gocrawl. All but Extender are optional and have working defaults, but the UserAgent and RobotUserAgent options should be set to a custom value fitting for your project.

  • UserAgent : The user-agent string used to fetch the pages. Defaults to the Firefox 15 on Windows user-agent string. Should be changed to contain a reference to your robot's name and a contact link (see the example).

  • RobotUserAgent : The robot's user-agent string used to find a matching policy in the robots.txt file. Defaults to Googlebot (gocrawl vM.m) where M.m is the major and minor version of gocrawl. This should always be changed to a custom value such as the name of your project (see the example). See the robots exclusion protocol (full specification as interpreted by Google here) for details about the rule-matching based on the robot's user agent. It is good practice to include contact information in the user agent should the site owner need to contact you.

  • MaxVisits : The maximum number of pages visited before stopping the crawl. Probably more useful for development purposes. Note that the Crawler will send its stop signal once this number of visits is reached, but workers may be in the process of visiting other pages, so when the crawling stops, the number of pages visited will be at least MaxVisits, possibly more (worst case is MaxVisits + number of active workers). Defaults to zero, no maximum.

  • EnqueueChanBuffer : The size of the buffer for the Enqueue channel (the channel that allows the extender to arbitrarily enqueue new URLs in the crawler). Defaults to 100.

  • HostBufferFactor : The factor (multiplier) for the size of the workers map and the communication channel when SameHostOnly is set to false. When SameHostOnly is true, the Crawler knows exactly the required size (the number of different hosts based on the seed URLs), but when it is false, the size may grow exponentially. By default, a factor of 10 is used (size is set to 10 times the number of different hosts based on the seed URLs).

  • CrawlDelay : The time to wait between each request to the same host. The delay starts as soon as the response is received from the host. This is a time.Duration type, so it can be specified with 5 * time.Second for example (which is the default value, 5 seconds). If a crawl delay is specified in the robots.txt file, in the group matching the robot's user-agent, by default this delay is used instead. Crawl delay can be customized further by implementing the ComputeDelay extender function.

  • WorkerIdleTTL : The idle time-to-live allowed for a worker before it is cleared (its goroutine terminated). Defaults to 10 seconds. The crawl delay is not part of idle time, this is specifically the time when the worker is available, but there are no URLs to process.

  • SameHostOnly : Limit the URLs to enqueue only to those links targeting the same host, which is true by default.

  • HeadBeforeGet : Asks the crawler to issue a HEAD request (and a subsequent RequestGet() extender method call) before making the eventual GET request. This is set to false by default. See also the URLContext structure explained below.

  • URLNormalizationFlags : The flags to apply when normalizing the URL using the purell library. The URLs are normalized before being enqueued and passed around to the Extender methods in the URLContext structure. Defaults to the most aggressive normalization allowed by purell, purell.FlagsAllGreedy.

  • LogFlags : The level of verbosity for logging. Defaults to errors only (LogError). Can be a set of flags (i.e. LogError | LogTrace).

  • Extender : The instance implementing the Extender interface. This implements the various callbacks offered by gocrawl. Must be specified when creating a Crawler (or when creating an Options to pass to NewCrawlerWithOptions constructor). A default extender is provided as a valid default implementation, DefaultExtender. It can be used by embedding it as an anonymous field to implement a custom extender when not all methods need customization (see the example above).

The Extender interface

This last option field, Extender, is crucial in using gocrawl, so here are the details for each callback function required by the Extender interface.

  • Start : Start(seeds interface{}) interface{}. Called when Run is called on the crawler, with the seeds passed to Run as argument. It returns the data that will be used as actual seeds, so that this callback can control which seeds are processed by the crawler. See the various supported types for more information. By default, this is a passthrough, it returns the data received as argument.

  • End : End(err error). Called when the crawling ends, with the error or nil. This same error is also returned from the Crawler.Run() function. By default, this method is a no-op.

  • Error : Error(err *CrawlError). Called when a crawling error occurs. Errors do not stop the crawling execution. A CrawlError instance is passed as argument. This specialized error implementation includes - among other interesting fields - a Kind field that indicates the step where the error occurred, and an *URLContext field identifying the processed URL that caused the error. By default, this method is a no-op.

  • Log : Log(logFlags LogFlags, msgLevel LogFlags, msg string). The logging function. By default, prints to the standard error (Stderr), and outputs only the messages with a level included in the LogFlags option. If a custom Log() method is implemented, it is up to you to validate if the message should be considered, based on the level of verbosity requested (i.e. if logFlags&msgLevel == msgLevel ...), since the method always gets called for all messages.

  • ComputeDelay : ComputeDelay(host string, di *DelayInfo, lastFetch *FetchInfo) time.Duration. Called by a worker before requesting a URL. Arguments are the host's name (the normalized form of the *url.URL.Host), the crawl delay information (includes delays from the Options struct, from the robots.txt, and the last used delay), and the last fetch information, so that it is possible to adapt to the current responsiveness of the host. It returns the delay to use.

The remaining extension functions are all called in the context of a given URL, so their first argument is always a pointer to an URLContext structure. So before documenting these methods, here is an explanation of all URLContext fields and methods:

  • HeadBeforeGet bool : This field is initialized with the global setting from the crawler's Options structure. It can be overridden at any time, though to be useful it should be done before the call to Fetch, where the decision to make a HEAD request or not is made.
  • State interface{} : This field holds the arbitrary state data associated with the URL. It can be nil or a value of any type.
  • URL() *url.URL : The getter method that returns the parsed URL in non-normalized form.
  • NormalizedURL() *url.URL : The getter method that returns the parsed URL in normalized form.
  • SourceURL() *url.URL : The getter method that returns the source URL in non-normalized form. Can be nil for seeds or URLs enqueued via the EnqueueChan.
  • NormalizedSourceURL() *url.URL : The getter method that returns the source URL in normalized form. Can be nil for seeds or URLs enqueued via the EnqueueChan.
  • IsRobotsURL() bool : Indicates if the current URL is a robots.txt URL.

With this out of the way, here are the other Extender functions:

  • Fetch : Fetch(ctx *URLContext, userAgent string, headRequest bool) (*http.Response, error). Called by a worker to request the URL. The DefaultExtender.Fetch() implementation uses the public HttpClient variable (a custom http.Client) to fetch the pages without following redirections, instead returning a special error (ErrEnqueueRedirect) so that the worker can enqueue the redirect-to URL. This enforces the whitelisting by the Filter() of every URL fetched by the crawling process. If headRequest is true, a HEAD request is made instead of a GET. Note that as of gocrawl v0.3, the default Fetch implementation uses the non-normalized URL.
Internally, gocrawl sets its http.Client's `CheckRedirect()` function field to a custom implementation that follows redirections for robots.txt URLs only (since a redirect on robots.txt still means that the site owner wants us to use these rules for this host). The worker is aware of the `ErrEnqueueRedirect` error, so if a non-robots.txt URL asks for a redirection, `CheckRedirect()` returns this error, and the worker recognizes this and enqueues the redirect-to URL, stopping the processing of the current URL. It is possible to provide a custom `Fetch()` implementation based on the same logic. Any `CheckRedirect()` implementation that returns a `ErrEnqueueRedirect` error will behave this way - that is, the worker will detect this error and will enqueue the redirect-to URL. See the source files ext.go and worker.go for details.

The `HttpClient` variable being public, it is possible to customize it so that it uses another `CheckRedirect()` function, or a different `Transport` object, etc. This customization should be done prior to starting the crawler. It will then be used by the default `Fetch()` implementation, or it can also be used by a custom `Fetch()` if required. Note that this client is shared by all crawlers in your application. Should you need different http clients per crawler in the same application, a custom `Fetch()` using a private `http.Client` instance should be provided.
  • RequestGet : RequestGet(ctx *URLContext, headRes *http.Response) bool. Indicates if the crawler should proceed with a GET request based on the HEAD request's response. This method is only called if a HEAD was requested (based on the *URLContext.HeadBeforeGet field). The default implementation returns true if the HEAD response status code was 2xx.

  • RequestRobots : RequestRobots(ctx *URLContext, robotAgent string) (data []byte, request bool). Asks whether the robots.txt URL should be fetched. If false is returned as second value, the data value is considered to be the robots.txt cached content, and is used as such (if it is empty, it behaves as if there was no robots.txt). The DefaultExtender.RequestRobots implementation returns nil, true.

  • FetchedRobots : FetchedRobots(ctx *URLContext, res *http.Response). Called when the robots.txt URL has been fetched from the host, so that it is possible to cache its content and feed it back to future RequestRobots() calls. By default, this is a no-op.

  • Filter : Filter(ctx *URLContext, isVisited bool) bool. Called when deciding if a URL should be enqueued for visiting. It receives the *URLContext and a bool "is visited" flag, indicating if this URL has already been visited in this crawling execution. It returns a bool flag ordering gocrawl to visit (true) or ignore (false) the URL. Even if the function returns true to enqueue the URL for visiting, the normalized form of the URL must still comply to these rules:

  1. It must be an absolute URL

  2. It must have a http/https scheme

  3. It must have the same host if the SameHostOnly flag is set

    The DefaultExtender.Filter implementation returns true if the URL has not been visited yet (the visited flag is based on the normalized version of the URLs), false otherwise.

  • Enqueued : Enqueued(ctx *URLContext). Called when a URL has been enqueued by the crawler. An enqueued URL may still be disallowed by a robots.txt policy, so it may end up not being fetched. By default, this method is a no-op.

  • Visit : Visit(ctx *URLContext, res *http.Response, doc *goquery.Document) (harvested interface{}, findLinks bool). Called when visiting a URL. It receives the URL context, a *http.Response response object, along with a ready-to-use *goquery.Document object (or nil if the response body could not be parsed). It returns the links to process (see above for the possible types), and a bool flag indicating if gocrawl should find the links himself. When this flag is true, the harvested return value is ignored and gocrawl searches the goquery document for links to enqueue. When false, the harvested data is enqueued, if any. The DefaultExtender.Visit implementation returns nil, true so that links from a visited page are automatically found and processed.

  • Visited : Visited(ctx *URLContext, harvested interface{}). Called after a page has been visited. The URL context and the URLs found during the visit (either by the Visit function or by gocrawl) are passed as argument. By default, this method is a no-op.

  • Disallowed : Disallowed(ctx *URLContext). Called when an enqueued URL gets denied acces by a robots.txt policy. By default, this method is a no-op.

Finally, by convention, if a field named EnqueueChan with the very specific type of chan<- interface{} exists and is accessible on the Extender instance, this field will get set to the enqueue channel, which accepts the expected types as data for URLs to enqueue. This data will then be processed by the crawler as if it had been harvested from a visit. It will trigger calls to Filter() and, if allowed, will get fetched and visited.

The DefaultExtender structure has a valid EnqueueChan field, so if it is embedded as an anonymous field in a custom Extender structure, this structure automatically gets the EnqueueChan functionality.

This channel can be useful to arbitrarily enqueue URLs that would otherwise not be processed by the crawling process. For example, if an URL raises a server error (status code 5xx), it could be re-enqueued in the Error() extender function, so that another fetch is attempted.


  • Richard Penman
  • Dmitry Bondarenko
  • Markus Sonderegger
  • @lionking6792


The BSD 3-Clause license.

  • Support base tags for URL prefix

    Support base tags for URL prefix

    I guess it's not bullet-proof, so please let me know if you would have done it differently. Also, since I'm new to Go, I'm not sure how to run your tests. I didn't see any Makefile or script to run them.

    Please let me know! Thanks!

    opened by juliend2 13
  • Can I have a Crawler that blocks waiting for more urls on EnqueueChan?

    Can I have a Crawler that blocks waiting for more urls on EnqueueChan?

    My usage scenario of the crawler is different from the one-time shot from the sample testcase. I'd like to instantiate a Crawler and pass nil to Run, since the arguments doesn't help on keep feeding the Crawler. I would then use EnqueueChan instead to constantly feed seeds. Digging the sources, it looks like termination of Run based on EnqueueChan is solely due to len(EnqueueChan) which would require me to make sure that EnqueueChan is aways non-empty to avoid the restarting of the crawling process.

    opened by oblitum 10
  • Treat example.com/robots.txt and www.example.com/robots.txt differently

    Treat example.com/robots.txt and www.example.com/robots.txt differently

    Different hosts have different robots.txt. It is inconsistent to use normalized URL (strip www.) to build /robots.txt URL while subsequent crawling URLs are not normalized (with www.).

    opened by ranvis 8
  • HeadBeforeGet() error

    HeadBeforeGet() error

    I'm wanting to use gocrawl for an upcoming project, however I need to use the HeadBeforeGet() method to make sure I only download text/html pages, and don't end up downloading 300mb video files. HeadBeforeGet looks like the ideal way to achieve this, however I always received the following with that option activated:

    2014/04/14 19:49:29 Unsolicited response received on idle HTTP channel starting with "\x1f"; err=<nil>

    Simplified Code:

    package main
    import (
        // "log"
    type ExampleExtender struct {
        gocrawl.DefaultExtender // Will use the default implementation of all but Visit()
    func (this *ExampleExtender) Visit(ctx *gocrawl.URLContext, res *http.Response, doc *goquery.Document) (interface{}, bool) {
        return nil, true
    func main() {
        opts := gocrawl.NewOptions(new(ExampleExtender))
        opts.CrawlDelay = 1 * time.Second
        // opts.LogFlags = gocrawl.LogTrace
        opts.HeadBeforeGet = true
        c := gocrawl.NewCrawlerWithOptions(opts)

    Any idea what's going on?

    opened by JakeAustwick 8
  • Keep User-Agent the same across requests

    Keep User-Agent the same across requests

    I think RobotUserAgent may well be able to be deprecated. User-Agent header of a crawler is expected to be the same if it is for the same purpose of crawling (like for text search indexing.) gocrawl seems to change the User-Agent header when accessing robots.txt. Also it includes Googlebot even though it is not Googlebot, which makes site owners feel strange.

    Sorry for me going to post multiple issues at once. No offense to you but I believe these ideas useful to make this popular crawler even politer.

    opened by ranvis 7
  • httpClient.Transport


    Currently I'm having a little problem with changing http client transport. It is caused by two facts:

    • httpClient (ext.go:120) is private
    • DefaultExtender.Fetch uses it

    In my case I needed to change the standard transport to add timeouts:

    client.Transport = &http.Transport{
                Dial: func(netw, addr string) (net.Conn, error) {
                    deadline := time.Now().Add(ext.Info.Timeout)
                    c, err := net.DialTimeout(netw, addr, ext.Info.Timeout)
                    if err != nil {
                        return nil, err
                    return c, nil

    In current situation I ended up copypasting your default 'Fetch' code (as-is) and 'CheckRedirect' part of your httpClient (as-is). Also, I ended up copypasting your 'isRobotsTxt' func :)

    But I think it would be better to allow somehow to change just parts of this logic, without rewriting it all.

    For example, if we talk about Transport, it could be OK to add a Transport field to Options and to use it in the instantiation code (probably then it should be moved from package-level vars to DefaultExtender):

    httpClient = &http.Client{Transport: options.Transport,  CheckRedirect: func(req *http.Request, via []*http.Request) error {
        // For robots.txt URLs, allow up to 10 redirects, like the default http client.
        // Rationale: the site owner explicitly tells us that this specific robots.txt
        // should be used for this domain.
        if isRobotsTxtUrl(req.URL) {
            if len(via) >= 10 {
                return errors.New("stopped after 10 redirects")
            return nil
        // For all other URLs, do NOT follow redirections, the default Fetch() implementation
        // will ask the worker to enqueue the new (redirect-to) URL. Returning an error
        // will make httpClient.Do() return a url.Error, with the URL field containing the new URL.
        return &EnqueueRedirectError{"redirection not followed"}

    I'm not sure if it is already in your package reorganization plans, I thought that I should submit it just in case.

    opened by ghost 7
  • Default user-agent is likely going to be blocked in most cases

    Default user-agent is likely going to be blocked in most cases

    First of all, Why (by defaults) have two user agents?

    You are using

    Mozilla/5.0 (Windows NT 6.1; rv:15.0) Gecko/20120716 Firefox/15.0a2 for page crawling and then Googlebot (gocrawl v0.4) for when you are grabbing robots.txt from what I can see.

    Impersonating googlebot is already playing with fire against systems that detect and block things that claim to be Google Bot.

    But also why have two different? Does this actually gain anything, Why not ship a user agent that explains that your program is a crawler, it's better to be honest than a WAF detecting that your request isnt what it says it is.

    I understand these are defaults, but encouraging people to use these User-Agent strings by default IMO is a bad idea.

    opened by benjojo 5
  • invalid memory

    invalid memory

    when I run my gocrawler, but suddenly it's stop. and I get message panic: runtime error: invalid memory address or nil pointer dereference how I fix it

    opened by tommoholmes10 5
  • Additional pre-fetch step: HEAD request

    Additional pre-fetch step: HEAD request

    This is a debatable proposal, which seems logical to me, but we could discuss it, maybe some other ways to do the same are better.

    The problem

    The problem is: often I decide to visit a URL or not, depending on its header fields such as 'Content-Type' or 'Content-Length'. So, when deciding to visit or not, I would like to send HEAD request to a resource, then check, whether it is a good 'text/html' or it is some big pdf file/image. Only after I see that its content is worth downloading, I would like to visit it.

    Currently, I implement that in Extender.Filter, where I create my own HEAD request, get the results and return false in case it has the wrong content or something like that.

    The proposal

    I think that this step is pretty common to crawlers (imo, 'Filter' decision is based on content-type even more frequently than on 'from/u' parameters) and it deserves a 'get-head' option in Options and a hook in the Extender. Or, maybe it can be added as a parameter to the Filter itself:

    Filter(u *url.URL, from *url.URL, isVisited bool, head *http.Header ) (enqueue bool, priority int)

    In the last case, the 'head' will be nil if the 'get-head' option is disabled.

    What do you think about it? Thanks!

    opened by ghost 5
  • How to crawl a site that has `<meta name=` ?">

    How to crawl a site that has `` ?

    Here is my example code:

    package main
    import (
    // Create the Extender implementation, based on the gocrawl-provided DefaultExtender,
    // because we don't want/need to override all methods.
    type ExampleExtender struct {
    	gocrawl.DefaultExtender // Will use the default implementation of all but Visit() and Filter()
    func ExampleCrawl(baseUrl string) {
    	// Set custom options
    	opts := gocrawl.NewOptions(new(ExampleExtender))
    	opts.CrawlDelay = 1 * time.Second
    	opts.LogFlags = gocrawl.LogAll
    	c := gocrawl.NewCrawlerWithOptions(opts)
    func (this *ExampleExtender) RequestRobots(ctx *gocrawl.URLContext, robotAgent string) (data []byte, request bool) {
    	fmt.Println("Disobey robots")
    	return []byte(""), false // as if there was no robots.txt at all to obey
    func main() {
    	opts := gocrawl.NewOptions(new(ExampleExtender))
    	opts.CrawlDelay = 1 * time.Second
    	opts.LogFlags = gocrawl.LogAll
    	c := gocrawl.NewCrawlerWithOptions(opts)

    I changed the URL because it's secret. But the only thing that seem to prevent me from crawling that site is the <meta name="robots" content="noindex, nofollow">. It also has this robots.txt, but as you can see above, I ignore it:

    User-agent: *
    Disallow: /

    So, apart from the robots.txt file, is there something in gocrawl or its dependencies that could stop the crawl if it finds a <meta name="robots" content="noindex, nofollow"> tag the HTML response?

    Thanks in advance, and thank you for gocrawl as well, BTW!

    opened by juliend2 4
  • Reduce memory usage?

    Reduce memory usage?

    So far I've tried keeping all my data in a DB, and setting the queue buffer size down to 20. While both of these certainly helped, on pages with a LOT of links the memory usage just climbs and climbs.

    My most recent test has been running for ~10 minutes and is already using 400 MB of ram and counting!

    opened by sosodev 4
  • Graceful & Proper Termination, both internally and via Extender

    Graceful & Proper Termination, both internally and via Extender

    The current stop functionality via Crawler.Stop is insufficient for multiple reasons:

    • Calling Stop twice results in a panic (because it would close c.stop twice)
    • Inside the functions Extender.Visit and Extender.Error there is right now no way of terminating the crawler. Often though, there is absolutely the need to terminate the crawler, for example if some limits are reached (like max amount of visited URLs, determined in a custom Extender.Visit function)
    • In Crawler.collectUrls when res.idleDeath is true, it may delete workers. However, if there are 0 workers it does NOT terminate the entire crawler, leaving it in limbo (memory leak). This I consider a bug.

    I'll create a pull request with a fix. Since it affects the Extender functions, it will break compatibility, but that's a good thing here.

    opened by Kleissner 2
  • Optional Read Limit for reading Response Body

    Optional Read Limit for reading Response Body

    There should be an optional read limit for reading the response body - otherwise a website could literally respond with GBs of data.

    The problem was already raised (and incorrectly dismissed) in 2013 with issue #28. The solution is NOT to check first the content-length header, which may be incorrect and arbitrary.

    The solution is to use an io.LimitReader if a read limit is defined. I'm going to create a pull request which implements that.

    opened by Kleissner 0
  • Parse <img> tags for crawling, if the option ParseImageTags is set.

    Parse tags for crawling, if the option ParseImageTags is set.

    Implement the code for #69 Parsing image tags to include them in crawling. By default the new setting ParseImageTags is set to false, so the current default behavior does not change.

    opened by Kleissner 1
  • Parse image tags

    Parse image tags

    The crawler should optionally parse image tags that are in the form <img src="..."> and use it as input for crawling (just as it would consider <a src=""> links to images).

    I'm creating a pull request which implements that in a few lines.

    opened by Kleissner 2
Nada is a JS runtime, just like Nodejs. The difference is that Nada allows JS developers to easily achieve millions of concurrent applications.

Nada is a JS runtime, just like Nodejs. The difference is that Nada allows JS developers to easily achieve millions of concurrent applications. It also adds some new enhancements to THE JS syntax (types, interfaces, generics) that fundamentally address JS's perennial complaints.

null 22 Jul 11, 2022
Extract structured data from web sites. Web sites scraping.

Dataflow kit Dataflow kit ("DFK") is a Web Scraping framework for Gophers. It extracts data from web pages, following the specified CSS Selectors. You

Dmitry Narizhnykh 562 Jan 7, 2023
記帳-PWA-web-app (Bookkeeping-PWA-web-app)

GoKeep (bookkeeping web app) 記帳-PWA-web-app (Bookkeeping-PWA-web-app) demo link : https://bookkepping.herokuapp.com/ 測試用帳密 : tester002 , tester002 (亦可

Yu-Zhuang Lin 5 Jan 31, 2022
log4jScanner: provides you with the ability to scan internal (only) subnets for vulnerable log4j web servicelog4jScanner: provides you with the ability to scan internal (only) subnets for vulnerable log4j web service

log4jScanner Goals This tool provides you with the ability to scan internal (only) subnets for vulnerable log4j web services. It will attempt to send

Profero 482 Jan 5, 2023
Web terminal - A (unsafe) technical demo to export a shell to web browser

Web Terminal A (unsafe) technical demo to export a shell to web browser. This pr

null 67 Dec 27, 2022
Go-web-scaffold - A simple scaffold for building web app quickly

Go-web-scaffold A simple scaffold for building web app quickly. features This sc

Statrue 3 Jan 21, 2022
Azanul Haque 7 Oct 1, 2021
A web forum built in Golang and SQLite and designed in SCSS

Forum "Fairfax" ?? What is it? A web forum built in Golang and SQLite and designed in SCSS. Members of the forum can take a personality test and be so

null 2 Nov 10, 2021
Automated penetration and auxiliary systems, providing XSS, XXE, DNS log, SSRF, RCE, web netcat and other Servers,gin-vue-admin

Simple DNS log Server,easy to ACME DNS challenge log easy send to elasticsearch https://github.com/hktalent/DNS_Server go4Hacker Automated penetration

51pwn 69 Dec 30, 2022
A quick and easy password protected web server for your files. httpfolder makes downloading/uploading files from your current working directory easy, even for fairly large files.

httpfolder A quick and easy password protected web server for your files. httpfolder makes downloading/uploading files from your current working direc

null 14 Sep 12, 2022
Generate a modern Web project with Go and Angular, React or Vue in seconds 🚀

Goxygen Generate a Web project with Go and Angular, React or Vue. Goxygen aims at saving your time while setting up a new project. It creates a skelet

Sasha Shpota 2.5k Jan 5, 2023
Simple web app using Go and Gin framework

go-gin-app Simple web app using Go and Gin framework Golang 과 Gin 프레임워크를 사용한 간단한 웹 앱 How to get Started Install Gin and have Go installed on your syst

Sean Hong(홍성민) 0 Oct 18, 2021
Birthdays is a web service that stores the birthday date of users and calculates the remaining days until the user's birthday.

Birthdays is a web service that stores the birthday date of users and calculates the remaining days until the user's birthday. Features Metrics servic

Hugo Meneses 1 May 2, 2022
This service finds and — if necessary — generates icons for web sites

favicon-service (besticon) This is a favicon service: Supports favicon.ico and apple-touch-icon.png Simple URL API Fallback icon generation Docker ima

MIFSH 0 Nov 2, 2021
Store files in Telegram messages for free and access them from a modern Web UI

Telegram Storage Store files in Telegram messages for free and access them in a nice web UI. Telegram allows to store files (of max 2GB each) for a un

Jack 30 Jan 4, 2023
A web app that displays a new random famous quote every day (UTC timezone) and allows people to like/unlike the quote

Intro A web app that displays a new random famous quote every day (UTC timezone) and allows people to like/unlike the quote. The app display the curre

Quang Nguyen 0 Dec 8, 2021
REST API for a shoe store using Go and Gin Web Framework

REST API for a shoe store using Go and Gin Web Framework This API uses a local PostgreSQL database that's set through the /gopostgres/driverConfig.go

Nalbert Wattam 0 Dec 26, 2021
Go-service-gin - Simple Web api application developed in Golang and Gin

Simple Web api application developed in Golang and Gin Initial Tutorial URL http

Nurul Huda Robin 0 Jan 4, 2022
A new way to create web applications using go and sdf framework.

SDF GO A new way to create web applications using go and sdf framework Explore the docs » View Demo · Report Bug · Request Feature Table of Contents A

Metin Simsek 1 Sep 27, 2022