A faster, simpler way to drive browsers supporting the Chrome DevTools Protocol.

Overview

About chromedp GoDoc

Package chromedp is a faster, simpler way to drive browsers supporting the Chrome DevTools Protocol in Go without external dependencies (like Selenium or PhantomJS).

Installing

Install in the usual Go way:

go get -u github.com/chromedp/chromedp

Examples

Refer to the GoDoc page for the documentation and examples. Additionally, the examples repository contains more complex examples.

Frequently Asked Questions

I can't see any Chrome browser window

By default, Chrome is run in headless mode. See DefaultExecAllocatorOptions, and an example to override the default options.

I'm seeing "context canceled" errors

When the connection to the browser is lost, chromedp cancels the context, and it may result in this error. This occurs, for example, if the browser is closed manually, or if the browser process has been killed or otherwise terminated.

Chrome exits as soon as my Go program finishes

On Linux, chromedp is configured to avoid leaking resources by force-killing any started Chrome child processes. If you need to launch a long-running Chrome instance, manually start Chrome and connect using RemoteAllocator.

Executing an action without Run results in "invalid context"

By default, a chromedp context does not have an executor, however one can be specified manually if necessary; see issue #326 for an example.

I can't use an Action with Run because it returns many values

Wrap it with an ActionFunc:

chromedp.Run(ctx, chromedp.ActionFunc(func(ctx context.Context) error {
	_, err := domain.SomeAction().Do(ctx)
	return err
}))

I want to use chromedp on a headless environment

The simplest way is to run the Go program that uses chromedp inside the chromedp/headless-shell image. That image contains headless-shell, a smaller headless build of Chrome, which chromedp is able to find out of the box.

Resources

Comments
  • domEvent: timeout waiting for node

    domEvent: timeout waiting for node

    This is more a question than a bug report.

    When a dom event listener is about to wait for a node and the node gets concurrently removed (or its parent), WaitNode fails with timeout. The problem here is when that happens, the whole action handlers stale for 10 seconds. At least this is what I think happens. The behaviour is random, if WaitNode finishes before it's removed, then everything works as expected (I think also Go scheduler contributes to the racy behaviour).

    Debug log for timeout scenario:

    scrap.go:91: -> {"method":"DOM.childNodeInserted","params":{"parentNodeId":7,"previousNodeId":8,"node":{"nodeId":9,"backendNodeId":9,"nodeType":1,"nodeName":"DIV","localName":"div","nodeValue":"","childNodeCount":0,"attributes":[]}}}
    scrap.go:91: -> {"method":"DOM.attributeModified","params":{"nodeId":9,"name":"id","value":"someotherid"}}
    scrap.go:91: -> {"method":"DOM.childNodeInserted","params":{"parentNodeId":7,"previousNodeId":0,"node":{"nodeId":11,"backendNodeId":11,"nodeType":1,"nodeName":"BODY","localName":"body","nodeValue":"","childNodeCount":1,"attributes":["style","visibility: hidden; width: 0px; height: 0px; border: 0px; margin: 0px; background: none;"]}}}
    scrap.go:91: -> {"method":"DOM.childNodeCountUpdated","params":{"nodeId":11,"childNodeCount":0}}
    scrap.go:91: -> {"method":"DOM.childNodeRemoved","params":{"parentNodeId":7,"nodeId":9}}
    scrap.go:91: -> {"method":"DOM.childNodeRemoved","params":{"parentNodeId":7,"nodeId":11}}
    scrap.go:91: could not perform (attributeModified) operation on node 9 (wait node): timeout waiting for node `9`
    scrap.go:91: could not perform (childNodeCountUpdated) operation on node 11 (wait node): timeout waiting for node `11`
    

    I was thinking about fixing this (the 10s stales are really bugging me) by introducing some kind of context cancellation / listening channel, so we break out of waiting loop in WaitNode when the node we're waiting for gets removed.

    Wdyt?

    opened by rjeczalik 32
  • won't work out of the box with a headless-shell container running as root

    won't work out of the box with a headless-shell container running as root

    What versions are you running?

    $ go list -m github.com/chromedp/chromedp
    github.com/chromedp/chromedp v0.1.4-0.20190409142133-92a77355f6df
    $ chromium --version
    Chromium 73.0.3683.75 built on Debian 9.8, running on Debian 9.8
    $ go version
    go version go1.12.3 linux/amd64
    

    What did you do?

    I wrote the following test

    package main
    
    import (
            "context"
            "testing"
            "time"
    
            "github.com/chromedp/chromedp"
    )
    
    func Test(t *testing.T) {
            ctx, cancel := chromedp.NewContext(
                    context.Background(),
                    chromedp.WithLogf(t.Logf),
            )
            defer cancel()
    
            ctx, cancel = context.WithTimeout(ctx, 10*time.Second)
            defer cancel()
    
            err := chromedp.Run(ctx,
                    chromedp.Navigate("http://localhost:3000/"),
            )
            if err != nil {
                    t.Fatal(err)
            }
    }
    

    and tried to go test.

    What did you expect to see?

    go test succeeds.

    What did you see instead?

    main_test.go:25: malformed ws or wss URL

    and, sometimes, also a panic:

    panic: sync: negative WaitGroup counter
    
    goroutine 13 [running]:
    sync.(*WaitGroup).Add(0xc000332044, 0xffffffffffffffff)
            /usr/local/go/src/sync/waitgroup.go:74 +0x135
    sync.(*WaitGroup).Done(...)
            /usr/local/go/src/sync/waitgroup.go:99
    github.com/chromedp/chromedp.(*ExecAllocator).Allocate.func1(0xcdac40, 0xc000332060, 0xc000010888, 0xc000128e01, 0xc0000377c0, 0x1d, 0xc000332000, 0xc0002e4ea0)
            /go/pkg/mod/github.com/chromedp/[email protected]/allocate.go:126 +0xa2
    created by github.com/chromedp/chromedp.(*ExecAllocator).Allocate
            /go/pkg/mod/github.com/chromedp/[email protected]/allocate.go:109 +0x740
    
    opened by opennota 30
  • How to set http proxy user and password in chromedp?

    How to set http proxy user and password in chromedp?

    Hi, I know the chromedp could set http proxy on option "runner.Proxy(http://localhost:8000/)", but how to set the proxy's user and password? Thanks

    opened by shadow1163 30
  • An example to capture a screenshot of the full page

    An example to capture a screenshot of the full page

    Is it possible to get a screenshot of the full page, even when setting a specific width and height to emulate a mobile device?

    For example, here is a screenshot using dev tools and an iphone 6.

    I'm looking to achieve the same result.

    http://i.imgur.com/0Je8sgT.png

    So far, I'm getting an image which is cut off at the specified height.

    opened by paulm17 28
  • x509: certificate signed by unknown authority

    x509: certificate signed by unknown authority

    go install in docker image report x509: certificate signed by unknown

    What versions are you running?

    uname -r

    5.4.72-microsoft-standard-WSL2

    cat /etc/os-release

    PRETTY_NAME="Debian GNU/Linux 10 (buster)" NAME="Debian GNU/Linux" VERSION_ID="10" VERSION="10 (buster)" VERSION_CODENAME=buster ID=debian HOME_URL="https://www.debian.org/" SUPPORT_URL="https://www.debian.org/support" BUG_REPORT_URL="https://bugs.debian.org/"

    $ go list -m github.com/chromedp/chromedp go: github.com/chromedp/[email protected]: missing go.sum entry; to add it: go mod download github.com/chromedp/chromedp

    $ google-chrome --version sh: 137: google-chrome: not found

    $ go version go version go1.16.5 linux/amd64

    What did you do? Include clear steps.

    docker pull chromedp/headless-shell:74.0.3717.1 //download go1.16.5.linux-amd64.tar.gz and put into local /goinstall directoy docker run -v ${PWD}/goinstall/:/tmp/ -d --name chromedpcontainer chromedp/headless-shell

    docker exec -it sh cd /tmp tar -C /usr/local -xzf go1.16.5.linux-amd64.tar.gz export PATH=$PATH:/usr/local/go/bin cd /tmp go mod init helloworld go get -x github.com/chromedp/chromedp

    What did you expect to see?

    github.com/chromedp/chromedp installed. below added into go.mod

    require ( github.com/chromedp/chromedp v0.6.10 )

    What did you see instead?

    go: github.com/chromedp/[email protected]: Get "https://proxy.golang.org/github.com/chromedp/chromedp/@v/v0.6.10.mod": x509: certificate signed by unknown authority

    opened by davidzhongsydney 23
  • bypass headless chrome detection

    bypass headless chrome detection

    What versions are you running?

    $ go list -m github.com/chromedp/chromedp
    go list -m: not using modules
    
    $ google-chrome --version
    /Applications/Chromium.app/Contents/MacOS/Chromium --version
    Chromium 77.0.3830.0
    
    $ go version
    go version go1.12.6 darwin/amd64
    

    What did you do? Include clear steps.

    package main
    
    import (
    	"context"
    	"fmt"
    	"time"
    
    	"github.com/pkg/errors"
    
    	"github.com/chromedp/cdproto/runtime"
    	"github.com/chromedp/chromedp"
    )
    
    func main() {
    	opts := []chromedp.ExecAllocatorOption{
    		chromedp.ExecPath(`/Applications/Chromium.app/Contents/MacOS/Chromium`),
    		chromedp.UserAgent("Mozilla/5.0 (Macintosh; Intel Mac OS X 10_14_5) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/77.0.3830.0 Safari/537.36"),
    		chromedp.WindowSize(1920, 1080),
    		chromedp.NoFirstRun,
    		chromedp.NoDefaultBrowserCheck,
    		chromedp.Headless,
    		chromedp.DisableGPU,
    	}
    
    	ctx, cancel := chromedp.NewExecAllocator(context.Background(), opts...)
    	defer cancel()
    
    	ctx, cancel = chromedp.NewContext(ctx)
    	defer cancel()
    
    	err := chromedp.Run(ctx,
    		evalJS(`Object.defineProperty(navigator, 'plugins', {get: () => [1, 2, 3, 4, 5, 7, 8, 9]});`),
    		evalJS(`navigator.plugins.length`),
    		chromedp.Navigate("https://intoli.com/blog/not-possible-to-block-chrome-headless/chrome-headless-test.html"),
    		chromedp.ActionFunc(func(ctx context.Context) error {
    			time.Sleep(3 * time.Second)
    			return nil
    		}),
    		evalJS(`navigator.plugins.length`),
    	)
    	if err != nil {
    		panic(err)
    	}
    }
    
    func evalJS(js string) chromedp.Tasks {
    	var res *runtime.RemoteObject
    	return chromedp.Tasks{
    		chromedp.EvaluateAsDevTools(js, &res),
    		chromedp.ActionFunc(func(ctx context.Context) error {
    			b, err := res.MarshalJSON()
    			if err != nil {
    				return errors.Wrap(err, "marshal")
    			}
    			fmt.Println("result: ", string(b))
    			return nil
    		}),
    	}
    }
    
    

    What did you expect to see?

    result:  {"type":"object","className":"Navigator","description":"Navigator","objectId":"{\"injectedScriptId\":1,\"id\":1}"}
    result:  {"type":"number","value":8,"description":"8"}
    result:  {"type":"number","value":8,"description":"8"}
    

    I am trying to bypass headless browser detection as written in this article https://intoli.com/blog/not-possible-to-block-chrome-headless/. This works fine for puppeteer, but doesn't work for chromedp. I need to understand how to execute js code in chromedp, as in puppeteer. Because every time after run chromedp.Navigate() values are reset to default.

    It is possible to do with chromedp?

    What did you see instead?

    result:  {"type":"object","className":"Navigator","description":"Navigator","objectId":"{\"injectedScriptId\":1,\"id\":1}"}
    result:  {"type":"number","value":8,"description":"8"}
    result:  {"type":"number","value":0,"description":"0"} // after chromedp.Navigate() navigator.plugins.length has default value equal 0
    

    Works fine with puppeteer.

    const puppeteer = require('puppeteer');
    
    const userAgent = 'Mozilla/5.0 (X11; Linux x86_64)' +
      'AppleWebKit/537.36 (KHTML, like Gecko) Chrome/64.0.3282.39 Safari/537.36';
    
    (async () => {
      const browser = await puppeteer.launch({headless: true});
      const page = await browser.newPage();
      
      await page.setUserAgent(userAgent);
    
      await page.evaluateOnNewDocument(() => {
        // Overwrite the `plugins` property to use a custom getter.
        Object.defineProperty(navigator, 'plugins', {
          // This just needs to have `length > 0` for the current test,
          // but we could mock the plugins too if necessary.
          get: () => [1, 2, 3, 4, 5, 6, 7, 8],
        });
      });
    
      await page.goto('https://intoli.com/blog/not-possible-to-block-chrome-headless/chrome-headless-test.html');
      await page.screenshot({path: 'result.png'});
    
      await browser.close();
    })();
    
    opened by epanov 23
  • Problem with aws lambda

    Problem with aws lambda

    What versions are you running?

    $ go list -m github.com/chromedp/chromedp
    github.com/chromedp/chromedp v0.8.2
    
    $ docker pull chromedp/headless-shell:latest
    latest: Pulling from chromedp/headless-shell
    Digest: sha256:8fe1cbf5aba60474079fedf56867a3a3576340ab21b6e222674edcb6610a584d
    Status: Image is up to date for chromedp/headless-shell:latest
    docker.io/chromedp/headless-shell:latest
    
    $ docker run --rm golang:alpine go version
    go version go1.18.2 linux/amd64
    

    What did you do? Include clear steps.

    Here is an example repo. I want to deploy it to aws lambda.

    Code snippet

    func Handler(_ context.Context, _ json.RawMessage) error {
    	opts := []chromedp.ExecAllocatorOption{
    		chromedp.NoFirstRun,
    		chromedp.NoDefaultBrowserCheck,
    		chromedp.NoSandbox,
    		chromedp.DisableGPU,
    		chromedp.Headless,
    		chromedp.Flag("no-zygote", true),
    		chromedp.Flag("user-data-dir", "/tmp/chrome-user-data-dir"),
    		chromedp.Flag("homedir", "/tmp/chrome-home"),
    		chromedp.Flag("data-path", "/tmp/chrome-data-path"),
    		chromedp.Flag("disk-cache-dir", "/tmp/chrome-disk-cache-dir"),
    		chromedp.Flag("remote-debugging-port", "9222"),
    		chromedp.Flag("remote-debugging-address", "0.0.0.0"),
    		chromedp.Flag("disable-dev-shm-usage", true),
    		chromedp.Flag("enable-features", "NetworkService,NetworkServiceInProcess"),
    	}
    	ctx, cancel := chromedp.NewExecAllocator(context.Background(), opts...)
    	defer cancel()
    
    	ctx, cancel = chromedp.NewContext(ctx, chromedp.WithDebugf(log.Printf))
    	defer cancel()
    
    	var content string
    	if err := chromedp.Run(ctx, chromedp.Tasks{
    		chromedp.Navigate("https://example.com/"),
    		chromedp.Text("body > div > p:nth-child(2)", &content),
    	}); err != nil {
    		log.Fatal(err)
    	}
    	fmt.Println(content)
    	return nil
    }
    
    FROM golang:alpine AS builder
    
    RUN apk add git
    
    WORKDIR /app
    
    COPY go.mod go.sum ./
    RUN go mod download
    
    COPY . .
    
    RUN go build -o main
    
    FROM chromedp/headless-shell
    
    RUN apt-get update && \
        apt-get install -y dumb-init musl-dev && \
        ln -s /usr/lib/x86_64-linux-musl/libc.so /lib/libc.musl-x86_64.so.1 &&\
        rm -rf /var/lib/apt/lists/*
    
    COPY --from=builder /app/main .
    
    ENTRYPOINT [ "dumb-init", "--" ]
    CMD [ "./main" ]
    

    This code (without docker) run perfectly on my M1 macbook, and built docker container run perfectly on AWS EC2.

    But when I deploy it to aws lambda, it just stuck after last log 2022/06/01 04:13:38 -> {"id":3,"sessionId":"26082765C593E944A783B371227B397A","method":"Runtime.enable"}.

    What did you expect to see?

    I can run it on aws lambda.

    Log on ec2 running perfectly

    2022/06/01 05:13:03 -> {"id":1,"method":"Target.setDiscoverTargets","params":{"discover":true}}
    2022/06/01 05:13:03 <- {"method":"Target.targetCreated","params":{"targetInfo":{"targetId":"11278bc8-3ff8-436d-901f-a6404ecb61d1","type":"browser","title":"","url":"","attached":false,"canAccessOpener":false}}}
    2022/06/01 05:13:03 <- {"method":"Target.targetCreated","params":{"targetInfo":{"targetId":"412DC8003B59C78196B600AFBB7C0231","type":"page","title":"","url":"about:blank","attached":false,"canAccessOpener":false,"browserContextId":"1488447107EADF5939A51B3C61EA1F3B"}}}
    2022/06/01 05:13:03 <- {"method":"Target.targetCreated","params":{"targetInfo":{"targetId":"e42b9ee5-9df7-49b6-b6f2-0314165479f6","type":"browser","title":"","url":"","attached":true,"canAccessOpener":false}}}
    2022/06/01 05:13:03 <- {"id":1,"result":{}}
    2022/06/01 05:13:03 -> {"id":2,"method":"Target.attachToTarget","params":{"targetId":"412DC8003B59C78196B600AFBB7C0231","flatten":true}}
    2022/06/01 05:13:03 <- {"method":"Target.targetInfoChanged","params":{"targetInfo":{"targetId":"412DC8003B59C78196B600AFBB7C0231","type":"page","title":"","url":"about:blank","attached":true,"canAccessOpener":false,"browserContextId":"1488447107EADF5939A51B3C61EA1F3B"}}}
    2022/06/01 05:13:03 <- {"method":"Target.attachedToTarget","params":{"sessionId":"0F3F699532F154F976242DBF4AC0DFD2","targetInfo":{"targetId":"412DC8003B59C78196B600AFBB7C0231","type":"page","title":"","url":"about:blank","attached":true,"canAccessOpener":false,"browserContextId":"1488447107EADF5939A51B3C61EA1F3B"},"waitingForDebugger":false}}
    2022/06/01 05:13:03 <- {"id":2,"result":{"sessionId":"0F3F699532F154F976242DBF4AC0DFD2"}}
    2022/06/01 05:13:04 -> {"id":3,"sessionId":"0F3F699532F154F976242DBF4AC0DFD2","method":"Runtime.enable"}
    2022/06/01 05:13:04 <- {"method":"Target.targetInfoChanged","params":{"targetInfo":{"targetId":"412DC8003B59C78196B600AFBB7C0231","type":"page","title":"about:blank","url":"about:blank","attached":true,"canAccessOpener":false,"browserContextId":"1488447107EADF5939A51B3C61EA1F3B"}}}
    2022/06/01 05:13:04 <- {"method":"Runtime.executionContextCreated","params":{"context":{"id":1,"origin":"://","name":"","uniqueId":"5877140344713506672.6727992831272504814","auxData":{"isDefault":true,"type":"default","frameId":"412DC8003B59C78196B600AFBB7C0231"}}},"sessionId":"0F3F699532F154F976242DBF4AC0DFD2"}
    2022/06/01 05:13:04 <- {"id":3,"result":{},"sessionId":"0F3F699532F154F976242DBF4AC0DFD2"}
    2022/06/01 05:13:04 -> {"id":4,"sessionId":"0F3F699532F154F976242DBF4AC0DFD2","method":"Runtime.evaluate","params":{"expression":"self"}}
    2022/06/01 05:13:04 <- {"id":4,"result":{"result":{"type":"object","className":"Window","description":"Window","objectId":"5836470857787727263.1.1"}},"sessionId":"0F3F699532F154F976242DBF4AC0DFD2"}
    2022/06/01 05:13:04 -> {"id":5,"sessionId":"0F3F699532F154F976242DBF4AC0DFD2","method":"Log.enable"}
    2022/06/01 05:13:04 <- {"id":5,"result":{},"sessionId":"0F3F699532F154F976242DBF4AC0DFD2"}
    2022/06/01 05:13:04 -> {"id":6,"sessionId":"0F3F699532F154F976242DBF4AC0DFD2","method":"Network.enable","params":{}}
    2022/06/01 05:13:04 <- {"id":6,"result":{},"sessionId":"0F3F699532F154F976242DBF4AC0DFD2"}
    2022/06/01 05:13:04 -> {"id":7,"sessionId":"0F3F699532F154F976242DBF4AC0DFD2","method":"Inspector.enable"}
    2022/06/01 05:13:04 <- {"id":7,"result":{},"sessionId":"0F3F699532F154F976242DBF4AC0DFD2"}
    2022/06/01 05:13:04 -> {"id":8,"sessionId":"0F3F699532F154F976242DBF4AC0DFD2","method":"Page.enable"}
    2022/06/01 05:13:04 <- {"id":8,"result":{},"sessionId":"0F3F699532F154F976242DBF4AC0DFD2"}
    2022/06/01 05:13:04 -> {"id":9,"sessionId":"0F3F699532F154F976242DBF4AC0DFD2","method":"DOM.enable","params":{}}
    2022/06/01 05:13:04 <- {"id":9,"result":{},"sessionId":"0F3F699532F154F976242DBF4AC0DFD2"}
    2022/06/01 05:13:04 -> {"id":10,"sessionId":"0F3F699532F154F976242DBF4AC0DFD2","method":"CSS.enable"}
    2022/06/01 05:13:04 <- {"id":10,"result":{},"sessionId":"0F3F699532F154F976242DBF4AC0DFD2"}
    2022/06/01 05:13:04 -> {"id":11,"sessionId":"0F3F699532F154F976242DBF4AC0DFD2","method":"Target.setDiscoverTargets","params":{"discover":true}}
    2022/06/01 05:13:04 <- {"method":"Target.targetCreated","params":{"targetInfo":{"targetId":"11278bc8-3ff8-436d-901f-a6404ecb61d1","type":"browser","title":"","url":"","attached":false,"canAccessOpener":false}},"sessionId":"0F3F699532F154F976242DBF4AC0DFD2"}
    2022/06/01 05:13:04 <- {"method":"Target.targetCreated","params":{"targetInfo":{"targetId":"412DC8003B59C78196B600AFBB7C0231","type":"page","title":"about:blank","url":"about:blank","attached":true,"canAccessOpener":false,"browserContextId":"1488447107EADF5939A51B3C61EA1F3B"}},"sessionId":"0F3F699532F154F976242DBF4AC0DFD2"}
    2022/06/01 05:13:04 <- {"method":"Target.targetCreated","params":{"targetInfo":{"targetId":"e42b9ee5-9df7-49b6-b6f2-0314165479f6","type":"browser","title":"","url":"","attached":true,"canAccessOpener":false}},"sessionId":"0F3F699532F154F976242DBF4AC0DFD2"}
    2022/06/01 05:13:04 <- {"id":11,"result":{},"sessionId":"0F3F699532F154F976242DBF4AC0DFD2"}
    2022/06/01 05:13:04 -> {"id":12,"sessionId":"0F3F699532F154F976242DBF4AC0DFD2","method":"Target.setAutoAttach","params":{"autoAttach":true,"waitForDebuggerOnStart":false,"flatten":true}}
    2022/06/01 05:13:04 <- {"id":12,"result":{},"sessionId":"0F3F699532F154F976242DBF4AC0DFD2"}
    2022/06/01 05:13:04 -> {"id":13,"sessionId":"0F3F699532F154F976242DBF4AC0DFD2","method":"Page.setLifecycleEventsEnabled","params":{"enabled":true}}
    2022/06/01 05:13:04 <- {"method":"Page.lifecycleEvent","params":{"frameId":"412DC8003B59C78196B600AFBB7C0231","loaderId":"8E275FC88E861B68C971049B227D2C13","name":"commit","timestamp":55583.941934},"sessionId":"0F3F699532F154F976242DBF4AC0DFD2"}
    2022/06/01 05:13:04 <- {"method":"Page.lifecycleEvent","params":{"frameId":"412DC8003B59C78196B600AFBB7C0231","loaderId":"8E275FC88E861B68C971049B227D2C13","name":"DOMContentLoaded","timestamp":55583.942021},"sessionId":"0F3F699532F154F976242DBF4AC0DFD2"}
    2022/06/01 05:13:04 <- {"method":"Page.lifecycleEvent","params":{"frameId":"412DC8003B59C78196B600AFBB7C0231","loaderId":"8E275FC88E861B68C971049B227D2C13","name":"load","timestamp":55583.942425},"sessionId":"0F3F699532F154F976242DBF4AC0DFD2"}
    2022/06/01 05:13:04 <- {"method":"Page.lifecycleEvent","params":{"frameId":"412DC8003B59C78196B600AFBB7C0231","loaderId":"8E275FC88E861B68C971049B227D2C13","name":"networkAlmostIdle","timestamp":55583.942883},"sessionId":"0F3F699532F154F976242DBF4AC0DFD2"}
    2022/06/01 05:13:04 <- {"method":"Page.lifecycleEvent","params":{"frameId":"412DC8003B59C78196B600AFBB7C0231","loaderId":"8E275FC88E861B68C971049B227D2C13","name":"networkIdle","timestamp":55583.942883},"sessionId":"0F3F699532F154F976242DBF4AC0DFD2"}
    2022/06/01 05:13:04 <- {"id":13,"result":{},"sessionId":"0F3F699532F154F976242DBF4AC0DFD2"}
    2022/06/01 05:13:04 -> {"id":14,"sessionId":"0F3F699532F154F976242DBF4AC0DFD2","method":"Page.navigate","params":{"url":"https://example.com/"}}
    2022/06/01 05:13:04 <- {"method":"Network.requestWillBeSent","params":{"requestId":"C9980CD6274A43FBE9998E7B5E95629B","loaderId":"C9980CD6274A43FBE9998E7B5E95629B","documentURL":"https://example.com/","request":{"url":"https://example.com/","method":"GET","headers":{"Upgrade-Insecure-Requests":"1","User-Agent":"Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/102.0.5005.63 Safari/537.36"},"mixedContentType":"none","initialPriority":"VeryHigh","referrerPolicy":"strict-origin-when-cross-origin","isSameSite":true},"timestamp":55583.963522,"wallTime":1654060384.050588,"initiator":{"type":"other"},"redirectHasExtraInfo":false,"type":"Document","frameId":"412DC8003B59C78196B600AFBB7C0231","hasUserGesture":false},"sessionId":"0F3F699532F154F976242DBF4AC0DFD2"}
    2022/06/01 05:13:04 <- {"method":"Network.requestWillBeSentExtraInfo","params":{"requestId":"C9980CD6274A43FBE9998E7B5E95629B","associatedCookies":[],"headers":{":authority":"example.com",":method":"GET",":path":"/",":scheme":"https","accept":"text/html,application/xhtml+xml,application/xml;q=0.9,image/avif,image/webp,image/apng,*/*;q=0.8,application/signed-exchange;v=b3;q=0.9","accept-encoding":"gzip, deflate, br","accept-language":"[email protected]","sec-fetch-dest":"document","sec-fetch-mode":"navigate","sec-fetch-site":"none","sec-fetch-user":"?1","upgrade-insecure-requests":"1","user-agent":"Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/102.0.5005.63 Safari/537.36"},"connectTiming":{"requestTime":55583.965293}},"sessionId":"0F3F699532F154F976242DBF4AC0DFD2"}
    2022/06/01 05:13:04 <- {"method":"Network.responseReceivedExtraInfo","params":{"requestId":"C9980CD6274A43FBE9998E7B5E95629B","blockedCookies":[],"headers":{"accept-ranges":"bytes","age":"267163","cache-control":"max-age=604800","content-encoding":"gzip","content-length":"648","content-type":"text/html; charset=UTF-8","date":"Wed, 01 Jun 2022 05:13:04 GMT","etag":"\"3147526947\"","expires":"Wed, 08 Jun 2022 05:13:04 GMT","last-modified":"Thu, 17 Oct 2019 07:18:26 GMT","server":"ECS (oxr/8370)","vary":"Accept-Encoding","x-cache":"HIT"},"resourceIPAddressSpace":"Public","statusCode":200},"sessionId":"0F3F699532F154F976242DBF4AC0DFD2"}
    2022/06/01 05:13:04 <- {"method":"Network.responseReceived","params":{"requestId":"C9980CD6274A43FBE9998E7B5E95629B","loaderId":"C9980CD6274A43FBE9998E7B5E95629B","timestamp":55584.700562,"type":"Document","response":{"url":"https://example.com/","status":200,"statusText":"","headers":{"accept-ranges":"bytes","age":"267163","cache-control":"max-age=604800","content-encoding":"gzip","content-length":"648","content-type":"text/html; charset=UTF-8","date":"Wed, 01 Jun 2022 05:13:04 GMT","etag":"\"3147526947\"","expires":"Wed, 08 Jun 2022 05:13:04 GMT","last-modified":"Thu, 17 Oct 2019 07:18:26 GMT","server":"ECS (oxr/8370)","vary":"Accept-Encoding","x-cache":"HIT"},"mimeType":"text/html","connectionReused":false,"connectionId":13,"remoteIPAddress":"93.184.216.34","remotePort":443,"fromDiskCache":false,"fromServiceWorker":false,"fromPrefetchCache":false,"encodedDataLength":193,"timing":{"requestTime":55583.965293,"proxyStart":-1,"proxyEnd":-1,"dnsStart":23.354,"dnsEnd":24.929,"connectStart":24.929,"connectEnd":562.419,"sslStart":194.584,"sslEnd":562.39,"workerStart":-1,"workerReady":-1,"workerFetchStart":-1,"workerRespondWithSettled":-1,"sendStart":562.684,"sendEnd":562.847,"pushStart":0,"pushEnd":0,"receiveHeadersEnd":733.232},"responseTime":1.654060384785279e+12,"protocol":"h2","securityState":"secure","securityDetails":{"protocol":"TLS 1.3","keyExchange":"","keyExchangeGroup":"P-256","cipher":"AES_256_GCM","certificateId":0,"subjectName":"www.example.org","sanList":["www.example.org","example.net","example.edu","example.com","example.org","www.example.com","www.example.edu","www.example.net"],"issuer":"DigiCert TLS RSA SHA256 2020 CA1","validFrom":1647216000,"validTo":1678838399,"signedCertificateTimestampList":[],"certificateTransparencyCompliance":"unknown"}},"hasExtraInfo":true,"frameId":"412DC8003B59C78196B600AFBB7C0231"},"sessionId":"0F3F699532F154F976242DBF4AC0DFD2"}
    2022/06/01 05:13:04 <- {"id":14,"result":{"frameId":"412DC8003B59C78196B600AFBB7C0231","loaderId":"C9980CD6274A43FBE9998E7B5E95629B"},"sessionId":"0F3F699532F154F976242DBF4AC0DFD2"}
    2022/06/01 05:13:04 <- {"method":"Page.lifecycleEvent","params":{"frameId":"412DC8003B59C78196B600AFBB7C0231","loaderId":"8E275FC88E861B68C971049B227D2C13","name":"networkAlmostIdle","timestamp":55583.942883},"sessionId":"0F3F699532F154F976242DBF4AC0DFD2"}
    2022/06/01 05:13:04 <- {"method":"Page.lifecycleEvent","params":{"frameId":"412DC8003B59C78196B600AFBB7C0231","loaderId":"8E275FC88E861B68C971049B227D2C13","name":"networkIdle","timestamp":55583.942883},"sessionId":"0F3F699532F154F976242DBF4AC0DFD2"}
    2022/06/01 05:13:04 <- {"method":"Page.frameStartedLoading","params":{"frameId":"412DC8003B59C78196B600AFBB7C0231"},"sessionId":"0F3F699532F154F976242DBF4AC0DFD2"}
    2022/06/01 05:13:04 <- {"method":"Page.lifecycleEvent","params":{"frameId":"412DC8003B59C78196B600AFBB7C0231","loaderId":"C9980CD6274A43FBE9998E7B5E95629B","name":"init","timestamp":55584.704893},"sessionId":"0F3F699532F154F976242DBF4AC0DFD2"}
    2022/06/01 05:13:04 <- {"method":"Target.targetInfoChanged","params":{"targetInfo":{"targetId":"412DC8003B59C78196B600AFBB7C0231","type":"page","title":"https://example.com","url":"https://example.com/","attached":true,"canAccessOpener":false,"browserContextId":"1488447107EADF5939A51B3C61EA1F3B"}}}
    2022/06/01 05:13:04 <- {"method":"Target.targetInfoChanged","params":{"targetInfo":{"targetId":"412DC8003B59C78196B600AFBB7C0231","type":"page","title":"https://example.com","url":"https://example.com/","attached":true,"canAccessOpener":false,"browserContextId":"1488447107EADF5939A51B3C61EA1F3B"}},"sessionId":"0F3F699532F154F976242DBF4AC0DFD2"}
    2022/06/01 05:13:04 <- {"method":"Runtime.executionContextsCleared","params":{},"sessionId":"0F3F699532F154F976242DBF4AC0DFD2"}
    2022/06/01 05:13:04 <- {"method":"Page.frameNavigated","params":{"frame":{"id":"412DC8003B59C78196B600AFBB7C0231","loaderId":"C9980CD6274A43FBE9998E7B5E95629B","url":"https://example.com/","domainAndRegistry":"example.com","securityOrigin":"https://example.com","mimeType":"text/html","adFrameStatus":{"adFrameType":"none"},"secureContextType":"Secure","crossOriginIsolatedContextType":"NotIsolated","gatedAPIFeatures":[]},"type":"Navigation"},"sessionId":"0F3F699532F154F976242DBF4AC0DFD2"}
    2022/06/01 05:13:04 <- {"method":"DOM.documentUpdated","params":{},"sessionId":"0F3F699532F154F976242DBF4AC0DFD2"}
    2022/06/01 05:13:04 <- {"method":"Network.dataReceived","params":{"requestId":"C9980CD6274A43FBE9998E7B5E95629B","timestamp":55584.707376,"dataLength":1256,"encodedDataLength":0},"sessionId":"0F3F699532F154F976242DBF4AC0DFD2"}
    2022/06/01 05:13:04 <- {"method":"Runtime.executionContextCreated","params":{"context":{"id":2,"origin":"https://example.com","name":"","uniqueId":"964267974370550981.7417314755448916594","auxData":{"isDefault":true,"type":"default","frameId":"412DC8003B59C78196B600AFBB7C0231"}}},"sessionId":"0F3F699532F154F976242DBF4AC0DFD2"}
    2022/06/01 05:13:04 -> {"id":15,"sessionId":"0F3F699532F154F976242DBF4AC0DFD2","method":"DOM.getDocument","params":{}}
    2022/06/01 05:13:04 <- {"method":"Network.loadingFinished","params":{"requestId":"C9980CD6274A43FBE9998E7B5E95629B","timestamp":55584.699583,"encodedDataLength":850,"shouldReportCorbBlocking":false},"sessionId":"0F3F699532F154F976242DBF4AC0DFD2"}
    2022/06/01 05:13:04 <- {"method":"CSS.styleSheetAdded","params":{"header":{"styleSheetId":"38.0","frameId":"412DC8003B59C78196B600AFBB7C0231","sourceURL":"https://example.com/","origin":"regular","title":"","ownerNode":9,"disabled":false,"isInline":true,"isMutable":false,"isConstructed":false,"startLine":8,"startColumn":27,"length":650,"endLine":34,"endColumn":4}},"sessionId":"0F3F699532F154F976242DBF4AC0DFD2"}
    2022/06/01 05:13:04 <- {"id":15,"result":{"root":{"nodeId":1,"backendNodeId":3,"nodeType":9,"nodeName":"#document","localName":"","nodeValue":"","childNodeCount":2,"children":[{"nodeId":2,"parentId":1,"backendNodeId":10,"nodeType":10,"nodeName":"html","localName":"","nodeValue":"","publicId":"","systemId":""},{"nodeId":3,"parentId":1,"backendNodeId":4,"nodeType":1,"nodeName":"HTML","localName":"html","nodeValue":"","childNodeCount":2,"children":[{"nodeId":4,"parentId":3,"backendNodeId":11,"nodeType":1,"nodeName":"HEAD","localName":"head","nodeValue":"","childNodeCount":5,"attributes":[]},{"nodeId":5,"parentId":3,"backendNodeId":12,"nodeType":1,"nodeName":"BODY","localName":"body","nodeValue":"","childNodeCount":1,"attributes":[]}],"attributes":[],"frameId":"412DC8003B59C78196B600AFBB7C0231"}],"documentURL":"https://example.com/","baseURL":"https://example.com/","xmlVersion":"","compatibilityMode":"NoQuirksMode"}},"sessionId":"0F3F699532F154F976242DBF4AC0DFD2"}
    2022/06/01 05:13:04 <- {"method":"Page.loadEventFired","params":{"timestamp":55584.726037},"sessionId":"0F3F699532F154F976242DBF4AC0DFD2"}
    2022/06/01 05:13:04 <- {"method":"Page.lifecycleEvent","params":{"frameId":"412DC8003B59C78196B600AFBB7C0231","loaderId":"C9980CD6274A43FBE9998E7B5E95629B","name":"load","timestamp":55584.726037},"sessionId":"0F3F699532F154F976242DBF4AC0DFD2"}
    2022/06/01 05:13:04 <- {"method":"Page.frameStoppedLoading","params":{"frameId":"412DC8003B59C78196B600AFBB7C0231"},"sessionId":"0F3F699532F154F976242DBF4AC0DFD2"}
    2022/06/01 05:13:04 <- {"method":"DOM.documentUpdated","params":{},"sessionId":"0F3F699532F154F976242DBF4AC0DFD2"}
    2022/06/01 05:13:04 <- {"method":"Page.domContentEventFired","params":{"timestamp":55584.726372},"sessionId":"0F3F699532F154F976242DBF4AC0DFD2"}
    2022/06/01 05:13:04 <- {"method":"Page.lifecycleEvent","params":{"frameId":"412DC8003B59C78196B600AFBB7C0231","loaderId":"C9980CD6274A43FBE9998E7B5E95629B","name":"DOMContentLoaded","timestamp":55584.726372},"sessionId":"0F3F699532F154F976242DBF4AC0DFD2"}
    2022/06/01 05:13:04 -> {"id":16,"sessionId":"0F3F699532F154F976242DBF4AC0DFD2","method":"DOM.getDocument","params":{}}
    2022/06/01 05:13:04 <- {"id":16,"result":{"root":{"nodeId":6,"backendNodeId":3,"nodeType":9,"nodeName":"#document","localName":"","nodeValue":"","childNodeCount":2,"children":[{"nodeId":7,"parentId":6,"backendNodeId":10,"nodeType":10,"nodeName":"html","localName":"","nodeValue":"","publicId":"","systemId":""},{"nodeId":8,"parentId":6,"backendNodeId":4,"nodeType":1,"nodeName":"HTML","localName":"html","nodeValue":"","childNodeCount":2,"children":[{"nodeId":9,"parentId":8,"backendNodeId":11,"nodeType":1,"nodeName":"HEAD","localName":"head","nodeValue":"","childNodeCount":5,"attributes":[]},{"nodeId":10,"parentId":8,"backendNodeId":12,"nodeType":1,"nodeName":"BODY","localName":"body","nodeValue":"","childNodeCount":1,"attributes":[]}],"attributes":[],"frameId":"412DC8003B59C78196B600AFBB7C0231"}],"documentURL":"https://example.com/","baseURL":"https://example.com/","xmlVersion":"","compatibilityMode":"NoQuirksMode"}},"sessionId":"0F3F699532F154F976242DBF4AC0DFD2"}
    2022/06/01 05:13:04 -> {"id":17,"sessionId":"0F3F699532F154F976242DBF4AC0DFD2","method":"DOM.performSearch","params":{"query":"body \u003e div \u003e p:nth-child(2)"}}
    2022/06/01 05:13:04 <- {"id":17,"result":{"searchId":"38.1","resultCount":1},"sessionId":"0F3F699532F154F976242DBF4AC0DFD2"}
    2022/06/01 05:13:04 -> {"id":18,"sessionId":"0F3F699532F154F976242DBF4AC0DFD2","method":"DOM.getSearchResults","params":{"searchId":"38.1","fromIndex":0,"toIndex":1}}
    2022/06/01 05:13:04 <- {"method":"DOM.setChildNodes","params":{"parentId":10,"nodes":[{"nodeId":11,"parentId":10,"backendNodeId":5,"nodeType":1,"nodeName":"DIV","localName":"div","nodeValue":"","childNodeCount":3,"attributes":[]}]},"sessionId":"0F3F699532F154F976242DBF4AC0DFD2"}
    2022/06/01 05:13:04 <- {"method":"DOM.setChildNodes","params":{"parentId":11,"nodes":[{"nodeId":12,"parentId":11,"backendNodeId":13,"nodeType":1,"nodeName":"H1","localName":"h1","nodeValue":"","childNodeCount":1,"children":[{"nodeId":13,"parentId":12,"backendNodeId":6,"nodeType":3,"nodeName":"#text","localName":"","nodeValue":"Example Domain"}],"attributes":[]},{"nodeId":14,"parentId":11,"backendNodeId":14,"nodeType":1,"nodeName":"P","localName":"p","nodeValue":"","childNodeCount":1,"children":[{"nodeId":15,"parentId":14,"backendNodeId":7,"nodeType":3,"nodeName":"#text","localName":"","nodeValue":"This domain is for use in illustrative examples in documents. You may use this\n    domain in literature without prior coordination or asking for permission."}],"attributes":[]},{"nodeId":16,"parentId":11,"backendNodeId":15,"nodeType":1,"nodeName":"P","localName":"p","nodeValue":"","childNodeCount":1,"attributes":[]}]},"sessionId":"0F3F699532F154F976242DBF4AC0DFD2"}
    2022/06/01 05:13:04 <- {"id":18,"result":{"nodeIds":[14]},"sessionId":"0F3F699532F154F976242DBF4AC0DFD2"}
    2022/06/01 05:13:04 -> {"id":19,"sessionId":"0F3F699532F154F976242DBF4AC0DFD2","method":"DOM.performSearch","params":{"query":"body \u003e div \u003e p:nth-child(2)"}}
    2022/06/01 05:13:04 <- {"id":19,"result":{"searchId":"38.2","resultCount":1},"sessionId":"0F3F699532F154F976242DBF4AC0DFD2"}
    2022/06/01 05:13:04 -> {"id":20,"sessionId":"0F3F699532F154F976242DBF4AC0DFD2","method":"DOM.getSearchResults","params":{"searchId":"38.2","fromIndex":0,"toIndex":1}}
    2022/06/01 05:13:04 <- {"id":20,"result":{"nodeIds":[14]},"sessionId":"0F3F699532F154F976242DBF4AC0DFD2"}
    2022/06/01 05:13:04 -> {"id":21,"sessionId":"0F3F699532F154F976242DBF4AC0DFD2","method":"DOM.resolveNode","params":{"nodeId":14}}
    2022/06/01 05:13:04 <- {"id":21,"result":{"object":{"type":"object","subtype":"node","className":"HTMLParagraphElement","description":"p","objectId":"5836470857787727263.2.1"}},"sessionId":"0F3F699532F154F976242DBF4AC0DFD2"}
    2022/06/01 05:13:04 -> {"id":22,"sessionId":"0F3F699532F154F976242DBF4AC0DFD2","method":"Runtime.callFunctionOn","params":{"functionDeclaration":"function text() {\n    if (this.offsetWidth || this.offsetHeight || this.getClientRects().length) {\n        return this.innerText;\n    }\n    return '';\n}\n","objectId":"5836470857787727263.2.1","silent":true,"returnByValue":true}}
    2022/06/01 05:13:04 <- {"id":22,"result":{"result":{"type":"string","value":"This domain is for use in illustrative examples in documents. You may use this domain in literature without prior coordination or asking for permission."}},"sessionId":"0F3F699532F154F976242DBF4AC0DFD2"}
    2022/06/01 05:13:04 -> {"id":23,"sessionId":"0F3F699532F154F976242DBF4AC0DFD2","method":"Runtime.releaseObject","params":{"objectId":"5836470857787727263.2.1"}}
    2022/06/01 05:13:04 <- {"id":23,"result":{},"sessionId":"0F3F699532F154F976242DBF4AC0DFD2"}
    This domain is for use in illustrative examples in documents. You may use this domain in literature without prior coordination or asking for permission.
    

    What did you see instead?

    It stuck and timeout (aws lambda timeout).

    Log on aws lambda.

    START RequestId: f829c5ca-e956-4bfe-bef0-dae4f36d50d1 Version: $LATEST
    2022/06/01 04:13:38 -> {"id":1,"method":"Target.setDiscoverTargets","params":{"discover":true}}
    2022/06/01 04:13:38 <- {"method":"Target.targetCreated","params":{"targetInfo":{"targetId":"D86C302EE0B44DAED223378F7C17BB78","type":"page","title":"","url":"about:blank","attached":false,"canAccessOpener":false,"browserContextId":"0C22F0D3B02A5878B60E8F3500944994"}}}
    2022/06/01 04:13:38 <- {"method":"Target.targetCreated","params":{"targetInfo":{"targetId":"b08ebc85-843e-44d0-bcb3-42ea7bd5a146","type":"browser","title":"","url":"","attached":true,"canAccessOpener":false}}}
    2022/06/01 04:13:38 <- {"method":"Target.targetCreated","params":{"targetInfo":{"targetId":"edd4fc71-a027-494d-8a43-eba9239cd0bb","type":"browser","title":"","url":"","attached":false,"canAccessOpener":false}}}
    2022/06/01 04:13:38 <- {"id":1,"result":{}}
    2022/06/01 04:13:38 -> {"id":2,"method":"Target.attachToTarget","params":{"targetId":"D86C302EE0B44DAED223378F7C17BB78","flatten":true}}
    2022/06/01 04:13:38 <- {"method":"Target.targetInfoChanged","params":{"targetInfo":{"targetId":"D86C302EE0B44DAED223378F7C17BB78","type":"page","title":"","url":"","attached":true,"canAccessOpener":false,"browserContextId":"0C22F0D3B02A5878B60E8F3500944994"}}}
    2022/06/01 04:13:38 <- {"method":"Target.attachedToTarget","params":{"sessionId":"26082765C593E944A783B371227B397A","targetInfo":{"targetId":"D86C302EE0B44DAED223378F7C17BB78","type":"page","title":"","url":"","attached":true,"canAccessOpener":false,"browserContextId":"0C22F0D3B02A5878B60E8F3500944994"},"waitingForDebugger":false}}
    2022/06/01 04:13:38 <- {"id":2,"result":{"sessionId":"26082765C593E944A783B371227B397A"}}
    2022/06/01 04:13:38 -> {"id":3,"sessionId":"26082765C593E944A783B371227B397A","method":"Runtime.enable"}
    END RequestId: f829c5ca-e956-4bfe-bef0-dae4f36d50d1
    REPORT RequestId: f829c5ca-e956-4bfe-bef0-dae4f36d50d1	Duration: 20019.61 ms	Billed Duration: 20552 ms	Memory Size: 10240 MB	Max Memory Used: 125 MB	Init Duration: 532.33 ms	
    2022-06-01T04:13:57.470Z f829c5ca-e956-4bfe-bef0-dae4f36d50d1 Task timed out after 20.02 seconds
    
    opened by Andiedie 21
  • add a ModifyCmdFunc ExecAllocatorOption

    add a ModifyCmdFunc ExecAllocatorOption

    ModifyCmdFunc allows a function to be passed which will run on the exec.Cmd used to start the browser. This can be used to prevent chromedp from automatically passing os signals through to managed browsers, and to prevent browsers from being automatically killed on program exit.

    Addresses #670

    opened by pmurley 21
  • Has anyone successfully run chromedp in Lambda?

    Has anyone successfully run chromedp in Lambda?

    What versions are you running?

    github.com/chromedp/chromedp v0.1.3
    go version go1.12.1
    

    What did you do?

    I've written a small headless chromedp utility which runs fine on my laptop (Ubuntu) but it fails in Lambda with this error "fork/exec /usr/bin/google-chrome: operation not permitted".
    This appears to be an issue with the restricted Lambda environment

    What did you expect to see?

    I expected my CLI tool to run successfully.

    What did you see instead?

    The above error.

    opened by matthewcummings 20
  • Have Cancel gracefully close the Browser

    Have Cancel gracefully close the Browser

    I am doing some work collecting code coverage data on Chromium (not JavaScript coverage, but coverage of the browser source/C++). In order for Chromium to correctly generate the output files containing this coverage data, I need it to close gracefully. I believe the DevTools Protocol Browser.close() would work fine.

    1. My understanding is that when you cancel a context, the associated browser is closed with SIGKILL. Is that accurate?

    2. If so, is there an easy way I can circumvent that to close the browser gracefully? I think that right now, this if statement is preventing it. Will there be unintended effects if I just remove that check?

    opened by pmurley 19
  • hide that webdriver is being used

    hide that webdriver is being used

    What versions are you running?

    Lastest

    chrome 79 and chromium 80/81

    using enable-automation invalid, Webdriver detected using javascript

    window.navigator.webdriver
    
    help wanted 
    opened by wanghuiyt 17
  • about without cookies

    about without cookies

    this is i want test url: https://zhidao.baidu.com/search?lm=0&rn=10&pn=0&fr=search&ie=gbk&dyTabStr=null&word=php

    chromedp.Flag("headless", false),   
    

    when i run code, the chrome well open,but i got blank page.

    so,i change my code: 1、i vist https://zhidao.baidu.com/ 2、get the cookies 3、set cookies to https://zhidao.baidu.com/search?lm=0&rn=10&pn=0&fr=search&ie=gbk&dyTabStr=null&word=php if set cookie,work fine. so, i need vist https://zhidao.baidu.com/ each time? have any other solution?

    opened by langziyang 3
  • cdp.Node: Filter Children

    cdp.Node: Filter Children

    It'd be nice to do something similar to https://pkg.go.dev/github.com/PuerkitoBio/goquery#Selection.ChildrenFiltered

    With selector + chromedp.Selector.

    This would simplify recursive parsing.

    needs info 
    opened by jtagcat 1
  • Target.domEvent panics with a concurrent frame read/write

    Target.domEvent panics with a concurrent frame read/write

    What versions are you running?

    v7.2

    What did you do? Include clear steps.

    chromedp.ListenTarget(ctx, func(ev interface{}) {
        switch ev := ev.(type) {
            case *page.EventLifecycleEvent:
    		# stop timeout and log
    	case *runtime.EventConsoleAPICalled:
    		# logging
    	case *runtime.EventExceptionThrown:
    		# more logging
    	}
    })
    chromedp.Run(ctx)
    

    What did you expect to see?

    Logging, standard run behavior

    What did you see instead?

    fatal error: concurrent map read and map write
    
    github.com/chromedp/chromedp.(*Target).domEvent
    	external/com_github_chromedp_chromedp/target.go:362
    

    Digging into target.go, there are 7 occurrences where t.frames is accessed. Most have a surrounding read lock

    https://github.com/chromedp/chromedp/blob/03c1c7402666a6d042354c57bfb8463f704684b9/target.go#L260-L262

    or write lock

    https://github.com/chromedp/chromedp/blob/03c1c7402666a6d042354c57bfb8463f704684b9/target.go#L298-L305

    except for domEvent, on line 362

    https://github.com/chromedp/chromedp/blob/03c1c7402666a6d042354c57bfb8463f704684b9/target.go#L361-L364

    opened by WalkerGriggs 1
  • monitor chrome performance

    monitor chrome performance

    Hello, I would like to ask if chromedp can monitor the performance of each tab, such as cpu usage, memory space, similar to chrome's task manager, I appreciate it and look forward to your answer

    opened by TYtrack 3
  • feat: allow tab to stay opened after context cancel

    feat: allow tab to stay opened after context cancel

    I have a use case in which it is important to not close the tab / browser after the context gets cancelled.

    Therefore I introduced a new ContextOption. Feel free to discuss :-)

    opened by mariusziemke 7
Releases(v0.8.4)
  • v0.8.4(Aug 14, 2022)

    • Update to latest cdproto (106.0.5236.1_10.6.152 definitions)
    • refactor MatchedStyle to resist change of css.GetMatchedStylesForNode #1127
    • Allow to set custom WS URL waiting time #1036
    • fix example in readme #1135
    • ensure the user data directory is removed #1136
    Source code(tar.gz)
    Source code(zip)
  • v0.8.3(Jul 25, 2022)

    • Update to latest cdproto (106.0.5201.1_10.6.8 definitions)
    • Bug fixes related to cdproto changes
    • Fix issues with Poll() #1087
    • Fix hang issues #1076
    • Deflake TestNagivateWhileLoading #1080
    • Fix issue with TestGracefulBrowserShutdown with headless-shell #1078
    Source code(tar.gz)
    Source code(zip)
  • v0.8.2(May 26, 2022)

    • Update to latest cdproto (104.0.5064.1_10.4.3 definitions)
    • Bring back support for Go 1.16 (which is dropped in v0.8.1)

    Note: Go <= 1.15 is not supported due to the use of go:embed

    Source code(tar.gz)
    Source code(zip)
  • v0.8.1(May 5, 2022)

    • Update to latest cdproto (103.0.5029.1_10.3.67 definitions)
    • Code generator cleanup

    Note: It drops support for Go 1.16 (due to the use of go:build).

    Source code(tar.gz)
    Source code(zip)
  • v0.8.0(Mar 21, 2022)

    Minor updates for cdproto, go1.18

    Updates to latest cdproto which includes minor API revisions, and updates GitHub workflows and go.mod for go1.18.

    Source code(tar.gz)
    Source code(zip)
  • v0.7.8(Mar 11, 2022)

  • v0.7.7(Feb 4, 2022)

  • v0.7.6(Nov 26, 2021)

  • v0.7.5(Nov 26, 2021)

  • v0.7.4(Jul 13, 2021)

    • Update to latest cdproto (93.0.4574.1_9.3.334 definitions)
    • Add command line option: IgnoreCertErrors (#809)
    • Upgrade github.com/gobwas/ws to v1.1.0
    • Use CallFunctionOn to execute js on node (#852) Note: This change uses DOM.scrollIntoViewIfNeeded which was introduced in Chromium 82.0.4056.0. So this could be a breaking change for older browsers (see https://github.com/chromedp/chromedp/issues/881#issuecomment-881925149 for more information). Thank you @FournyP for reporting the issue.
    • Make the screenshot actions act the same as Chrome commands (#863)
    • Minor refactoring and source code clean up.
    Source code(tar.gz)
    Source code(zip)
  • v0.7.3(May 27, 2021)

    • Update to latest cdproto (93.0.4522.3_9.3.2 definitions)
      • And always emit deltaX and deltaY for Input.dispatchMouseEvent (https://github.com/chromedp/pdlgen/pull/11)
    • Dispatch input and change events at the chromedp.SetValue() target (#819)
    • Make NewRemoteAllocator accept url without devtools/browser/... (#817)
    • Allow passing nil as res to chromedp.Evaluate() (#816)
    • Add a ModifyCmdFunc ExecAllocatorOption (#674)
    Source code(tar.gz)
    Source code(zip)
  • v0.7.2(May 10, 2021)

    • Update cdproto to 92.0.4501.1_9.2.173 definitions
    • Search chrome binary in different locations on different OS systems. (#811)
    • Expose process of the Browser (#737)
    • Wrap errors with go 1.13 %w verb (#743)
    • Fix comment about websocket timeout in allocate.go (#727)
    Source code(tar.gz)
    Source code(zip)
  • v0.7.1(May 10, 2021)

  • v0.7.0(May 10, 2021)

    • Add crhomedp.FullScreenshot action
    • Minor cleanup to CI
    • Update the list of ignored page events (#780)
    • Rename TranslateUI to Translate in the default disable-features (#796)
    • Update dependencies
    Source code(tar.gz)
    Source code(zip)
  • v0.6.12(May 10, 2021)

    • Auto grows the buffer to workaround the lack of fragmentation support (#782) It depends on a pre-release of github.com/gobwas/ws (v1.1.0-rc.5)
    • Fix some of the issues in the chromedp.Poll action (#778)
    Source code(tar.gz)
    Source code(zip)
  • v0.6.11(May 10, 2021)

    • Call submit/reset from prototype in case they are masked (#803)
    • Update README.md
    • Add Output to example tests
    • Add chromedp.Poll action to imitate puppeteer's page.waitForFunction (#766)
    • Fix GitHub actions
    Source code(tar.gz)
    Source code(zip)
  • v0.6.4(May 10, 2021)

  • v0.6.2(May 10, 2021)

  • v0.6.0(Jan 5, 2021)

  • v0.5.3(Jan 16, 2020)

    Update to latest cdproto dependency, as backward-incompatible name changes were introduced with the latest cdproto-gen variant.

    Note: this update could not be delayed any further, as the cdproto-gen changes had been made well over a month ago, but the changes to cdproto were not pushed out. For projects not using Go modules, this will cause a breakage if cdproto and chromedp are not kept in sync.

    Source code(tar.gz)
    Source code(zip)
  • v0.5.2(Nov 27, 2019)

    • Discard detached targets properly, which was causing hangs in some sites using iframes
    • Stop discarding EventExceptionThrown events for targets
    • Make ExecAllocator more robust, with bufio.Reader and a timeout
    • Stop erroring about EventDownloadWillBegin events
    • Support a user's explicit remote-debugging-port flag
    • Avoid a few potential concurrency issues in edge cases
    Source code(tar.gz)
    Source code(zip)
  • v0.4.2(Oct 5, 2019)

  • v0.4.0(Aug 15, 2019)

    • add WaitNewTarget to grab a tab opened by an existing tab
    • add CombinedOutput to gather the browser's output
    • add ByJSPath to select nodes via a JS expression
    • add Emulate and Device to easily emulate devices
    • mouse actions now use float64 coordinates to improve high-DPI support
    • fix an edge case where FullXPath could return an incorrect path
    • make DefaultExecAllocatorOptions an array, to discourage racy usage
    • give better errors when an unexpected undefined is encountered
    • fix a possible panic if page.Navigate is used directly
    • split Action into many interfaces to organise the docs into sections
    • add a number of examples and improve the documentation
    Source code(tar.gz)
    Source code(zip)
  • v0.3.1(Jun 26, 2019)

    • fix a regression when sending large messages to Chrome over websocket
    • increase the timeout when waiting for new pages to appear, for slow devices
    • avoid hanging in Navigate actions if the context is canceled
    • fix a data race if Navigate was used while a navigation was already happening
    • fix a regression where large queries could deadlock the target handler
    • revert back to using one goroutine per selector, to reduce CPU use when idle
    Source code(tar.gz)
    Source code(zip)
  • v0.3.0(May 13, 2019)

    • allow listening for browser or tab events
    • navigation actions now wait for the frame to load
    • allow using tabs opened by existing tabs
    • fix the screenshot actions to work on high DPIs
    • work with headless-shell containers out of the box
    Source code(tar.gz)
    Source code(zip)
  • v0.2.0(Apr 22, 2019)

    • Rewrite API to be context-based
    • Remove all internal timeouts, thanks to contexts
    • Handle all events sequentially, fixing occasional races
    • Redesign the Pool type into an Allocator interface
    • Allow managing multiple tabs per browser
    • Greatly speed up the tests, now taking a few seconds in total
    Source code(tar.gz)
    Source code(zip)
Owner
Chrome DevTools Protocol clients and tools for Go
Chrome DevTools Protocol clients and tools for Go
Chrome DevTools Protocol clients and tools for Go
Package cdp provides type-safe bindings for the Chrome DevTools Protocol (CDP), written in the Go programming language.

cdp Package cdp provides type-safe bindings for the Chrome DevTools Protocol (CDP), written in the Go programming language. The bindings are generated

Mathias Fredriksson 637 Sep 27, 2022
A Devtools driver for web automation and scraping

Overview Documentation | API reference Rod is a high-level driver directly based on DevTools Protocol. It's designed for web automation and scraping.

Rod 2.8k Sep 24, 2022
Selenium Hub successor running browsers within containers. Scalable, immutable, self hosted Selenium-Grid on any platform with single binary.

Selenoid Selenoid is a powerful implementation of Selenium hub using Docker containers to launch browsers. Features One-command Installation Start bro

Aerokube 2.2k Sep 27, 2022
Prototype pollution scanner using headless chrome

plution Prototype pollution scanner using headless chrome What this is Plution is a convenient way to scan at scale for pages that are vulnerable to c

null 139 Jul 21, 2022
Generate PlantUML diagrams from Chrome or Firefox network inspections

hoofli Generate PlantUML diagrams from Chrome or Firefox network inspections This tool reads browser HAR files stored on your local disk and transform

Pascal Dennerly 3 Feb 22, 2022
Drive higher confidence in making changes by detecting large blocks of untested functionality

go-coverage Increase code coverage of Go projects Table of Contents About The Project Getting Started Prerequisites Installation Usage Roadmap About T

Gojek 41 Sep 22, 2022
Easier way to unit test terraform

Unit testing terraform (WIP) Disclaimer Currently, the only way to compare values is using JSON query path and all types are strings. want := terraf

Thiago Nache Carvalho 51 Aug 16, 2022
A clipboard-based mocking framework for Go that gets out of your way.

A clipboard-based mocking framework for Go that gets out of your way. This tool has been built with inspiration lovingly taken from Moq, and fuelled b

Roger Guldbrandsen 40 Jun 13, 2022
Toy-redis-mock - Experimentation with parsing the redis wire protocol from scratch

Overview Simple app for practicing implementing server-side Redis wire protocol

Jeannette Pepin 0 Jan 9, 2022
A faster, simpler way to drive browsers supporting the Chrome DevTools Protocol.

About chromedp Package chromedp is a faster, simpler way to drive browsers supporting the Chrome DevTools Protocol in Go without external dependencies

Chrome DevTools Protocol clients and tools for Go 8.2k Sep 23, 2022
Package cdp provides type-safe bindings for the Chrome DevTools Protocol (CDP), written in the Go programming language.

cdp Package cdp provides type-safe bindings for the Chrome DevTools Protocol (CDP), written in the Go programming language. The bindings are generated

Mathias Fredriksson 637 Sep 27, 2022
Chrome-Password-Dumper - Chrome password dumper written in Go for Linux and Windows

Chrome-Password-Dumper Chrome password dumper written in Go for Linux and Window

null 5 May 25, 2022
htf (Host That File) is a tool to make serving up your favorite pentest tools simpler and faster.

htf htf (Host That File) is a tool to make serving up your favorite pentest tools simpler and faster. All you need to do is populate the htf configura

Alexis Rodriguez 1 Nov 28, 2021
"rsync for cloud storage" - Google Drive, S3, Dropbox, Backblaze B2, One Drive, Swift, Hubic, Wasabi, Google Cloud Storage, Yandex Files

Website | Documentation | Download | Contributing | Changelog | Installation | Forum Rclone Rclone ("rsync for cloud storage") is a command-line progr

rclone 34.7k Sep 24, 2022
azqlite is a lightweight wrapper around Azure's SDK to interact with the Azure Storage Queue service in a simpler and more idiomatic way.

azqlite azqlite is a lightweight wrapper around github.com/Azure/azure-storage-queue-go to interact with the Azure Storage Queue service in a simpler

Jose Garcia 1 Mar 12, 2022
A Devtools driver for web automation and scraping

Overview Documentation | API reference Rod is a high-level driver directly based on DevTools Protocol. It's designed for web automation and scraping.

Rod 2.8k Sep 24, 2022
The web framework for writing faster sites, faster

Gondola The web framework for writing faster sites, faster. Written in Go. View documentation at http://gondolaweb.com. Unless indicated otherwise at

Rainy Cape S.L. 309 Jan 24, 2022
A faster RWLock primitive in Go, 2-3 times faster than RWMutex. A Go implementation of concurrency control algorithm in paper

Go Left Right Concurrency A Go implementation of the left-right concurrency control algorithm in paper <Left-Right - A Concurrency Control Technique w

wangyi 42 Aug 28, 2022
Selenium Hub successor running browsers within containers. Scalable, immutable, self hosted Selenium-Grid on any platform with single binary.

Selenoid Selenoid is a powerful implementation of Selenium hub using Docker containers to launch browsers. Features One-command Installation Start bro

Aerokube 2.2k Sep 27, 2022
Interact with Chromium-based browsers' debug port to view open tabs, installed extensions, and cookies

WhiteChocolateMacademiaNut Description Interacts with Chromium-based browsers' debug port to view open tabs, installed extensions, and cookies. Tested

Justin Bui 90 Sep 20, 2022