A faster, simpler way to drive browsers supporting the Chrome DevTools Protocol.

Chrome DevTools Protocol clients and tools for Go

Last update: Jan 4, 2023

Overview

About chromedp

Package chromedp is a faster, simpler way to drive browsers supporting the Chrome DevTools Protocol in Go without external dependencies (like Selenium or PhantomJS).

Installing

Install in the usual Go way:

go get -u github.com/chromedp/chromedp

Examples

Refer to the GoDoc page for the documentation and examples. Additionally, the examples repository contains more complex examples.

Frequently Asked Questions

I can't see any Chrome browser window

By default, Chrome is run in headless mode. See DefaultExecAllocatorOptions, and an example to override the default options.

I'm seeing "context canceled" errors

When the connection to the browser is lost, chromedp cancels the context, and it may result in this error. This occurs, for example, if the browser is closed manually, or if the browser process has been killed or otherwise terminated.

Chrome exits as soon as my Go program finishes

On Linux, chromedp is configured to avoid leaking resources by force-killing any started Chrome child processes. If you need to launch a long-running Chrome instance, manually start Chrome and connect using RemoteAllocator.

Executing an action without Run results in "invalid context"

By default, a chromedp context does not have an executor, however one can be specified manually if necessary; see issue #326 for an example.

I can't use an Action with Run because it returns many values

Wrap it with an ActionFunc:

chromedp.Run(ctx, chromedp.ActionFunc(func(ctx context.Context) error {
	_, err := domain.SomeAction().Do(ctx)
	return err
}))

I want to use chromedp on a headless environment

The simplest way is to run the Go program that uses chromedp inside the chromedp/headless-shell image. That image contains headless-shell, a smaller headless build of Chrome, which chromedp is able to find out of the box.

Resources

chromedp: A New Way to Drive the Web - GopherCon SG 2017 talk
Chrome DevTools Protocol - Chrome DevTools Protocol Domain documentation
chromedp examples - various chromedp examples
github.com/chromedp/cdproto - GoDoc listing for the CDP domains used by chromedp
github.com/chromedp/cdproto-gen - tool used to generate cdproto
github.com/chromedp/chromedp-proxy - a simple CDP proxy for logging CDP clients and browsers

Comments

domEvent: timeout waiting for node

This is more a question than a bug report.

When a dom event listener is about to wait for a node and the node gets concurrently removed (or its parent), WaitNode fails with timeout. The problem here is when that happens, the whole action handlers stale for 10 seconds. At least this is what I think happens. The behaviour is random, if WaitNode finishes before it's removed, then everything works as expected (I think also Go scheduler contributes to the racy behaviour).

Debug log for timeout scenario:

scrap.go:91: -> {"method":"DOM.childNodeInserted","params":{"parentNodeId":7,"previousNodeId":8,"node":{"nodeId":9,"backendNodeId":9,"nodeType":1,"nodeName":"DIV","localName":"div","nodeValue":"","childNodeCount":0,"attributes":[]}}}
scrap.go:91: -> {"method":"DOM.attributeModified","params":{"nodeId":9,"name":"id","value":"someotherid"}}
scrap.go:91: -> {"method":"DOM.childNodeInserted","params":{"parentNodeId":7,"previousNodeId":0,"node":{"nodeId":11,"backendNodeId":11,"nodeType":1,"nodeName":"BODY","localName":"body","nodeValue":"","childNodeCount":1,"attributes":["style","visibility: hidden; width: 0px; height: 0px; border: 0px; margin: 0px; background: none;"]}}}
scrap.go:91: -> {"method":"DOM.childNodeCountUpdated","params":{"nodeId":11,"childNodeCount":0}}
scrap.go:91: -> {"method":"DOM.childNodeRemoved","params":{"parentNodeId":7,"nodeId":9}}
scrap.go:91: -> {"method":"DOM.childNodeRemoved","params":{"parentNodeId":7,"nodeId":11}}
scrap.go:91: could not perform (attributeModified) operation on node 9 (wait node): timeout waiting for node `9`
scrap.go:91: could not perform (childNodeCountUpdated) operation on node 11 (wait node): timeout waiting for node `11`

I was thinking about fixing this (the 10s stales are really bugging me) by introducing some kind of context cancellation / listening channel, so we break out of waiting loop in WaitNode when the node we're waiting for gets removed.

Wdyt?

opened by rjeczalik 32

won't work out of the box with a headless-shell container running as root

What versions are you running?

$ go list -m github.com/chromedp/chromedp
github.com/chromedp/chromedp v0.1.4-0.20190409142133-92a77355f6df
$ chromium --version
Chromium 73.0.3683.75 built on Debian 9.8, running on Debian 9.8
$ go version
go version go1.12.3 linux/amd64

What did you do?

I wrote the following test

package main

import (
        "context"
        "testing"
        "time"

        "github.com/chromedp/chromedp"
)

func Test(t *testing.T) {
        ctx, cancel := chromedp.NewContext(
                context.Background(),
                chromedp.WithLogf(t.Logf),
        )
        defer cancel()

        ctx, cancel = context.WithTimeout(ctx, 10*time.Second)
        defer cancel()

        err := chromedp.Run(ctx,
                chromedp.Navigate("http://localhost:3000/"),
        )
        if err != nil {
                t.Fatal(err)
        }
}

and tried to go test.

What did you expect to see?

go test succeeds.

What did you see instead?

main_test.go:25: malformed ws or wss URL

and, sometimes, also a panic:

panic: sync: negative WaitGroup counter

goroutine 13 [running]:
sync.(*WaitGroup).Add(0xc000332044, 0xffffffffffffffff)
        /usr/local/go/src/sync/waitgroup.go:74 +0x135
sync.(*WaitGroup).Done(...)
        /usr/local/go/src/sync/waitgroup.go:99
github.com/chromedp/chromedp.(*ExecAllocator).Allocate.func1(0xcdac40, 0xc000332060, 0xc000010888, 0xc000128e01, 0xc0000377c0, 0x1d, 0xc000332000, 0xc0002e4ea0)
        /go/pkg/mod/github.com/chromedp/[email protected]/allocate.go:126 +0xa2
created by github.com/chromedp/chromedp.(*ExecAllocator).Allocate
        /go/pkg/mod/github.com/chromedp/chrom[email protected]/allocate.go:109 +0x740

opened by opennota 30

How to set http proxy user and password in chromedp?

Hi, I know the chromedp could set http proxy on option "runner.Proxy(http://localhost:8000/)", but how to set the proxy's user and password? Thanks

opened by shadow1163 30
An example to capture a screenshot of the full page

Is it possible to get a screenshot of the full page, even when setting a specific width and height to emulate a mobile device?

For example, here is a screenshot using dev tools and an iphone 6.

I'm looking to achieve the same result.

http://i.imgur.com/0Je8sgT.png

So far, I'm getting an image which is cut off at the specified height.

opened by paulm17 28

bypass headless chrome detection

What versions are you running?

$ go list -m github.com/chromedp/chromedp
go list -m: not using modules

$ google-chrome --version
/Applications/Chromium.app/Contents/MacOS/Chromium --version
Chromium 77.0.3830.0

$ go version
go version go1.12.6 darwin/amd64

What did you do? Include clear steps.

package main

import (
	"context"
	"fmt"
	"time"

	"github.com/pkg/errors"

	"github.com/chromedp/cdproto/runtime"
	"github.com/chromedp/chromedp"
)

func main() {
	opts := []chromedp.ExecAllocatorOption{
		chromedp.ExecPath(`/Applications/Chromium.app/Contents/MacOS/Chromium`),
		chromedp.UserAgent("Mozilla/5.0 (Macintosh; Intel Mac OS X 10_14_5) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/77.0.3830.0 Safari/537.36"),
		chromedp.WindowSize(1920, 1080),
		chromedp.NoFirstRun,
		chromedp.NoDefaultBrowserCheck,
		chromedp.Headless,
		chromedp.DisableGPU,
	}

	ctx, cancel := chromedp.NewExecAllocator(context.Background(), opts...)
	defer cancel()

	ctx, cancel = chromedp.NewContext(ctx)
	defer cancel()

	err := chromedp.Run(ctx,
		evalJS(`Object.defineProperty(navigator, 'plugins', {get: () => [1, 2, 3, 4, 5, 7, 8, 9]});`),
		evalJS(`navigator.plugins.length`),
		chromedp.Navigate("https://intoli.com/blog/not-possible-to-block-chrome-headless/chrome-headless-test.html"),
		chromedp.ActionFunc(func(ctx context.Context) error {
			time.Sleep(3 * time.Second)
			return nil
		}),
		evalJS(`navigator.plugins.length`),
	)
	if err != nil {
		panic(err)
	}
}

func evalJS(js string) chromedp.Tasks {
	var res *runtime.RemoteObject
	return chromedp.Tasks{
		chromedp.EvaluateAsDevTools(js, &res),
		chromedp.ActionFunc(func(ctx context.Context) error {
			b, err := res.MarshalJSON()
			if err != nil {
				return errors.Wrap(err, "marshal")
			}
			fmt.Println("result: ", string(b))
			return nil
		}),
	}
}

What did you expect to see?

result:  {"type":"object","className":"Navigator","description":"Navigator","objectId":"{\"injectedScriptId\":1,\"id\":1}"}
result:  {"type":"number","value":8,"description":"8"}
result:  {"type":"number","value":8,"description":"8"}

I am trying to bypass headless browser detection as written in this article https://intoli.com/blog/not-possible-to-block-chrome-headless/. This works fine for puppeteer, but doesn't work for chromedp. I need to understand how to execute js code in chromedp, as in puppeteer. Because every time after run chromedp.Navigate() values are reset to default.

It is possible to do with chromedp?

What did you see instead?

result:  {"type":"object","className":"Navigator","description":"Navigator","objectId":"{\"injectedScriptId\":1,\"id\":1}"}
result:  {"type":"number","value":8,"description":"8"}
result:  {"type":"number","value":0,"description":"0"} // after chromedp.Navigate() navigator.plugins.length has default value equal 0

Works fine with puppeteer.

const puppeteer = require('puppeteer');

const userAgent = 'Mozilla/5.0 (X11; Linux x86_64)' +
  'AppleWebKit/537.36 (KHTML, like Gecko) Chrome/64.0.3282.39 Safari/537.36';

(async () => {
  const browser = await puppeteer.launch({headless: true});
  const page = await browser.newPage();
  
  await page.setUserAgent(userAgent);

  await page.evaluateOnNewDocument(() => {
    // Overwrite the `plugins` property to use a custom getter.
    Object.defineProperty(navigator, 'plugins', {
      // This just needs to have `length > 0` for the current test,
      // but we could mock the plugins too if necessary.
      get: () => [1, 2, 3, 4, 5, 6, 7, 8],
    });
  });

  await page.goto('https://intoli.com/blog/not-possible-to-block-chrome-headless/chrome-headless-test.html');
  await page.screenshot({path: 'result.png'});

  await browser.close();
})();

opened by epanov 24

x509: certificate signed by unknown authority

go install in docker image report x509: certificate signed by unknown

What versions are you running?

uname -r

5.4.72-microsoft-standard-WSL2

cat /etc/os-release

PRETTY_NAME="Debian GNU/Linux 10 (buster)" NAME="Debian GNU/Linux" VERSION_ID="10" VERSION="10 (buster)" VERSION_CODENAME=buster ID=debian HOME_URL="https://www.debian.org/" SUPPORT_URL="https://www.debian.org/support" BUG_REPORT_URL="https://bugs.debian.org/"

$ go list -m github.com/chromedp/chromedp go: github.com/chromedp/[email protected]: missing go.sum entry; to add it: go mod download github.com/chromedp/chromedp

$ google-chrome --version sh: 137: google-chrome: not found

$ go version go version go1.16.5 linux/amd64

What did you do? Include clear steps.

docker pull chromedp/headless-shell:74.0.3717.1 //download go1.16.5.linux-amd64.tar.gz and put into local /goinstall directoy docker run -v ${PWD}/goinstall/:/tmp/ -d --name chromedpcontainer chromedp/headless-shell

docker exec -it sh cd /tmp tar -C /usr/local -xzf go1.16.5.linux-amd64.tar.gz export PATH=$PATH:/usr/local/go/bin cd /tmp go mod init helloworld go get -x github.com/chromedp/chromedp

What did you expect to see?

github.com/chromedp/chromedp installed. below added into go.mod

require ( github.com/chromedp/chromedp v0.6.10 )

What did you see instead?

go: github.com/chromedp/[email protected]: Get "https://proxy.golang.org/github.com/chromedp/chromedp/@v/v0.6.10.mod": x509: certificate signed by unknown authority

opened by davidzhongsydney 23

Problem with aws lambda

What versions are you running?

$ go list -m github.com/chromedp/chromedp
github.com/chromedp/chromedp v0.8.2

$ docker pull chromedp/headless-shell:latest
latest: Pulling from chromedp/headless-shell
Digest: sha256:8fe1cbf5aba60474079fedf56867a3a3576340ab21b6e222674edcb6610a584d
Status: Image is up to date for chromedp/headless-shell:latest
docker.io/chromedp/headless-shell:latest

$ docker run --rm golang:alpine go version
go version go1.18.2 linux/amd64

What did you do? Include clear steps.

Here is an example repo. I want to deploy it to aws lambda.

Code snippet

func Handler(_ context.Context, _ json.RawMessage) error {
	opts := []chromedp.ExecAllocatorOption{
		chromedp.NoFirstRun,
		chromedp.NoDefaultBrowserCheck,
		chromedp.NoSandbox,
		chromedp.DisableGPU,
		chromedp.Headless,
		chromedp.Flag("no-zygote", true),
		chromedp.Flag("user-data-dir", "/tmp/chrome-user-data-dir"),
		chromedp.Flag("homedir", "/tmp/chrome-home"),
		chromedp.Flag("data-path", "/tmp/chrome-data-path"),
		chromedp.Flag("disk-cache-dir", "/tmp/chrome-disk-cache-dir"),
		chromedp.Flag("remote-debugging-port", "9222"),
		chromedp.Flag("remote-debugging-address", "0.0.0.0"),
		chromedp.Flag("disable-dev-shm-usage", true),
		chromedp.Flag("enable-features", "NetworkService,NetworkServiceInProcess"),
	}
	ctx, cancel := chromedp.NewExecAllocator(context.Background(), opts...)
	defer cancel()

	ctx, cancel = chromedp.NewContext(ctx, chromedp.WithDebugf(log.Printf))
	defer cancel()

	var content string
	if err := chromedp.Run(ctx, chromedp.Tasks{
		chromedp.Navigate("https://example.com/"),
		chromedp.Text("body > div > p:nth-child(2)", &content),
	}); err != nil {
		log.Fatal(err)
	}
	fmt.Println(content)
	return nil
}

FROM golang:alpine AS builder

RUN apk add git

WORKDIR /app

COPY go.mod go.sum ./
RUN go mod download

COPY . .

RUN go build -o main

FROM chromedp/headless-shell

RUN apt-get update && \
    apt-get install -y dumb-init musl-dev && \
    ln -s /usr/lib/x86_64-linux-musl/libc.so /lib/libc.musl-x86_64.so.1 &&\
    rm -rf /var/lib/apt/lists/*

COPY --from=builder /app/main .

ENTRYPOINT [ "dumb-init", "--" ]
CMD [ "./main" ]

This code (without docker) run perfectly on my M1 macbook, and built docker container run perfectly on AWS EC2.

But when I deploy it to aws lambda, it just stuck after last log 2022/06/01 04:13:38 -> {"id":3,"sessionId":"26082765C593E944A783B371227B397A","method":"Runtime.enable"}.

What did you expect to see?

I can run it on aws lambda.

Log on ec2 running perfectly

2022/06/01 05:13:03 -> {"id":1,"method":"Target.setDiscoverTargets","params":{"discover":true}}
2022/06/01 05:13:03 <- {"method":"Target.targetCreated","params":{"targetInfo":{"targetId":"11278bc8-3ff8-436d-901f-a6404ecb61d1","type":"browser","title":"","url":"","attached":false,"canAccessOpener":false}}}
2022/06/01 05:13:03 <- {"method":"Target.targetCreated","params":{"targetInfo":{"targetId":"412DC8003B59C78196B600AFBB7C0231","type":"page","title":"","url":"about:blank","attached":false,"canAccessOpener":false,"browserContextId":"1488447107EADF5939A51B3C61EA1F3B"}}}
2022/06/01 05:13:03 <- {"method":"Target.targetCreated","params":{"targetInfo":{"targetId":"e42b9ee5-9df7-49b6-b6f2-0314165479f6","type":"browser","title":"","url":"","attached":true,"canAccessOpener":false}}}
2022/06/01 05:13:03 <- {"id":1,"result":{}}
2022/06/01 05:13:03 -> {"id":2,"method":"Target.attachToTarget","params":{"targetId":"412DC8003B59C78196B600AFBB7C0231","flatten":true}}
2022/06/01 05:13:03 <- {"method":"Target.targetInfoChanged","params":{"targetInfo":{"targetId":"412DC8003B59C78196B600AFBB7C0231","type":"page","title":"","url":"about:blank","attached":true,"canAccessOpener":false,"browserContextId":"1488447107EADF5939A51B3C61EA1F3B"}}}
2022/06/01 05:13:03 <- {"method":"Target.attachedToTarget","params":{"sessionId":"0F3F699532F154F976242DBF4AC0DFD2","targetInfo":{"targetId":"412DC8003B59C78196B600AFBB7C0231","type":"page","title":"","url":"about:blank","attached":true,"canAccessOpener":false,"browserContextId":"1488447107EADF5939A51B3C61EA1F3B"},"waitingForDebugger":false}}
2022/06/01 05:13:03 <- {"id":2,"result":{"sessionId":"0F3F699532F154F976242DBF4AC0DFD2"}}
2022/06/01 05:13:04 -> {"id":3,"sessionId":"0F3F699532F154F976242DBF4AC0DFD2","method":"Runtime.enable"}
2022/06/01 05:13:04 <- {"method":"Target.targetInfoChanged","params":{"targetInfo":{"targetId":"412DC8003B59C78196B600AFBB7C0231","type":"page","title":"about:blank","url":"about:blank","attached":true,"canAccessOpener":false,"browserContextId":"1488447107EADF5939A51B3C61EA1F3B"}}}
2022/06/01 05:13:04 <- {"method":"Runtime.executionContextCreated","params":{"context":{"id":1,"origin":"://","name":"","uniqueId":"5877140344713506672.6727992831272504814","auxData":{"isDefault":true,"type":"default","frameId":"412DC8003B59C78196B600AFBB7C0231"}}},"sessionId":"0F3F699532F154F976242DBF4AC0DFD2"}
2022/06/01 05:13:04 <- {"id":3,"result":{},"sessionId":"0F3F699532F154F976242DBF4AC0DFD2"}
2022/06/01 05:13:04 -> {"id":4,"sessionId":"0F3F699532F154F976242DBF4AC0DFD2","method":"Runtime.evaluate","params":{"expression":"self"}}
2022/06/01 05:13:04 <- {"id":4,"result":{"result":{"type":"object","className":"Window","description":"Window","objectId":"5836470857787727263.1.1"}},"sessionId":"0F3F699532F154F976242DBF4AC0DFD2"}
2022/06/01 05:13:04 -> {"id":5,"sessionId":"0F3F699532F154F976242DBF4AC0DFD2","method":"Log.enable"}
2022/06/01 05:13:04 <- {"id":5,"result":{},"sessionId":"0F3F699532F154F976242DBF4AC0DFD2"}
2022/06/01 05:13:04 -> {"id":6,"sessionId":"0F3F699532F154F976242DBF4AC0DFD2","method":"Network.enable","params":{}}
2022/06/01 05:13:04 <- {"id":6,"result":{},"sessionId":"0F3F699532F154F976242DBF4AC0DFD2"}
2022/06/01 05:13:04 -> {"id":7,"sessionId":"0F3F699532F154F976242DBF4AC0DFD2","method":"Inspector.enable"}
2022/06/01 05:13:04 <- {"id":7,"result":{},"sessionId":"0F3F699532F154F976242DBF4AC0DFD2"}
2022/06/01 05:13:04 -> {"id":8,"sessionId":"0F3F699532F154F976242DBF4AC0DFD2","method":"Page.enable"}
2022/06/01 05:13:04 <- {"id":8,"result":{},"sessionId":"0F3F699532F154F976242DBF4AC0DFD2"}
2022/06/01 05:13:04 -> {"id":9,"sessionId":"0F3F699532F154F976242DBF4AC0DFD2","method":"DOM.enable","params":{}}
2022/06/01 05:13:04 <- {"id":9,"result":{},"sessionId":"0F3F699532F154F976242DBF4AC0DFD2"}
2022/06/01 05:13:04 -> {"id":10,"sessionId":"0F3F699532F154F976242DBF4AC0DFD2","method":"CSS.enable"}
2022/06/01 05:13:04 <- {"id":10,"result":{},"sessionId":"0F3F699532F154F976242DBF4AC0DFD2"}
2022/06/01 05:13:04 -> {"id":11,"sessionId":"0F3F699532F154F976242DBF4AC0DFD2","method":"Target.setDiscoverTargets","params":{"discover":true}}
2022/06/01 05:13:04 <- {"method":"Target.targetCreated","params":{"targetInfo":{"targetId":"11278bc8-3ff8-436d-901f-a6404ecb61d1","type":"browser","title":"","url":"","attached":false,"canAccessOpener":false}},"sessionId":"0F3F699532F154F976242DBF4AC0DFD2"}
2022/06/01 05:13:04 <- {"method":"Target.targetCreated","params":{"targetInfo":{"targetId":"412DC8003B59C78196B600AFBB7C0231","type":"page","title":"about:blank","url":"about:blank","attached":true,"canAccessOpener":false,"browserContextId":"1488447107EADF5939A51B3C61EA1F3B"}},"sessionId":"0F3F699532F154F976242DBF4AC0DFD2"}
2022/06/01 05:13:04 <- {"method":"Target.targetCreated","params":{"targetInfo":{"targetId":"e42b9ee5-9df7-49b6-b6f2-0314165479f6","type":"browser","title":"","url":"","attached":true,"canAccessOpener":false}},"sessionId":"0F3F699532F154F976242DBF4AC0DFD2"}
2022/06/01 05:13:04 <- {"id":11,"result":{},"sessionId":"0F3F699532F154F976242DBF4AC0DFD2"}
2022/06/01 05:13:04 -> {"id":12,"sessionId":"0F3F699532F154F976242DBF4AC0DFD2","method":"Target.setAutoAttach","params":{"autoAttach":true,"waitForDebuggerOnStart":false,"flatten":true}}
2022/06/01 05:13:04 <- {"id":12,"result":{},"sessionId":"0F3F699532F154F976242DBF4AC0DFD2"}
2022/06/01 05:13:04 -> {"id":13,"sessionId":"0F3F699532F154F976242DBF4AC0DFD2","method":"Page.setLifecycleEventsEnabled","params":{"enabled":true}}
2022/06/01 05:13:04 <- {"method":"Page.lifecycleEvent","params":{"frameId":"412DC8003B59C78196B600AFBB7C0231","loaderId":"8E275FC88E861B68C971049B227D2C13","name":"commit","timestamp":55583.941934},"sessionId":"0F3F699532F154F976242DBF4AC0DFD2"}
2022/06/01 05:13:04 <- {"method":"Page.lifecycleEvent","params":{"frameId":"412DC8003B59C78196B600AFBB7C0231","loaderId":"8E275FC88E861B68C971049B227D2C13","name":"DOMContentLoaded","timestamp":55583.942021},"sessionId":"0F3F699532F154F976242DBF4AC0DFD2"}
2022/06/01 05:13:04 <- {"method":"Page.lifecycleEvent","params":{"frameId":"412DC8003B59C78196B600AFBB7C0231","loaderId":"8E275FC88E861B68C971049B227D2C13","name":"load","timestamp":55583.942425},"sessionId":"0F3F699532F154F976242DBF4AC0DFD2"}
2022/06/01 05:13:04 <- {"method":"Page.lifecycleEvent","params":{"frameId":"412DC8003B59C78196B600AFBB7C0231","loaderId":"8E275FC88E861B68C971049B227D2C13","name":"networkAlmostIdle","timestamp":55583.942883},"sessionId":"0F3F699532F154F976242DBF4AC0DFD2"}
2022/06/01 05:13:04 <- {"method":"Page.lifecycleEvent","params":{"frameId":"412DC8003B59C78196B600AFBB7C0231","loaderId":"8E275FC88E861B68C971049B227D2C13","name":"networkIdle","timestamp":55583.942883},"sessionId":"0F3F699532F154F976242DBF4AC0DFD2"}
2022/06/01 05:13:04 <- {"id":13,"result":{},"sessionId":"0F3F699532F154F976242DBF4AC0DFD2"}
2022/06/01 05:13:04 -> {"id":14,"sessionId":"0F3F699532F154F976242DBF4AC0DFD2","method":"Page.navigate","params":{"url":"https://example.com/"}}
2022/06/01 05:13:04 <- {"method":"Network.requestWillBeSent","params":{"requestId":"C9980CD6274A43FBE9998E7B5E95629B","loaderId":"C9980CD6274A43FBE9998E7B5E95629B","documentURL":"https://example.com/","request":{"url":"https://example.com/","method":"GET","headers":{"Upgrade-Insecure-Requests":"1","User-Agent":"Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/102.0.5005.63 Safari/537.36"},"mixedContentType":"none","initialPriority":"VeryHigh","referrerPolicy":"strict-origin-when-cross-origin","isSameSite":true},"timestamp":55583.963522,"wallTime":1654060384.050588,"initiator":{"type":"other"},"redirectHasExtraInfo":false,"type":"Document","frameId":"412DC8003B59C78196B600AFBB7C0231","hasUserGesture":false},"sessionId":"0F3F699532F154F976242DBF4AC0DFD2"}
2022/06/01 05:13:04 <- {"method":"Network.requestWillBeSentExtraInfo","params":{"requestId":"C9980CD6274A43FBE9998E7B5E95629B","associatedCookies":[],"headers":{":authority":"example.com",":method":"GET",":path":"/",":scheme":"https","accept":"text/html,application/xhtml+xml,application/xml;q=0.9,image/avif,image/webp,image/apng,*/*;q=0.8,application/signed-exchange;v=b3;q=0.9","accept-encoding":"gzip, deflate, br","accept-language":"en-US@posix","sec-fetch-dest":"document","sec-fetch-mode":"navigate","sec-fetch-site":"none","sec-fetch-user":"?1","upgrade-insecure-requests":"1","user-agent":"Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/102.0.5005.63 Safari/537.36"},"connectTiming":{"requestTime":55583.965293}},"sessionId":"0F3F699532F154F976242DBF4AC0DFD2"}
2022/06/01 05:13:04 <- {"method":"Network.responseReceivedExtraInfo","params":{"requestId":"C9980CD6274A43FBE9998E7B5E95629B","blockedCookies":[],"headers":{"accept-ranges":"bytes","age":"267163","cache-control":"max-age=604800","content-encoding":"gzip","content-length":"648","content-type":"text/html; charset=UTF-8","date":"Wed, 01 Jun 2022 05:13:04 GMT","etag":"\"3147526947\"","expires":"Wed, 08 Jun 2022 05:13:04 GMT","last-modified":"Thu, 17 Oct 2019 07:18:26 GMT","server":"ECS (oxr/8370)","vary":"Accept-Encoding","x-cache":"HIT"},"resourceIPAddressSpace":"Public","statusCode":200},"sessionId":"0F3F699532F154F976242DBF4AC0DFD2"}
2022/06/01 05:13:04 <- {"method":"Network.responseReceived","params":{"requestId":"C9980CD6274A43FBE9998E7B5E95629B","loaderId":"C9980CD6274A43FBE9998E7B5E95629B","timestamp":55584.700562,"type":"Document","response":{"url":"https://example.com/","status":200,"statusText":"","headers":{"accept-ranges":"bytes","age":"267163","cache-control":"max-age=604800","content-encoding":"gzip","content-length":"648","content-type":"text/html; charset=UTF-8","date":"Wed, 01 Jun 2022 05:13:04 GMT","etag":"\"3147526947\"","expires":"Wed, 08 Jun 2022 05:13:04 GMT","last-modified":"Thu, 17 Oct 2019 07:18:26 GMT","server":"ECS (oxr/8370)","vary":"Accept-Encoding","x-cache":"HIT"},"mimeType":"text/html","connectionReused":false,"connectionId":13,"remoteIPAddress":"93.184.216.34","remotePort":443,"fromDiskCache":false,"fromServiceWorker":false,"fromPrefetchCache":false,"encodedDataLength":193,"timing":{"requestTime":55583.965293,"proxyStart":-1,"proxyEnd":-1,"dnsStart":23.354,"dnsEnd":24.929,"connectStart":24.929,"connectEnd":562.419,"sslStart":194.584,"sslEnd":562.39,"workerStart":-1,"workerReady":-1,"workerFetchStart":-1,"workerRespondWithSettled":-1,"sendStart":562.684,"sendEnd":562.847,"pushStart":0,"pushEnd":0,"receiveHeadersEnd":733.232},"responseTime":1.654060384785279e+12,"protocol":"h2","securityState":"secure","securityDetails":{"protocol":"TLS 1.3","keyExchange":"","keyExchangeGroup":"P-256","cipher":"AES_256_GCM","certificateId":0,"subjectName":"www.example.org","sanList":["www.example.org","example.net","example.edu","example.com","example.org","www.example.com","www.example.edu","www.example.net"],"issuer":"DigiCert TLS RSA SHA256 2020 CA1","validFrom":1647216000,"validTo":1678838399,"signedCertificateTimestampList":[],"certificateTransparencyCompliance":"unknown"}},"hasExtraInfo":true,"frameId":"412DC8003B59C78196B600AFBB7C0231"},"sessionId":"0F3F699532F154F976242DBF4AC0DFD2"}
2022/06/01 05:13:04 <- {"id":14,"result":{"frameId":"412DC8003B59C78196B600AFBB7C0231","loaderId":"C9980CD6274A43FBE9998E7B5E95629B"},"sessionId":"0F3F699532F154F976242DBF4AC0DFD2"}
2022/06/01 05:13:04 <- {"method":"Page.lifecycleEvent","params":{"frameId":"412DC8003B59C78196B600AFBB7C0231","loaderId":"8E275FC88E861B68C971049B227D2C13","name":"networkAlmostIdle","timestamp":55583.942883},"sessionId":"0F3F699532F154F976242DBF4AC0DFD2"}
2022/06/01 05:13:04 <- {"method":"Page.lifecycleEvent","params":{"frameId":"412DC8003B59C78196B600AFBB7C0231","loaderId":"8E275FC88E861B68C971049B227D2C13","name":"networkIdle","timestamp":55583.942883},"sessionId":"0F3F699532F154F976242DBF4AC0DFD2"}
2022/06/01 05:13:04 <- {"method":"Page.frameStartedLoading","params":{"frameId":"412DC8003B59C78196B600AFBB7C0231"},"sessionId":"0F3F699532F154F976242DBF4AC0DFD2"}
2022/06/01 05:13:04 <- {"method":"Page.lifecycleEvent","params":{"frameId":"412DC8003B59C78196B600AFBB7C0231","loaderId":"C9980CD6274A43FBE9998E7B5E95629B","name":"init","timestamp":55584.704893},"sessionId":"0F3F699532F154F976242DBF4AC0DFD2"}
2022/06/01 05:13:04 <- {"method":"Target.targetInfoChanged","params":{"targetInfo":{"targetId":"412DC8003B59C78196B600AFBB7C0231","type":"page","title":"https://example.com","url":"https://example.com/","attached":true,"canAccessOpener":false,"browserContextId":"1488447107EADF5939A51B3C61EA1F3B"}}}
2022/06/01 05:13:04 <- {"method":"Target.targetInfoChanged","params":{"targetInfo":{"targetId":"412DC8003B59C78196B600AFBB7C0231","type":"page","title":"https://example.com","url":"https://example.com/","attached":true,"canAccessOpener":false,"browserContextId":"1488447107EADF5939A51B3C61EA1F3B"}},"sessionId":"0F3F699532F154F976242DBF4AC0DFD2"}
2022/06/01 05:13:04 <- {"method":"Runtime.executionContextsCleared","params":{},"sessionId":"0F3F699532F154F976242DBF4AC0DFD2"}
2022/06/01 05:13:04 <- {"method":"Page.frameNavigated","params":{"frame":{"id":"412DC8003B59C78196B600AFBB7C0231","loaderId":"C9980CD6274A43FBE9998E7B5E95629B","url":"https://example.com/","domainAndRegistry":"example.com","securityOrigin":"https://example.com","mimeType":"text/html","adFrameStatus":{"adFrameType":"none"},"secureContextType":"Secure","crossOriginIsolatedContextType":"NotIsolated","gatedAPIFeatures":[]},"type":"Navigation"},"sessionId":"0F3F699532F154F976242DBF4AC0DFD2"}
2022/06/01 05:13:04 <- {"method":"DOM.documentUpdated","params":{},"sessionId":"0F3F699532F154F976242DBF4AC0DFD2"}
2022/06/01 05:13:04 <- {"method":"Network.dataReceived","params":{"requestId":"C9980CD6274A43FBE9998E7B5E95629B","timestamp":55584.707376,"dataLength":1256,"encodedDataLength":0},"sessionId":"0F3F699532F154F976242DBF4AC0DFD2"}
2022/06/01 05:13:04 <- {"method":"Runtime.executionContextCreated","params":{"context":{"id":2,"origin":"https://example.com","name":"","uniqueId":"964267974370550981.7417314755448916594","auxData":{"isDefault":true,"type":"default","frameId":"412DC8003B59C78196B600AFBB7C0231"}}},"sessionId":"0F3F699532F154F976242DBF4AC0DFD2"}
2022/06/01 05:13:04 -> {"id":15,"sessionId":"0F3F699532F154F976242DBF4AC0DFD2","method":"DOM.getDocument","params":{}}
2022/06/01 05:13:04 <- {"method":"Network.loadingFinished","params":{"requestId":"C9980CD6274A43FBE9998E7B5E95629B","timestamp":55584.699583,"encodedDataLength":850,"shouldReportCorbBlocking":false},"sessionId":"0F3F699532F154F976242DBF4AC0DFD2"}
2022/06/01 05:13:04 <- {"method":"CSS.styleSheetAdded","params":{"header":{"styleSheetId":"38.0","frameId":"412DC8003B59C78196B600AFBB7C0231","sourceURL":"https://example.com/","origin":"regular","title":"","ownerNode":9,"disabled":false,"isInline":true,"isMutable":false,"isConstructed":false,"startLine":8,"startColumn":27,"length":650,"endLine":34,"endColumn":4}},"sessionId":"0F3F699532F154F976242DBF4AC0DFD2"}
2022/06/01 05:13:04 <- {"id":15,"result":{"root":{"nodeId":1,"backendNodeId":3,"nodeType":9,"nodeName":"#document","localName":"","nodeValue":"","childNodeCount":2,"children":[{"nodeId":2,"parentId":1,"backendNodeId":10,"nodeType":10,"nodeName":"html","localName":"","nodeValue":"","publicId":"","systemId":""},{"nodeId":3,"parentId":1,"backendNodeId":4,"nodeType":1,"nodeName":"HTML","localName":"html","nodeValue":"","childNodeCount":2,"children":[{"nodeId":4,"parentId":3,"backendNodeId":11,"nodeType":1,"nodeName":"HEAD","localName":"head","nodeValue":"","childNodeCount":5,"attributes":[]},{"nodeId":5,"parentId":3,"backendNodeId":12,"nodeType":1,"nodeName":"BODY","localName":"body","nodeValue":"","childNodeCount":1,"attributes":[]}],"attributes":[],"frameId":"412DC8003B59C78196B600AFBB7C0231"}],"documentURL":"https://example.com/","baseURL":"https://example.com/","xmlVersion":"","compatibilityMode":"NoQuirksMode"}},"sessionId":"0F3F699532F154F976242DBF4AC0DFD2"}
2022/06/01 05:13:04 <- {"method":"Page.loadEventFired","params":{"timestamp":55584.726037},"sessionId":"0F3F699532F154F976242DBF4AC0DFD2"}
2022/06/01 05:13:04 <- {"method":"Page.lifecycleEvent","params":{"frameId":"412DC8003B59C78196B600AFBB7C0231","loaderId":"C9980CD6274A43FBE9998E7B5E95629B","name":"load","timestamp":55584.726037},"sessionId":"0F3F699532F154F976242DBF4AC0DFD2"}
2022/06/01 05:13:04 <- {"method":"Page.frameStoppedLoading","params":{"frameId":"412DC8003B59C78196B600AFBB7C0231"},"sessionId":"0F3F699532F154F976242DBF4AC0DFD2"}
2022/06/01 05:13:04 <- {"method":"DOM.documentUpdated","params":{},"sessionId":"0F3F699532F154F976242DBF4AC0DFD2"}
2022/06/01 05:13:04 <- {"method":"Page.domContentEventFired","params":{"timestamp":55584.726372},"sessionId":"0F3F699532F154F976242DBF4AC0DFD2"}
2022/06/01 05:13:04 <- {"method":"Page.lifecycleEvent","params":{"frameId":"412DC8003B59C78196B600AFBB7C0231","loaderId":"C9980CD6274A43FBE9998E7B5E95629B","name":"DOMContentLoaded","timestamp":55584.726372},"sessionId":"0F3F699532F154F976242DBF4AC0DFD2"}
2022/06/01 05:13:04 -> {"id":16,"sessionId":"0F3F699532F154F976242DBF4AC0DFD2","method":"DOM.getDocument","params":{}}
2022/06/01 05:13:04 <- {"id":16,"result":{"root":{"nodeId":6,"backendNodeId":3,"nodeType":9,"nodeName":"#document","localName":"","nodeValue":"","childNodeCount":2,"children":[{"nodeId":7,"parentId":6,"backendNodeId":10,"nodeType":10,"nodeName":"html","localName":"","nodeValue":"","publicId":"","systemId":""},{"nodeId":8,"parentId":6,"backendNodeId":4,"nodeType":1,"nodeName":"HTML","localName":"html","nodeValue":"","childNodeCount":2,"children":[{"nodeId":9,"parentId":8,"backendNodeId":11,"nodeType":1,"nodeName":"HEAD","localName":"head","nodeValue":"","childNodeCount":5,"attributes":[]},{"nodeId":10,"parentId":8,"backendNodeId":12,"nodeType":1,"nodeName":"BODY","localName":"body","nodeValue":"","childNodeCount":1,"attributes":[]}],"attributes":[],"frameId":"412DC8003B59C78196B600AFBB7C0231"}],"documentURL":"https://example.com/","baseURL":"https://example.com/","xmlVersion":"","compatibilityMode":"NoQuirksMode"}},"sessionId":"0F3F699532F154F976242DBF4AC0DFD2"}
2022/06/01 05:13:04 -> {"id":17,"sessionId":"0F3F699532F154F976242DBF4AC0DFD2","method":"DOM.performSearch","params":{"query":"body \u003e div \u003e p:nth-child(2)"}}
2022/06/01 05:13:04 <- {"id":17,"result":{"searchId":"38.1","resultCount":1},"sessionId":"0F3F699532F154F976242DBF4AC0DFD2"}
2022/06/01 05:13:04 -> {"id":18,"sessionId":"0F3F699532F154F976242DBF4AC0DFD2","method":"DOM.getSearchResults","params":{"searchId":"38.1","fromIndex":0,"toIndex":1}}
2022/06/01 05:13:04 <- {"method":"DOM.setChildNodes","params":{"parentId":10,"nodes":[{"nodeId":11,"parentId":10,"backendNodeId":5,"nodeType":1,"nodeName":"DIV","localName":"div","nodeValue":"","childNodeCount":3,"attributes":[]}]},"sessionId":"0F3F699532F154F976242DBF4AC0DFD2"}
2022/06/01 05:13:04 <- {"method":"DOM.setChildNodes","params":{"parentId":11,"nodes":[{"nodeId":12,"parentId":11,"backendNodeId":13,"nodeType":1,"nodeName":"H1","localName":"h1","nodeValue":"","childNodeCount":1,"children":[{"nodeId":13,"parentId":12,"backendNodeId":6,"nodeType":3,"nodeName":"#text","localName":"","nodeValue":"Example Domain"}],"attributes":[]},{"nodeId":14,"parentId":11,"backendNodeId":14,"nodeType":1,"nodeName":"P","localName":"p","nodeValue":"","childNodeCount":1,"children":[{"nodeId":15,"parentId":14,"backendNodeId":7,"nodeType":3,"nodeName":"#text","localName":"","nodeValue":"This domain is for use in illustrative examples in documents. You may use this\n    domain in literature without prior coordination or asking for permission."}],"attributes":[]},{"nodeId":16,"parentId":11,"backendNodeId":15,"nodeType":1,"nodeName":"P","localName":"p","nodeValue":"","childNodeCount":1,"attributes":[]}]},"sessionId":"0F3F699532F154F976242DBF4AC0DFD2"}
2022/06/01 05:13:04 <- {"id":18,"result":{"nodeIds":[14]},"sessionId":"0F3F699532F154F976242DBF4AC0DFD2"}
2022/06/01 05:13:04 -> {"id":19,"sessionId":"0F3F699532F154F976242DBF4AC0DFD2","method":"DOM.performSearch","params":{"query":"body \u003e div \u003e p:nth-child(2)"}}
2022/06/01 05:13:04 <- {"id":19,"result":{"searchId":"38.2","resultCount":1},"sessionId":"0F3F699532F154F976242DBF4AC0DFD2"}
2022/06/01 05:13:04 -> {"id":20,"sessionId":"0F3F699532F154F976242DBF4AC0DFD2","method":"DOM.getSearchResults","params":{"searchId":"38.2","fromIndex":0,"toIndex":1}}
2022/06/01 05:13:04 <- {"id":20,"result":{"nodeIds":[14]},"sessionId":"0F3F699532F154F976242DBF4AC0DFD2"}
2022/06/01 05:13:04 -> {"id":21,"sessionId":"0F3F699532F154F976242DBF4AC0DFD2","method":"DOM.resolveNode","params":{"nodeId":14}}
2022/06/01 05:13:04 <- {"id":21,"result":{"object":{"type":"object","subtype":"node","className":"HTMLParagraphElement","description":"p","objectId":"5836470857787727263.2.1"}},"sessionId":"0F3F699532F154F976242DBF4AC0DFD2"}
2022/06/01 05:13:04 -> {"id":22,"sessionId":"0F3F699532F154F976242DBF4AC0DFD2","method":"Runtime.callFunctionOn","params":{"functionDeclaration":"function text() {\n    if (this.offsetWidth || this.offsetHeight || this.getClientRects().length) {\n        return this.innerText;\n    }\n    return '';\n}\n","objectId":"5836470857787727263.2.1","silent":true,"returnByValue":true}}
2022/06/01 05:13:04 <- {"id":22,"result":{"result":{"type":"string","value":"This domain is for use in illustrative examples in documents. You may use this domain in literature without prior coordination or asking for permission."}},"sessionId":"0F3F699532F154F976242DBF4AC0DFD2"}
2022/06/01 05:13:04 -> {"id":23,"sessionId":"0F3F699532F154F976242DBF4AC0DFD2","method":"Runtime.releaseObject","params":{"objectId":"5836470857787727263.2.1"}}
2022/06/01 05:13:04 <- {"id":23,"result":{},"sessionId":"0F3F699532F154F976242DBF4AC0DFD2"}
This domain is for use in illustrative examples in documents. You may use this domain in literature without prior coordination or asking for permission.

What did you see instead?

It stuck and timeout (aws lambda timeout).

Log on aws lambda.

START RequestId: f829c5ca-e956-4bfe-bef0-dae4f36d50d1 Version: $LATEST
2022/06/01 04:13:38 -> {"id":1,"method":"Target.setDiscoverTargets","params":{"discover":true}}
2022/06/01 04:13:38 <- {"method":"Target.targetCreated","params":{"targetInfo":{"targetId":"D86C302EE0B44DAED223378F7C17BB78","type":"page","title":"","url":"about:blank","attached":false,"canAccessOpener":false,"browserContextId":"0C22F0D3B02A5878B60E8F3500944994"}}}
2022/06/01 04:13:38 <- {"method":"Target.targetCreated","params":{"targetInfo":{"targetId":"b08ebc85-843e-44d0-bcb3-42ea7bd5a146","type":"browser","title":"","url":"","attached":true,"canAccessOpener":false}}}
2022/06/01 04:13:38 <- {"method":"Target.targetCreated","params":{"targetInfo":{"targetId":"edd4fc71-a027-494d-8a43-eba9239cd0bb","type":"browser","title":"","url":"","attached":false,"canAccessOpener":false}}}
2022/06/01 04:13:38 <- {"id":1,"result":{}}
2022/06/01 04:13:38 -> {"id":2,"method":"Target.attachToTarget","params":{"targetId":"D86C302EE0B44DAED223378F7C17BB78","flatten":true}}
2022/06/01 04:13:38 <- {"method":"Target.targetInfoChanged","params":{"targetInfo":{"targetId":"D86C302EE0B44DAED223378F7C17BB78","type":"page","title":"","url":"","attached":true,"canAccessOpener":false,"browserContextId":"0C22F0D3B02A5878B60E8F3500944994"}}}
2022/06/01 04:13:38 <- {"method":"Target.attachedToTarget","params":{"sessionId":"26082765C593E944A783B371227B397A","targetInfo":{"targetId":"D86C302EE0B44DAED223378F7C17BB78","type":"page","title":"","url":"","attached":true,"canAccessOpener":false,"browserContextId":"0C22F0D3B02A5878B60E8F3500944994"},"waitingForDebugger":false}}
2022/06/01 04:13:38 <- {"id":2,"result":{"sessionId":"26082765C593E944A783B371227B397A"}}
2022/06/01 04:13:38 -> {"id":3,"sessionId":"26082765C593E944A783B371227B397A","method":"Runtime.enable"}
END RequestId: f829c5ca-e956-4bfe-bef0-dae4f36d50d1
REPORT RequestId: f829c5ca-e956-4bfe-bef0-dae4f36d50d1	Duration: 20019.61 ms	Billed Duration: 20552 ms	Memory Size: 10240 MB	Max Memory Used: 125 MB	Init Duration: 532.33 ms	
2022-06-01T04:13:57.470Z f829c5ca-e956-4bfe-bef0-dae4f36d50d1 Task timed out after 20.02 seconds

opened by Andiedie 21

add a ModifyCmdFunc ExecAllocatorOption

ModifyCmdFunc allows a function to be passed which will run on the exec.Cmd used to start the browser. This can be used to prevent chromedp from automatically passing os signals through to managed browsers, and to prevent browsers from being automatically killed on program exit.

Addresses #670

opened by pmurley 21
Has anyone successfully run chromedp in Lambda?
What versions are you running?

github.com/chromedp/chromedp v0.1.3 go version go1.12.1

What did you do?

I've written a small headless chromedp utility which runs fine on my laptop (Ubuntu) but it fails in Lambda with this error "fork/exec /usr/bin/google-chrome: operation not permitted".
This appears to be an issue with the restricted Lambda environment

What did you expect to see?

I expected my CLI tool to run successfully.

What did you see instead?

The above error.
opened by matthewcummings 20
PDF & emulation.SetDefaultBackgroundColorOverride: does it work?
Hello 👋 ,

I'm currently implementing a feature in Gotenberg that mimic the omitBackground feature from Puppeteer.

I did exactly the same thing as they do (i.e., using emulation to override the default white background), but without success so far.

I was wondering if you may have some insights to share, because I'm kinda lost here 😄

What versions are you running?

$ go list -m github.com/chromedp/chromedp github.com/chromedp/chromedp v0.8.6 $ google-chrome --version Google Chrome 106.0.5249.119 $ go version go version go1.19 darwin/amd64

What did you do? Include clear steps.

I added the following ActionFunc in my list of chromedp.Tasks:

chromedp.ActionFunc(func(ctx context.Context) error { // See https://github.com/gotenberg/gotenberg/issues/226. if !options.OmitBackground { logger.Debug("default white background not hidden") return nil } logger.Debug("hide default white background") err := emulation.SetDefaultBackgroundColorOverride().WithColor( &cdp.RGBA{ R: 0, G: 0, B: 0, A: 0, }).Do(ctx) if err == nil { return nil } return fmt.Errorf("hide default white background: %w", err) }),

You may find the related PR here: https://github.com/gotenberg/gotenberg/pull/528

What did you expect to see?

I did expect the PDFs to have a transparent background.

What did you see instead?

The PDFs still have a white background, only the size differ (a few bytes).
opened by gulien 19
Have Cancel gracefully close the Browser
I am doing some work collecting code coverage data on Chromium (not JavaScript coverage, but coverage of the browser source/C++). In order for Chromium to correctly generate the output files containing this coverage data, I need it to close gracefully. I believe the DevTools Protocol Browser.close() would work fine.

My understanding is that when you cancel a context, the associated browser is closed with SIGKILL. Is that accurate?

If so, is there an easy way I can circumvent that to close the browser gracefully? I think that right now, this if statement is preventing it. Will there be unintended effects if I just remove that check?
opened by pmurley 19

Click or WaitNotVisible/Present Not Working for Page

On this Javascript app driven page, if I run this in the Javascript console,

document.querySelector('#gallery_next')

it will change all IDs matching .gallery_item, thus signifying that navigation to the next page has occurred.

However, when I try the same Click command using chromedp, it times out waiting for an element matching what should be changing ID to be no longer visible or present (I have tried both).

package main

import (
	"context"
	"fmt"
	"time"

	"github.com/chromedp/chromedp"
)

func handleErr(err error) {
	if err != nil {
		panic(err)
	}
}

func main() {
	ctx, cancel := chromedp.NewContext(context.Background())
	defer cancel()
	time.AfterFunc(10*time.Second, func() {
		fmt.Println("check timed out")
		cancel()
	})

	var id string
	err := chromedp.Run(ctx,
		chromedp.Navigate("https://orgasmsoundlibrary.com/#gallery"),
		chromedp.WaitVisible(`.gallery_item`),
		chromedp.AttributeValue(`.gallery_item`, "id", &id, nil),
	)
	handleErr(err)

	err = chromedp.Run(ctx,
		chromedp.Click("#gallery_next"),
		// These both always time out
		// chromedp.WaitNotVisible("#"+id),
		chromedp.WaitNotPresent("#"+id),
	)
	handleErr(err)
}

opened by drgrib 11

add exposefunc feature

In some scenarios, need to bind go func to the browser environment. when js called, the function executes in go process and returns a Promise refer https://pptr.dev/api/puppeteer.page.exposefunction/

opened by hillguo 1

Cepture screenshot returns full height screenshot instead of viewport height

Hey! I'm using chromedp to capture the website's screenshot for the given viewport. But when I call chromedp.CaptureScreenshot(&buf), full height screenshot is taken, instead of just the viewport height.

What versions are you running?

$ go list -m github.com/chromedp/chromedp
github.com/chromedp/chromedp v0.8.6
$ google-chrome --version
Chromium 108.0.5359.94 built on Debian 11.5, running on Debian 11.5
$ go version
go version go1.18.9 linux/arm64

What did you do? Include clear steps.

package crawler

import (
    "context"
    "github.com/chromedp/chromedp"
    "log"
    "time"
)

type Crawler struct {
    URL            string
    ViewportWidth  int64
    ViewportHeight int64
}

func (c *Crawler) GetScreenshot() (error, []byte) {
    ctx, cancel := chromedp.NewContext(
        context.Background(),
    )
    defer cancel()

    ctx, cancel = context.WithTimeout(ctx, 10*time.Second)
    defer cancel()

    var buf []byte

    println(c.ViewportWidth, c.ViewportHeight)

    if err := chromedp.Run(ctx,
        chromedp.EmulateViewport(c.ViewportWidth, c.ViewportHeight),
        chromedp.Navigate(c.URL),
        chromedp.CaptureScreenshot(&buf),
    ); err != nil {
        log.Println(err)
        return err, nil
    }

    return nil, buf
}

You can also check the source code here: https://github.com/marketron-app/image-engine/blob/main/crawler/crawler.go

What did you expect to see?

Get a screenshot with height of a viewport.

What did you see instead?

Got a screenshot with full height of the website.

opened by tavsec 0

How to reuse tab

What versions are you running?

$ go list -m github.com/chromedp/chromedp
github.com/chromedp/chromedp v0.8.6hromedp
$ google-chrome --version
Google Chrome 108.0.5359.94
$ go version
19

What did you do? Include clear steps.

chromedp.Run(ctx,

		chromedp.ActionFunc(func(c context.Context) error {
			for u := range urls {
				p := PlayStoreData{}
				p.URL = u
				chromedp.Navigate(u).Do(c)
							
					chromedp.Text(`div[class="TT9eCd"]`, &p.Rating, chromedp.BySearch, chromedp.NodeVisible).Do(c)
					chromedp.Text(`div[class=xg1aie]`, &p.UpdatedDate, chromedp.BySearch, chromedp.NodeVisible).Do(c)
					chromedp.Click(`button[class="VfPpkd-Bz112c-LgbsSe yHy1rc eT1oJ QDwDD mN1ivc VxpoF"]`, chromedp.ByQuery).Do(c)
					chromedp.Text(`(//div[.="Version"])[1]//following-sibling::div`, &p.Version, chromedp.BySearch, chromedp.NodeVisible).Do(c)

				}
				data <- p
			}

			return nil
		}),

What did you expect to see?

tchromedp reusing the same tab

What did you see instead?

after a few successful navigation on samepage the tab crashes/closes.

opened by Rishikesh01 3

Releases(v0.8.6)

v0.8.6(Nov 2, 2022)
add RetryInterval option to query actions (#1159) (Thank you @taylorchu)

Update to latest cdproto (108.0.5320.1_10.8.51 definitions)

Source code(tar.gz)
Source code(zip)
v0.8.5(Nov 2, 2022)
Update to latest cdproto (107.0.5263.1_10.7.36 definitions)

Source code(tar.gz)
Source code(zip)
v0.8.4(Aug 14, 2022)
Update to latest cdproto (106.0.5236.1_10.6.152 definitions)

refactor MatchedStyle to resist change of css.GetMatchedStylesForNode #1127

Allow to set custom WS URL waiting time #1036

fix example in readme #1135

ensure the user data directory is removed #1136

Source code(tar.gz)
Source code(zip)
v0.8.3(Jul 25, 2022)
Update to latest cdproto (106.0.5201.1_10.6.8 definitions)

Bug fixes related to cdproto changes

Fix issues with Poll() #1087

Fix hang issues #1076

Deflake TestNagivateWhileLoading #1080

Fix issue with TestGracefulBrowserShutdown with headless-shell #1078

Source code(tar.gz)
Source code(zip)
v0.8.2(May 26, 2022)
Update to latest cdproto (104.0.5064.1_10.4.3 definitions)

Bring back support for Go 1.16 (which is dropped in v0.8.1)

Note: Go <= 1.15 is not supported due to the use of go:embed
Source code(tar.gz)
Source code(zip)
v0.8.1(May 5, 2022)
Update to latest cdproto (103.0.5029.1_10.3.67 definitions)

Code generator cleanup

Note: It drops support for Go 1.16 (due to the use of go:build).
Source code(tar.gz)
Source code(zip)
v0.8.0(Mar 21, 2022)

Minor updates for cdproto, go1.18

Updates to latest cdproto which includes minor API revisions, and updates GitHub workflows and go.mod for go1.18.
Source code(tar.gz)
Source code(zip)
v0.7.8(Mar 11, 2022)

Updating device list and cdproto (100.0.4895.2_10.0.139 definitions)
Source code(tar.gz)
Source code(zip)
v0.7.7(Feb 4, 2022)
Update to latest cdproto (100.0.4862.2_10.0.51 definitions)

Avoid the unnecessary wait in the first iteration of a loop (#929)

Source code(tar.gz)
Source code(zip)
v0.7.6(Nov 26, 2021)

Point release; updates device profiles. See v0.7.5 for more information.
Source code(tar.gz)
Source code(zip)
v0.7.5(Nov 26, 2021)

Point release; minor bug fixes and dependency updates. Updates to 98.0.4731.1_9.8.101 browser protocol definitions.
Source code(tar.gz)
Source code(zip)
v0.7.4(Jul 13, 2021)
Update to latest cdproto (93.0.4574.1_9.3.334 definitions)

Add command line option: IgnoreCertErrors (#809)

Upgrade github.com/gobwas/ws to v1.1.0

Use CallFunctionOn to execute js on node (#852) Note: This change uses DOM.scrollIntoViewIfNeeded which was introduced in Chromium 82.0.4056.0. So this could be a breaking change for older browsers (see https://github.com/chromedp/chromedp/issues/881#issuecomment-881925149 for more information). Thank you @FournyP for reporting the issue.

Make the screenshot actions act the same as Chrome commands (#863)

Minor refactoring and source code clean up.

Source code(tar.gz)
Source code(zip)
v0.7.3(May 27, 2021)
Update to latest cdproto (93.0.4522.3_9.3.2 definitions)

And always emit deltaX and deltaY for Input.dispatchMouseEvent (https://github.com/chromedp/pdlgen/pull/11)

Dispatch input and change events at the chromedp.SetValue() target (#819)

Make NewRemoteAllocator accept url without devtools/browser/... (#817)

Allow passing nil as res to chromedp.Evaluate() (#816)

Add a ModifyCmdFunc ExecAllocatorOption (#674)

Source code(tar.gz)
Source code(zip)
v0.7.2(May 10, 2021)
Update cdproto to 92.0.4501.1_9.2.173 definitions

Search chrome binary in different locations on different OS systems. (#811)

Expose process of the Browser (#737)

Wrap errors with go 1.13 %w verb (#743)

Fix comment about websocket timeout in allocate.go (#727)

Source code(tar.gz)
Source code(zip)
v0.7.1(May 10, 2021)
Fix typo in the example

Source code(tar.gz)
Source code(zip)
v0.7.0(May 10, 2021)
Add crhomedp.FullScreenshot action

Minor cleanup to CI

Update the list of ignored page events (#780)

Rename TranslateUI to Translate in the default disable-features (#796)

Update dependencies

Source code(tar.gz)
Source code(zip)
v0.6.12(May 10, 2021)
Auto grows the buffer to workaround the lack of fragmentation support (#782) It depends on a pre-release of github.com/gobwas/ws (v1.1.0-rc.5)

Fix some of the issues in the chromedp.Poll action (#778)

Source code(tar.gz)
Source code(zip)
v0.6.11(May 10, 2021)
Call submit/reset from prototype in case they are masked (#803)

Update README.md

Add Output to example tests

Add chromedp.Poll action to imitate puppeteer's page.waitForFunction (#766)

Fix GitHub actions

Source code(tar.gz)
Source code(zip)
v0.6.4(May 10, 2021)
Do not set pdeathsig to the process on Lambda (#720)

Source code(tar.gz)
Source code(zip)
v0.6.2(May 10, 2021)
Fix deadlock in TestPrematureCancel (#742)

Fix data race when finding enclosing frame (#741)

Source code(tar.gz)
Source code(zip)
v0.6.0(Jan 5, 2021)

Updated to latest Chromium v89 protocol definitions, and some minor logic fixes.
Source code(tar.gz)
Source code(zip)
v0.5.3(Jan 16, 2020)

Update to latest cdproto dependency, as backward-incompatible name changes were introduced with the latest cdproto-gen variant.

Note: this update could not be delayed any further, as the cdproto-gen changes had been made well over a month ago, but the changes to cdproto were not pushed out. For projects not using Go modules, this will cause a breakage if cdproto and chromedp are not kept in sync.
Source code(tar.gz)
Source code(zip)
v0.5.2(Nov 27, 2019)
Discard detached targets properly, which was causing hangs in some sites using iframes

Stop discarding EventExceptionThrown events for targets

Make ExecAllocator more robust, with bufio.Reader and a timeout

Stop erroring about EventDownloadWillBegin events

Support a user's explicit remote-debugging-port flag

Avoid a few potential concurrency issues in edge cases

Source code(tar.gz)
Source code(zip)
v0.4.2(Oct 5, 2019)

Maintenance release. No major changes since v0.4.0 other than Go module dependency updates.
Source code(tar.gz)
Source code(zip)
v0.4.0(Aug 15, 2019)
add WaitNewTarget to grab a tab opened by an existing tab

add CombinedOutput to gather the browser's output

add ByJSPath to select nodes via a JS expression

add Emulate and Device to easily emulate devices

mouse actions now use float64 coordinates to improve high-DPI support

fix an edge case where FullXPath could return an incorrect path

make DefaultExecAllocatorOptions an array, to discourage racy usage

give better errors when an unexpected undefined is encountered

fix a possible panic if page.Navigate is used directly

split Action into many interfaces to organise the docs into sections

add a number of examples and improve the documentation

Source code(tar.gz)
Source code(zip)
v0.3.1(Jun 26, 2019)
fix a regression when sending large messages to Chrome over websocket

increase the timeout when waiting for new pages to appear, for slow devices

avoid hanging in Navigate actions if the context is canceled

fix a data race if Navigate was used while a navigation was already happening

fix a regression where large queries could deadlock the target handler

revert back to using one goroutine per selector, to reduce CPU use when idle

Source code(tar.gz)
Source code(zip)
v0.3.0(May 13, 2019)
allow listening for browser or tab events

navigation actions now wait for the frame to load

allow using tabs opened by existing tabs

fix the screenshot actions to work on high DPIs

work with headless-shell containers out of the box

Source code(tar.gz)
Source code(zip)
v0.2.0(Apr 22, 2019)
Rewrite API to be context-based

Remove all internal timeouts, thanks to contexts

Handle all events sequentially, fixing occasional races

Redesign the Pool type into an Allocator interface

Allow managing multiple tabs per browser

Greatly speed up the tests, now taking a few seconds in total

Source code(tar.gz)
Source code(zip)