Running Pydoll in a Docker Container

[misotrnka]

Hi,

I'm experimenting with Pydoll and I'm trying to make it run in a Docker container. I was able to start the Chrome browser with Xvfb, I'm able to ping it (via browser._connection_handler.ping()), but when I try to navigate somewhere (e.g. await page.go_to('https://www.example.com')), it hits a time out with this:

pydoll/browser/page.py", line 208, in go_to
    raise TimeoutError('Page load timed out')

Is there anyone here that has experience with containerizing Pydoll apps? Is there any caveat that I'm not aware of?

[thalissonvs]

Hi @misotrnka , the xvfb is really necessary? You can just set the headless to True. Try this dockerfile:

FROM python:3.11.0

# Install dependencies
RUN apt -y update
RUN apt install -y gnupg
RUN apt install -y wget
RUN apt install -y iproute2
RUN apt install -y wkhtmltopdf
RUN apt install -y libx11-xcb1
RUN apt install -y libdbus-glib-1-2
RUN apt install -y git
RUN apt install -y tini


# Install google chrome browser
RUN wget -q -O - https://dl-ssl.google.com/linux/linux_signing_key.pub | apt-key add -
RUN wget -O /tmp/chrome.deb https://dl.google.com/linux/chrome/deb/pool/main/g/google-chrome-stable/google-chrome-stable_$CHROMEDRIVER_VERSION-1_amd64.deb
RUN apt install -y /tmp/chrome.deb
RUN rm /tmp/chrome.deb

# APP
WORKDIR /app

RUN pip install --upgrade pip
RUN pip install --upgrade pip wheel

RUN pip install pydoll-python

COPY . /app/

ENV PYTHONPATH=/app

[FriendlyUser]

I have tried to use pydoll in docker but if I use docker I cant bypass cloudflare reCAPTCHA is this expected? Or is it something wrong on my end.

[thalissonvs]

To successfully bypass detection in Docker containers, you need to be careful with a few things. First, make sure to change the user-agent, since the default headless user-agents are problematic. Some browser options are also important. I especially recommend using --enable-webgl.

options = ChromiumOptions()
options.add_argument("--user-agent=your-user-agent")
options.add_argument("--enable-webgl")
options.add_argument("--no-sandbox")

If that doesn't solve the issue, you might need to resort to more aggressive alternatives, such as using Xvfb and setting headless to False, to simulate a more realistic environment.

[misotrnka]

Thank you @thalissonvs , but I don't think Xvfb is really the problem. The behaviour is the same when I run in headless mode without Xvfb. I've tracked the issue down to this so far:

• There are pages where the page never reaches "loaded" state, which is something the go_to() method is waiting for. I've modified this so I could investigate further by waiting for the "interactive" state. Then I try to select an element on the page, I send the CDP command but never get a response from the browser and then I get a timeout.

Here's how it looks:

Command: {'method': 'Runtime.evaluate', 'params': {'expression': 'window.location.href', 'returnByValue': False}}
Response: {"id": 1, "result": {"result": {"type": "string", "value": "chrome://new-tab-page/"}}}

Command: {'method': 'Page.navigate', 'params': {'url': 'https://www.gerulata.com'}}
Response: {"id": 2, "result": {"frameId": "E70E639CFD9E6CCB01768737DB01D665", "loaderId": "203A14857893D63D9011A18D4A2A4478"}}

Command: {'method': 'Runtime.evaluate', 'params': {'expression': 'document.readyState', 'returnByValue': False}}
Response: {"id": 3, "result": {"result": {"type": "string", "value": "loading"}}}

... waiting for the "loaded" state

Command: {'method': 'Runtime.evaluate', 'params': {'expression': 'document.readyState', 'returnByValue': False}}
Response: {"id": 5, "result": {"result": {"type": "string", "value": "interactive"}}}

Command: {'method': 'Runtime.evaluate', 'params': {'expression': 'document.readyState', 'returnByValue': False}}
Response: {"id": 6, "result": {"result": {"type": "string", "value": "interactive"}}}

Command: {'method': 'Runtime.evaluate', 'params': {'expression': 'document.readyState', 'returnByValue': False}}
Response: {"id": 7, "result": {"result": {"type": "string", "value": "interactive"}}}

Command: {'method': 'Runtime.evaluate', 'params': {'expression': 'document.readyState', 'returnByValue': False}}
Response: Never received.

TimeoutError: Page load timed out

With the modified code, where I proceed when reached the "interactive" state (should be enough since I only need the DOM parsed), it will proceed to selecting an element on the page, but again the CDP protocol times out:

...
Command: {'method': 'Runtime.evaluate', 'params': {'expression': 'document.readyState', 'returnByValue': False}}
Response: {"id": 5, "result": {"result": {"type": "string", "value": "interactive"}}}

Command: {'method': 'Runtime.evaluate', 'params': {'expression': 'document.querySelector("#contact-us");', 'returnByValue': False}}
Response: Never received.

TimeoutError: Page load timed out

I haven't encountered this behavior on pages that do not linger in the "interactive" state and properly reach "loaded" within some reasonable time. However, there are many pages that will never be fully loaded for a number of reasons. So perhaps Pydoll should have a mechanism to detect whether the page loading actualy finished and don't rely on the page "loaded" state.

Still however, running the selector on a document in "interactive" state SHOULD work afaik. So maybe there is some other problem with the CDP. Since this only happens when running in a container, I believe the issue is actualy somehow related to CDP communication.

[cleitonleonel]

Thank you @thalissonvs , but I don't think Xvfb is really the problem. The behaviour is the same when I run in headless mode without Xvfb. I've tracked the issue down to this so far:

• There are pages where the page never reaches "loaded" state, which is something the go_to() method is waiting for. I've modified this so I could investigate further by waiting for the "interactive" state. Then I try to select an element on the page, I send the CDP command but never get a response from the browser and then I get a timeout.

Here's how it looks:

Command: {'method': 'Runtime.evaluate', 'params': {'expression': 'window.location.href', 'returnByValue': False}}
Response: {"id": 1, "result": {"result": {"type": "string", "value": "chrome://new-tab-page/"}}}

Command: {'method': 'Page.navigate', 'params': {'url': 'https://www.gerulata.com'}}
Response: {"id": 2, "result": {"frameId": "E70E639CFD9E6CCB01768737DB01D665", "loaderId": "203A14857893D63D9011A18D4A2A4478"}}

Command: {'method': 'Runtime.evaluate', 'params': {'expression': 'document.readyState', 'returnByValue': False}}
Response: {"id": 3, "result": {"result": {"type": "string", "value": "loading"}}}

... waiting for the "loaded" state

Command: {'method': 'Runtime.evaluate', 'params': {'expression': 'document.readyState', 'returnByValue': False}}
Response: {"id": 5, "result": {"result": {"type": "string", "value": "interactive"}}}

Command: {'method': 'Runtime.evaluate', 'params': {'expression': 'document.readyState', 'returnByValue': False}}
Response: {"id": 6, "result": {"result": {"type": "string", "value": "interactive"}}}

Command: {'method': 'Runtime.evaluate', 'params': {'expression': 'document.readyState', 'returnByValue': False}}
Response: {"id": 7, "result": {"result": {"type": "string", "value": "interactive"}}}

Command: {'method': 'Runtime.evaluate', 'params': {'expression': 'document.readyState', 'returnByValue': False}}
Response: Never received.

TimeoutError: Page load timed out

With the modified code, where I proceed when reached the "interactive" state (should be enough since I only need the DOM parsed), it will proceed to selecting an element on the page, but again the CDP protocol times out:

...
Command: {'method': 'Runtime.evaluate', 'params': {'expression': 'document.readyState', 'returnByValue': False}}
Response: {"id": 5, "result": {"result": {"type": "string", "value": "interactive"}}}

Command: {'method': 'Runtime.evaluate', 'params': {'expression': 'document.querySelector("#contact-us");', 'returnByValue': False}}
Response: Never received.

TimeoutError: Page load timed out

I haven't encountered this behavior on pages that do not linger in the "interactive" state and properly reach "loaded" within some reasonable time. However, there are many pages that will never be fully loaded for a number of reasons. So perhaps Pydoll should have a mechanism to detect whether the page loading actualy finished and don't rely on the page "loaded" state.

Still however, running the selector on a document in "interactive" state SHOULD work afaik. So maybe there is some other problem with the CDP. Since this only happens when running in a container, I believe the issue is actualy somehow related to CDP communication.

Hello @misotrnka , could you provide the codes for your experiment and also the links to the pages used in the tests?

[rastarr]

@misotrnka Did you ever get this working?
i'd love to see a docker container that successfully operates with pydoll

[luisleozitei]

@FriendlyUser
Could you show me your Dockerfile and Pydoll options?

[FriendlyUser]

I have been using github actions so I havent updated the docker file in a while, dont think its worth showing keep in mind that linux + chromium could easily be detected by cloudflare.

[luisleozitei]

@FriendlyUser Yes, they are detected, but on a local Windows machine, the enable_auto_solve_cloudflare_captcha method from Pydoll works very well—it bypasses Cloudflare and allows navigation to continue. However, in the container, this method has no effect and the navigation remains blocked.

[luisleozitei]

@FriendlyUser
I’m literally at the same point as your initial comment:
"I have tried to use Pydoll in Docker but if I use Docker I can’t bypass Cloudflare reCAPTCHA. Is this expected? Or is it something wrong on my end?"

If I have any updates, I’ll post them here. Please do the same if you have any.

[FriendlyUser]

I suspected it was linux in a generic docker container, its clear its a docker image and/or linux, clearly detected as a bot, and not the fault of the framework, this logic was on hold for me until the fingerprinting pr is out.

Tried setting headless=True, a xvfb and other things, didnt really work

I should clarify some of the behaviour of cloudflare, perfectly possible you were detected as a bot and they are just wasting your time, hard for me to tell what the cause is or they are rejecting the clicks cause of bot detection like I said before.