0

I'm running python code in a VPS with Ubuntu 18. After much trying, I was able to run the selenium Chromedriver there headfully. Now, I'm finding that the webpages sometimes loads fully, but more often than not, it opens the driver but the page only loads without the JS in it.

I don't know if I need to install something else or why is this happening

Chromium-browser version 84.0.4147 chromedriver version 84.0.4147

The VPS has a GUI included, and I did sudo apt-get install xvfb

The code

#stest.py script

url="https://www.bet365.es"
options = webdriver.ChromeOptions()
options.add_argument('--disable-gpu')
options.add_argument('--no-sandbox')
options.add_argument('--disable-setuid-sandbox')
options.add_argument('--disable-dev-shm-usage')

#This for arguments above, I've tried with and without them. No difference

options.add_experimental_option("excludeSwitches", ["enable-automation"])
options.add_experimental_option('useAutomationExtension', False)
browser=webdriver.Chrome(options=options,executable_path="/usr/bin/chromedriver")
    
browser.execute_cdp_cmd("Page.addScriptToEvaluateOnNewDocument", {
                          "source": """
                            Object.defineProperty(navigator, 'webdriver', {
                              get: () => undefined
                            })
                          """
                        })
browser.execute_cdp_cmd('Network.setUserAgentOverride', 
                            {"userAgent": 'Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/83.0.4103.53 Safari/537.36'})
    
browser.get(url)
WebDriverWait(browser,10).until(
            EC.element_to_be_clickable((By.XPATH,'.//div[contains(@class,"-WebConsoleModule ")]')))

In the bash script I type the following to run the code xvfb-run python3 stest.py start. After the timeout error, when I check the page source, I can see what is in there is the same as if in a regular Chrome I do Ctrl+U, which is the page without the JS loaded.

For the record, the result I get is similar to the one I'd get if I pass the --headless argument, which for this page is not an option as it doesn't behave properly with it

As I said, I need for this to load fully every time, not just once in a while.

0

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.