requests-html icon indicating copy to clipboard operation
requests-html copied to clipboard

render() time out, user agent workaround not working, what protection is this website using?

Open Ashrakk opened this issue 3 years ago • 1 comments

Hello!

I'm creating a bot for a friend but I'm stuck, I have tried to change the user agent thinking it could be that blocking me from loading the rest of the page.

I've also tried to call the website API directly but the server denied the connection, as i have found out that there's a specific request which could give me directly the table I need from the web page.

I'm making this from scratch because his old bot didn't work too and it was using Selenium.

BTW I also have trouble loading the page on my computer.. sometimes in takes a second, and then, if i try again and reload the page, it won't load anymore in any browser (tried Firefox and Edge)...

any help is appreciated, even a clue. thank you :=)

from requests_html import HTMLSession

URL = 'https://www.bet365.it/#/AC/B92/C1/D50/E2/F163/'

session = HTMLSession(browser_args=["--no-sandbox", '--user-agent=Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/89.0.4389.114 Safari/537.36 Edg/89.0.774.68'])

page = session.get(URL)
page.html.render(timeout=20)

print(page.content)

Ashrakk avatar Apr 07 '21 16:04 Ashrakk

what is the contents of the page after render page.html.raw_html

tke578 avatar May 12 '21 00:05 tke578