urlwatch
urlwatch copied to clipboard
Proposals for "navigate": ability to set a custom user agent and to execute a javascript in the page context
I like this project, if you agree I think this two features will be an improvement, since some websites knock down the request when they spot an "*/headless" user agent.
I hope it will be easy to set a custom user agent and evaluate javascript in the _get_content() function of browser module using page.setUserAgent and page.evaluate method exposed.
I think also it will be a great addition using playwright as an alternative browser interface, since playwright IMHO has a much smoother experience handling automation and has a native python API
Both suggestions sound fine to me, patches gladly accepted.
In #700 I added an optional key for changing the useragent (but did nothing about the javascript evaluation).