Created a new hook called "on_page_created" which allows the user to inspect raw HTTP requests/responses, and more

[ifaddict1]

Dears, while using your library, I needed to access the raw content of requests being sent when a page is crawled (for instance, requests made to other APIs or files, etc.), the responses, and the associated headers. I realized that the easiest way to do this would be to leverage the existing Playwright hook on "Page" objects by using "Page.on()". Thus, I added a hook in the AsyncCrawlerStragegy class to achieve that.

Please let me know if this is actually useless and there is an easier/native way to do this with your library, but I haven't found any.

Cheers

[oscarnevarezleal]

I'd love to see this implemented. I'm also intending of reading http requests made by the page.

[aravindkarnam]

@ifaddict1 Again thanks for this wonderful PR, and it adds a really nice feature too. I just reviewed it, and found that there has been so many changes and refactors since you raised your PR.

I see that you have set the hook to execute immediately after context.new_page(). However if you visit the async_crawler_strategy now, you can see that that newly managed browsers have been introduced and there's a new get_page function which refactors the complexity of page creation.

So I'm afraid we will not be able to merge this PR as it is. But the feature request is still relevant, so I created this discussion here.

Going forward I encourage you to create feature requests first(rather than directly raising a PR), so that maintainers and community members have visibility to figure out where it fits in the roadmap, to plan, collaborate and release.