ferrum icon indicating copy to clipboard operation
ferrum copied to clipboard

Published 20 hours ago • verified publisher icon rubycdp

Handle pending connections losing context on frame navigation

Open francisbeaudoin opened this issue 1 year ago • 6 comments

Details

Similar to what is being proposed in #420, this PR fixes intermittent Ferrum::TimeoutError as the driver awaits for requests to be completed.

What

It was observed that in some scenarios the driver is not always receiving and associated Network.loadingFailed for requests that were in-flight as the main frame navigates. This may be happening with either main frame requests or sub-frame ones.

I'm not sure if this is the right approach, but it appears to be working. Basically, it flags requests which are no longer relevant to the current page context as unknown which are then assumed to be #finished.

Relates to #420

francisbeaudoin avatar Dec 15 '23 19:12 francisbeaudoin

@francisbeaudoin do you have an example of such a page in the public internet? I just want to check the CDP logs to better understand of what's going on. A test describing the issue would be more appropriate though.

route avatar Jan 06 '24 11:01 route

@route Thanks for following up.

I spent some time reproducing the issue by using a use case that was causing timeouts at the time, and found something interesting.

By using the headless browser instance opened by ferrum vs using a regular instance ran from the desktop, the network requests were not returning the same status. i.e. (canceled) vs (unknown). As far as I remember, the (unknown) ones were causing the library to raise a timeout as it couldn't infer if a request had completed.

I've identified the different behavior to the disable-site-isolation-trials option. If that option is removed thus enabled, Chrome shows a (unknown) network request as opposed to (canceled) status. In the current library state, the option is disabled (added 4 months ago) so it may not longer be an issue for this specific use case, but I suspect there are other scenarios where requests may end up in an unknown state thereby timing out.

This was reproduced by visiting this example, and toggling the disable-site-isolation-trials option.

With disable-site-isolation-trials disable-site-isolation-trials

Without disable-site-isolation-trials with-site-isolation-trials

francisbeaudoin avatar Jan 09 '24 18:01 francisbeaudoin

Hmmmm, nice thank you! Let me experiment a bit with that

route avatar Jan 11 '24 10:01 route

👋 @route By any chance you had time to look into this? I'd love to stop using a forked version in my project.

francisbeaudoin avatar Mar 26 '24 21:03 francisbeaudoin

Sorry not yet, was busy doing other things. Thanks for the reminder I will

route avatar Mar 27 '24 12:03 route