fingerprint-securedrop icon indicating copy to clipboard operation
fingerprint-securedrop copied to clipboard

Implement our own page load timeout function for the crawler

Open psivesely opened this issue 7 years ago • 1 comments

Selenium's page load timeout function is highly unreliable. If it doesn't close down a connection within 5s of when it's supposed to, we should stop a crawl by whatever means necessary (probably closing all circuits will be sufficient, but we already have a method for restarting TB if we need to). This will stop the crawler from wasting time getting stuck on these sites which load for minutes at a time. See fpsd/tests/test_sketchy_sites.py for some good example sites/ a good test case for this timeout function.

psivesely avatar Oct 29 '16 00:10 psivesely