fingerprint-securedrop
fingerprint-securedrop copied to clipboard
Implement our own page load timeout function for the crawler
Selenium's page load timeout function is highly unreliable. If it doesn't close down a connection within 5s of when it's supposed to, we should stop a crawl by whatever means necessary (probably closing all circuits will be sufficient, but we already have a method for restarting TB if we need to). This will stop the crawler from wasting time getting stuck on these sites which load for minutes at a time. See fpsd/tests/test_sketchy_sites.py
for some good example sites/ a good test case for this timeout function.