Steven Englehardt

We currently miss: * navigator.mediaDevices * navigator.webdriver * navigator.deviceMemory * window.devicePixelRatio * window.screen.width * window.screen.height * window.screen.availHeight * window.screen.availWidth * Date().getTimezoneOffset() * window.Intl.DateTimeFormat().resolvedOptions().timeZone * window.indexedDB * window.openDatabase * navigator.cpuClass *...

Requests from workers are attributed specifically to the worker script, rather than the tab or frame that created the worker (see #488 for context). If we instrument the APIs used...

Reduce the surface for bot detection

7

There are likely a number of ways to identify that we're running Firefox with Selenium/geckodriver. Back in the Selenium 2 days these were injected by the Selenium extension. We made...

Update schema and document it to make it more understandable

3

* **I'm submitting a ...** [ ] bug report [X] feature request [ ] question about the decisions made in the repository [ ] question about how to use this...

Add a config option to save the hashed content of data URLs

1

* **I'm submitting a ...** [ ] bug report [X] feature request [ ] question about the decisions made in the repository [ ] question about how to use this...

In #322 we had to disable our file upload test which records the request body for a couple file uploads. The Firefox WebExtension API does not currently expose the content...

Investigate alternatives to saving page content as individual gzipped files

4

Basically, the cost of reading a bunch of small files from S3 is much too high (see: http://garrens.com/blog/2017/11/04/big-data-spark-and-its-small-files-problem/). We should think about how we can batch the saving of content...

Refactor how browsers handle failures and restarts

2

The `Browser.restart_browser_manager()`,`Browser.reset()`, and `Browser.launch_browser_manager()` methods are needlessly complex and confusing. This had led to a multitude of bugs, particularly around profile handling (e.g. the simple one in 6e384db5a06ecad4cf1b3aa2369b519a757026cb). This section...

[WIP] Move dns_instrument from onCompleted to onHeadersReceived

3

Fixes #1020 WIP. I haven't tested this much.

Verify content hashing is deterministic

2

See #711 and the related PR #715. We should verify that this has fixed the issue after running a large crawl with the new fix.