Lars
It seems that Chrome sometime sends a request twice and then messes up the order in which it sends events, i.e. requestWillBeSent, requestWillBeSent (same id), loadingFinished. This results in the...
Add a command-line option that allows a) replacing the default click settings (click.yaml) and b) adding more of them at runtime. ``` --click-data=click.yaml --click-match="^example\.com" --click-selector="div.foo span.bar" ```
Matching CSS selectors is becoming increasingly difficult, as big sites usually obfuscate their CSS class names/use random names. Matching text could provide a way out. Additionally matching text could be...
Support matching hosted software like Disqus. It can run on any domain, so whitelisting them is not an option.
Old sites (before history.pushState was invented) use this for navigation. Decide whether this is recursion (probably yes) and how to handle it. Examples: - https://groups.google.com/ - https://bowden.info/recipes/
The API is not exactly pretty and it’s easy to mess things up. There are no plausibility checks and no validation. We want: - A nice/clean API that separates WARC...
It should be possible to add ignored URL pattern (regex) to recursive crawls. They should be updateable (add/remove pattern) while the job is running.
Instead of logging some information here and there we should dump (selected) internal state, so the dashboard can recover the current state more easily. Right now it just replays the...
It’s unlikely we’ll ever have a method to replay these, but it might be good to capture them nontheless.
Both `` and `` support resolution-based image loading. We’d like to fetch all images instead of just the one Google Chrome picked for us based on the *current* resolution. Right...