ArchiveBot
ArchiveBot copied to clipboard
Replace hardcoded cookies with a proper cookie jar
This classifies as a bug because it seems to prevent other cookies from being set. For example, this breaks Blogger's "content warning" thingy, see job 7c6i9gqmp0al3kuk27vat3dyv.
There are two ways how this could be implemented: there could be a cookie jar in the pipeline directory, or it could be pulled from the control node on starting the job. The latter would probably be better since it wouldn't require pipeline stopping and upgrading to get the new cookies for new jobs, but it's also a bit more complex to implement. If done that way, it should not use CouchDB if possible (#132).