browsertrix
browsertrix copied to clipboard
[Feature]: Add support for custom crawl headers
trafficstars
What change would you like to see?
Requested on IIPC Slack:
"We need the option to set a request header name and value in the configuration. It could be e.g. cookie: Zxcv1234 and it should then only be set in the request headers if domain name or URL is specified else set on all URLs request. Header Name = cookie Header Value = Zxcv1234"
Context
From IIPC Slack: "What we are missing is the option to manually add multiple custom headers and cookies to a crawl configuration, it could be the same place as user-agent is now. Some hosts offer that as the only access to their content behind a paywall."
This would require adding support for custom headers to the crawler as well as the Browsertrix workflow editor.