Y2Z
add option to send headers/cookies
It would be great to use this as a final step in a selenium scrape or other browser driver which may require session state. Then this could pick up the final step (session included) and dump the desired result.
@snshn Please give this request some attention, it's been a few years now and cookies and/or special headers are becoming more and more of a necessity for proper archiving.
Related to #55 , it's now implemented and in master, will be in the next release (flag -C for providing cookie file name, specifying charset has been moved to -E).
It is now in 2.8.0.
Please feel free to close this issue if it satisfies the originally desired functionality.
It is now in 2.8.0.
Please feel free to close this issue if it satisfies the originally desired functionality.
While that will cover most cases, some sites make use of the Authentication header, often in tandem, so a way to be able to assign a value to all headers, not just cookies, is necessary.
Good point. The tricky part is that we don't want to leak those credentials by sending them everywhere (with every network request). The cookie file is safe, since it contains domain/path information in it, but the Authentication header needs to have a target. I could make Monolith send it only to the hostname of the target, but then if the target URL redirects (e.g. from something.com to www.something.com), it wouldn't work.
Lemme see if
http://username:[email protected]/works.
Some sites don't want basic authentication, but rather a token. For instance, here's an Authentication header (with the letters and numbers randomized but I left the punctuation alone) from a site I sometimes scrape in which cookies alone do not suffice:
Bearer H3ViGutR6Xkedo6ztz6vNHRfrLU8vRIL2EM5.87d7ORVcSzAalNOZS8BAmDW44ZQ7FYdE6JeDa57SSphLLOC876nzOEfSUBO03hbs5K2HxK1NuVgBQMT95HC4GWxCTNpANVJ9l3UImpniukvvVZAsQbqOOb6ht1MS9SWdZlJRa9BDOdX45Z06lRyPjsws6iz0NMNFnkzSKmo3z4Px64PYBGajXZ2BzVtuiU2AgqbPzZDK7wqf.kZRTTyZqY_mpLPZ-H0bx4A0eGmumWPxL1EcpvsQB4Gl