go-sitemirror
go-sitemirror copied to clipboard
Add the "Referer" header to requests.
Problem: Many websites use the "Referer" header to protect the content on the page, causing their mirroring to fail.
Solution: Add the "Referer" header with the value of the website domain name to every request. This header will not change any content or change the server response to the request, but it will resolve hotlink blocking on most sites.