govuk_crawler_worker
govuk_crawler_worker copied to clipboard
Behaviour when getting a redirect as a response
Not sure what the current behaviour is, but maybe the behaviour should be:
- Create an HTML file with a meta refresh to point to the target location
- Add the target location to the queue
This will mean that redirects are supported on the mirrors.