govuk_crawler_worker icon indicating copy to clipboard operation
govuk_crawler_worker copied to clipboard

Behaviour when getting a redirect as a response

Open alexmuller opened this issue 8 years ago • 0 comments

Not sure what the current behaviour is, but maybe the behaviour should be:

  • Create an HTML file with a meta refresh to point to the target location
  • Add the target location to the queue

This will mean that redirects are supported on the mirrors.

alexmuller avatar May 27 '16 09:05 alexmuller