Fix crawler not being updated when manually following redirect

[cs278]

If you manually follow a redirect the crawler contains the response for the request before.

I've targeted this at the 3.x branch is that's what my project is using, it does apply cleanly to master if necessary but I'd appreciate a new 3.x release if that's not too much trouble.