Splash response.follow not working well

[deverhjj]

trafficstars

When I use response.follow, callback response.url is not correct. It makes force you to use SplashRequest and join url to follow link. Since SplashRequest extends Request, but the follow method become useless. it make more code to follow link.

The request url is detail url , but the response.url in callback is not the same as the request url. It seem 'url' is not prefilled from request url.

[apchirinoc]

This was happening to me too. Well, at least I think it's the same problem:

I was trying to send some requests and it was geting the same Spider.start_urls for every response.follow() request I made. I think it's because of the way the SplashMiddleware sets a default meta['splash']['url'] value. When we send our first request the SplashMiddleware (if it isn't specified) sets the meta['splash']['url'] by default with the request.url value. This means that our self.splash_meta (in your case self.splash_default_value) is updated directly. You can see this if you log self.logger.info(self.splash_default_value). So when we try to make other requests (having meta['splash']['url'] set) with response.follow(), it overrides the url value of the Request created sent to Splash with the value of meta['splash']['url']. So as from this point the url value is always set in the meta kwarg, it always uses the same url for every request.