gocrawl
gocrawl copied to clipboard
Optional Read Limit for reading Response Body
There should be an optional read limit for reading the response body - otherwise a website could literally respond with GBs of data.
The problem was already raised (and incorrectly dismissed) in 2013 with issue #28. The solution is NOT to check first the content-length header, which may be incorrect and arbitrary.
The solution is to use an io.LimitReader
if a read limit is defined. I'm going to create a pull request which implements that.