gocrawl icon indicating copy to clipboard operation
gocrawl copied to clipboard

Optional Read Limit for reading Response Body

Open Kleissner opened this issue 5 years ago • 0 comments

There should be an optional read limit for reading the response body - otherwise a website could literally respond with GBs of data.

The problem was already raised (and incorrectly dismissed) in 2013 with issue #28. The solution is NOT to check first the content-length header, which may be incorrect and arbitrary.

The solution is to use an io.LimitReader if a read limit is defined. I'm going to create a pull request which implements that.

Kleissner avatar Oct 27 '19 11:10 Kleissner