fasthttp client. Read body in chunks and stop downloading on condition.

[edwvee]

For example, I want to stop downloading content if it has wrong mime type. How can I do so? The documentation shows only resp.Body() which gets whole response. Below is how I use standard http library:

resp, err := client.Do(req)
if err != nil {
	return nil, err
}
body := resp.Body
defer body.Close()

checkedMime := false
readBuffer := make([]byte, 1 << 12)
bodyData := make([]byte, 0, typicalSize)
for {
	n, err := body.Read(readBuffer)
	if err != nil && err != io.EOF {
		return nil, err
	}
	bodyData = append(bodyData, readBuffer[0:n]...)
	if !checkedMime && len(bodyData) > 512{			
		mimeType := http.DetectContentType(bodyData)
		if isArchive(mimeType){
			isArchive = true
			break
		}
		if mimeType  != "text/html"{
			isUselessContent = true
			break
		}
		checkedMime = true
	}
	if len(bodyData) > tooBigContentSize {
		tooBigContent = true
		break
	}
	if err == io.EOF {
		break
	}
}
// do something with bodyData, isArchive, isUselessContent, tooBigContent
...

[erikdubbelboer]

You could try using https://pkg.go.dev/github.com/valyala/fasthttp?utm_source=godoc#Server.HeaderReceived Its ugly but you could potentially reject the request by setting MaxRequestBodySize to 1 for example.

I would also be open to a pull request that adds something to https://pkg.go.dev/github.com/valyala/fasthttp?utm_source=godoc#RequestConfig to reject the request at that point.

[edwvee]

But I have a client, not a server.

[edwvee]

Added a code using the net/http to achieve what I need as an example.

[moredure]

@erikdubbelboer, such a feature will be useful! I guess I have a patch for this, to skip downloading in case mime type in headers not expected, but not inside the actual body, but need to check.

[erikdubbelboer]

If you have something that works and you can make a pull request that would be great!

[moredure]

but it works like skip body if unwanted header met, and not allows to download body in chunks and check it for mime type