MaxResponseBodySize and the correct way to retrieve a sample preview of the HTTP response

Open slicingmelon opened this issue 7 months ago • 1 comments

Hello,

For the past few months, I've been working on a fasthttp based HTTP client. I struggled to find a way to read only a maximum number of bytes from the HTTP response. All testing was performed against live servers.

Obviously, MaxResponseBodysize is the important factor here, so I began testing various implementations. https://github.com/valyala/fasthttp/blob/4c71125994a1a67c8c6cb979142ae4269c5d89f1/client.go#L278

Other notable configurations: https://github.com/valyala/fasthttp/blob/4c71125994a1a67c8c6cb979142ae4269c5d89f1/client.go#L255

https://github.com/valyala/fasthttp/blob/4c71125994a1a67c8c6cb979142ae4269c5d89f1/client.go#L260

Below, I am providing sample code snippets that I tried until I got a working PoC. In all examples, I am setting MaxResponseBodySize to 12288 bytes as I figured out that this includes response headers too, and for example, sites like github.com use a long list of headers.

Approach 1

const (
	maxBodySize         = 12288 // Limit the maximum body size we are willing to read completely
	rwBufferSize        = maxBodySize + 4096
	RespBodyPreviewSize = 1024 // I want a sample preview of 1024 bytes from the response body
)

StreamResponseBody:  false // This is the default setting in fasthttp

Response read via resp.Body() https://github.com/valyala/fasthttp/blob/4c71125994a1a67c8c6cb979142ae4269c5d89f1/http.go#L411

Sample code snippet:

package main

import (
	"crypto/tls"
	"fmt"
	"log"

	"github.com/VictoriaMetrics/VictoriaMetrics/lib/bytesutil"
	"github.com/valyala/fasthttp"
)

var (
	strSpace      = []byte(" ")
	strCRLF       = []byte("\r\n")
	strColonSpace = []byte(": ")
)

var headerBufPool bytesutil.ByteBufferPool

type RawHTTPResponseDetails struct {
	StatusCode      int
	ResponsePreview []byte // I want a sample preview bytes from the response body
	ResponseHeaders []byte
	ContentType     []byte
	ContentLength   int64
	ServerInfo      []byte
	ResponseBytes   int // Total bytes of the response received (Headers + Body up to maxBodySize)
}

const (
	maxBodySize         = 12288 // Limit the maximum body size we are willing to read completely
	rwBufferSize        = maxBodySize + 4096
	RespBodyPreviewSize = 1024 // I want a sample preview of 1024 bytes from the response body
)

func main() {
	url := "https://www.sample-videos.com/video321/mp4/360/big_buck_bunny_360p_30mb.mp4" // 30 mb video

	tlsConfig := &tls.Config{
		InsecureSkipVerify: true,
	}

	client := &fasthttp.Client{
		MaxResponseBodySize: maxBodySize,  // Set the maximum response body size
		ReadBufferSize:      rwBufferSize, // Set read buffer size
		WriteBufferSize:     rwBufferSize, // Set write buffer size
		StreamResponseBody:  false,        // By default, fasthttp uses StreamResponseBody = false
		TLSConfig:           tlsConfig,
	}

	req := fasthttp.AcquireRequest()
	defer fasthttp.ReleaseRequest(req)
	req.SetRequestURI(url)
	req.Header.SetMethod(fasthttp.MethodGet)

	resp := fasthttp.AcquireResponse()
	defer fasthttp.ReleaseResponse(resp)

	err := client.Do(req, resp)

	// Check for errors but ALLOW ErrBodyTooLarge!
	if err != nil && err != fasthttp.ErrBodyTooLarge {
		log.Fatalf("Error fetching URL %s: %v", url, err)
	}

	respDetails := RawHTTPResponseDetails{}

	respDetails.StatusCode = resp.StatusCode()

	respDetails.ResponseHeaders = GetResponseHeaders(&resp.Header, respDetails.StatusCode, respDetails.ResponseHeaders)

	body := resp.Body()
	bodyLen := len(body)

	// Determine the preview size
	previewSize := RespBodyPreviewSize
	if bodyLen < previewSize {
		previewSize = bodyLen
	}

	// Create and populate the ResponsePreview
	preview := make([]byte, previewSize)
	copy(preview, body[:previewSize])
	respDetails.ResponsePreview = preview

	// Populate other details
	respDetails.ContentType = resp.Header.ContentType()
	respDetails.ContentLength = int64(resp.Header.ContentLength())
	respDetails.ServerInfo = resp.Header.Server()

	// Calculate total received bytes (header size + received body size)
	respDetails.ResponseBytes = resp.Header.Len() + bodyLen

	fmt.Println("--- Collected Response Details ---")
	fmt.Printf("Status Code: %d\n", respDetails.StatusCode)

	fmt.Printf("\nResponse Headers (%d bytes):\n", len(respDetails.ResponseHeaders))
	fmt.Println(string(respDetails.ResponseHeaders)) // Print headers including status line

	fmt.Printf("Content-Type: %s\n", string(respDetails.ContentType))
	fmt.Printf("Content-Length Header: %d\n", respDetails.ContentLength)
	fmt.Printf("Server: %s\n", string(respDetails.ServerInfo))
	fmt.Printf("Total Bytes Received (Headers + Body Preview/Truncated Body): %d\n", respDetails.ResponseBytes)

	fmt.Printf("\nResponse Body Preview (%d bytes):\n", len(respDetails.ResponsePreview))
	fmt.Println(string(respDetails.ResponsePreview))

	if err == fasthttp.ErrBodyTooLarge {
		fmt.Println("\nNote: Response body exceeded MaxResponseBodySize.")
		fmt.Printf("Full response body is likely larger than the received %d bytes (maxBodySize = %d).\n", bodyLen, maxBodySize)
	} else if int64(bodyLen) > RespBodyPreviewSize && respDetails.ContentLength > int64(RespBodyPreviewSize) {
		fmt.Printf("\nNote: Full response body (%d bytes reported by Content-Length) is larger than the preview size (%d bytes).\n", respDetails.ContentLength, RespBodyPreviewSize)
	} else if int64(bodyLen) > RespBodyPreviewSize {
		fmt.Printf("\nNote: Response body received (%d bytes) is larger than the preview size (%d bytes).\n", bodyLen, RespBodyPreviewSize)
	}

}

// GetResponseHeaders helper
func GetResponseHeaders(h *fasthttp.ResponseHeader, statusCode int, dest []byte) []byte {
	headerBuf := headerBufPool.Get()
	defer headerBufPool.Put(headerBuf)

	headerBuf.Write(h.Protocol())
	headerBuf.Write(strSpace)
	headerBuf.B = fasthttp.AppendUint(headerBuf.B, statusCode)
	headerBuf.Write(strSpace)
	headerBuf.Write(h.StatusMessage())
	headerBuf.Write(strCRLF)

	h.VisitAll(func(key, value []byte) {
		headerBuf.Write(key)
		headerBuf.Write(strColonSpace)
		headerBuf.Write(value)
		headerBuf.Write(strCRLF)
	})

	headerBuf.Write(strCRLF)

	return append(dest[:0], headerBuf.B...)
}

Output:

go run .\main.go
--- Collected Response Details ---
Status Code: 200

Response Headers (268 bytes):
HTTP/1.1 200 OK
Content-Length: 31466742
Content-Type: video/mp4
Server: Apache/2.4.6 (CentOS) OpenSSL/1.0.2k-fips PHP/7.3.23
Date: Wed, 09 Apr 2025 09:00:29 GMT
Last-Modified: Tue, 20 Oct 2020 10:45:05 GMT
Etag: "1e024f6-5b217ec8840b6"
Accept-Ranges: bytes


Content-Type: video/mp4
Content-Length Header: 31466742
Server: Apache/2.4.6 (CentOS) OpenSSL/1.0.2k-fips PHP/7.3.23
Total Bytes Received (Headers + Body Preview/Truncated Body): 7

Response Body Preview (0 bytes):


Note: Response body exceeded MaxResponseBodySize.
Full response body is likely larger than the received 0 bytes (maxBodySize = 12288).

This approach failed, and it also failed on many other servers that didn't include a Content-Lenght header in the response.

Approach 2

const (
	maxBodySize         = 12288 // Limit the maximum body size we are willing to read completely
	rwBufferSize        = maxBodySize + 4096
	RespBodyPreviewSize = 1024 // I want a sample preview of 1024 bytes from the response body
)

StreamResponseBody:  true // Enabled StreamResponseBody

Response is read via resp.BodyStream() https://github.com/valyala/fasthttp/blob/4c71125994a1a67c8c6cb979142ae4269c5d89f1/http.go#L333C1-L335C2

Based on these testcases, I tried to use resp.ReadLimitBody() and ensured resp.CloseBodyStream() https://github.com/valyala/fasthttp/blob/4c71125994a1a67c8c6cb979142ae4269c5d89f1/http_test.go#L2962 https://github.com/valyala/fasthttp/blob/4c71125994a1a67c8c6cb979142ae4269c5d89f1/http.go#L337

Looking back through my git history, other sample helper I tried for this approach:

// ReadLimitedResponseBodyStream reads limited bytes from a response body stream
// Appends the result to dest slice
func ReadLimitedResponseBodyStream(stream io.Reader, previewSize int, dest []byte) []byte {
	// Create a buffer for preview
	previewBuf := bytes.NewBuffer(make([]byte, 0, previewSize))

	// Read limited amount of data
	if _, err := io.CopyN(previewBuf, stream, int64(previewSize)); err != nil && err != io.EOF {
		return append(dest[:0], strErrorReadingPreview...)
	}

	return append(dest[:0], previewBuf.Bytes()...)
}

While this approach worked better and allowed me to successfully obtain a response preview, it still failed in most test cases, though not all. It was noted that it failed in concurrent requests. Sample errors:

Error: error when reading response headers: cannot find whitespace in the first line of response ";\xbb\xaa\x9a\x15\xb2\b\xb8\xa8\x00\xe7\xa1\xda=Ӻ\xac=\x03\xa6\x87\xf5Xz\xad:\xbb\xc2\xd8g\xc28\x8d&F`\xa6\x12\xd0\xeb\xa9\r\xa7\x97\x98Ϋ\xddxg\xd9HȢж\x9a\x8b\x89\b\xb4+\x90\x14\f\xf7h\xc3\xd0\xdcF\\\xef[\x0e\xf8C\x0f\xf9\x1c\xd6\x17\xf9Y\x14\xf2\xfb\x17X\x04\xc8\xe2\xb6J\xb8>\\\x85\x0e2\xa4f\xa1\xc6GE\xb5\xd9mvT\xb9\xd6[]\x96ړ\xa04\x9b\r\x1a\

Approach 3 - Almost working!

const (
	maxBodySize         = 12288 // Limit the maximum body size we are willing to read completely
	rwBufferSize        = maxBodySize + 4096
	RespBodyPreviewSize = 1024 // I want a sample preview of 1024 bytes from the response body
)

StreamResponseBody:  true // Enabled StreamResponseBody

Response read via resp.Body() https://github.com/valyala/fasthttp/blob/4c71125994a1a67c8c6cb979142ae4269c5d89f1/http.go#L411

Code snippet:

package main

import (
	"crypto/tls"
	"fmt"
	"log"

	"github.com/VictoriaMetrics/VictoriaMetrics/lib/bytesutil"
	"github.com/valyala/fasthttp"
)

var (
	strSpace      = []byte(" ")
	strCRLF       = []byte("\r\n")
	strColonSpace = []byte(": ")
)

var headerBufPool bytesutil.ByteBufferPool

type RawHTTPResponseDetails struct {
	StatusCode      int
	ResponsePreview []byte // I want a sample previw bytes from the response body
	ResponseHeaders []byte
	ContentType     []byte
	ContentLength   int64
	ServerInfo      []byte
	ResponseBytes   int // Total bytes of the response received (Headers + Body up to maxBodySize)
}

const (
	maxBodySize         = 12288 // Limit the maximum body size we are willing to read completely
	rwBufferSize        = maxBodySize + 4096
	RespBodyPreviewSize = 1024 // I want a sample previw of 1024 bytes from the response body
)

func main() {
	url := "https://www.sample-videos.com/video321/mp4/360/big_buck_bunny_360p_30mb.mp4" // 30 mb video

	tlsConfig := &tls.Config{
		InsecureSkipVerify: true,
	}

	client := &fasthttp.Client{
		MaxResponseBodySize: maxBodySize,  // Set the maximum response body size
		ReadBufferSize:      rwBufferSize, // Set read buffer size
		WriteBufferSize:     rwBufferSize, // Set write buffer size
		StreamResponseBody:  true,         // By default, fasthttp uses StreamResponseBody = false
		TLSConfig:           tlsConfig,
	}

	req := fasthttp.AcquireRequest()
	defer fasthttp.ReleaseRequest(req)
	req.SetRequestURI(url)
	req.Header.SetMethod(fasthttp.MethodGet)

	resp := fasthttp.AcquireResponse()
	defer fasthttp.ReleaseResponse(resp)

	err := client.Do(req, resp)

	// Check for errors but ALLOW ErrBodyTooLarge!
	if err != nil && err != fasthttp.ErrBodyTooLarge {
		log.Fatalf("Error fetching URL %s: %v", url, err)
	}

	respDetails := RawHTTPResponseDetails{}

	respDetails.StatusCode = resp.StatusCode()

	respDetails.ResponseHeaders = GetResponseHeaders(&resp.Header, respDetails.StatusCode, respDetails.ResponseHeaders)

	body := resp.Body()
	bodyLen := len(body)

	// Determine the preview size
	previewSize := RespBodyPreviewSize
	if bodyLen < previewSize {
		previewSize = bodyLen
	}

	// Create and populate the ResponsePreview
	preview := make([]byte, previewSize)
	copy(preview, body[:previewSize])
	respDetails.ResponsePreview = preview

	// Populate other details
	respDetails.ContentType = resp.Header.ContentType()
	respDetails.ContentLength = int64(resp.Header.ContentLength())
	respDetails.ServerInfo = resp.Header.Server()

	// Calculate total received bytes (header size + received body size)
	respDetails.ResponseBytes = resp.Header.Len() + bodyLen

	fmt.Println("--- Collected Response Details ---")
	fmt.Printf("Status Code: %d\n", respDetails.StatusCode)

	fmt.Printf("\nResponse Headers (%d bytes):\n", len(respDetails.ResponseHeaders))
	fmt.Println(string(respDetails.ResponseHeaders)) // Print headers including status line

	fmt.Printf("Content-Type: %s\n", string(respDetails.ContentType))
	fmt.Printf("Content-Length Header: %d\n", respDetails.ContentLength)
	fmt.Printf("Server: %s\n", string(respDetails.ServerInfo))
	fmt.Printf("Total Bytes Received (Headers + Body Preview/Truncated Body): %d\n", respDetails.ResponseBytes)

	fmt.Printf("\nResponse Body Preview (%d bytes):\n", len(respDetails.ResponsePreview))
	fmt.Println(string(respDetails.ResponsePreview))

	if err == fasthttp.ErrBodyTooLarge {
		fmt.Println("\nNote: Response body exceeded MaxResponseBodySize.")
		fmt.Printf("Full response body is likely larger than the received %d bytes (maxBodySize = %d).\n", bodyLen, maxBodySize)
	} else if int64(bodyLen) > RespBodyPreviewSize && respDetails.ContentLength > int64(RespBodyPreviewSize) {
		fmt.Printf("\nNote: Full response body (%d bytes reported by Content-Length) is larger than the preview size (%d bytes).\n", respDetails.ContentLength, RespBodyPreviewSize)
	} else if int64(bodyLen) > RespBodyPreviewSize {
		fmt.Printf("\nNote: Response body received (%d bytes) is larger than the preview size (%d bytes).\n", bodyLen, RespBodyPreviewSize)
	}

}

// GetResponseHeaders helper
func GetResponseHeaders(h *fasthttp.ResponseHeader, statusCode int, dest []byte) []byte {
	headerBuf := headerBufPool.Get()
	defer headerBufPool.Put(headerBuf)

	headerBuf.Write(h.Protocol())
	headerBuf.Write(strSpace)
	headerBuf.B = fasthttp.AppendUint(headerBuf.B, statusCode)
	headerBuf.Write(strSpace)
	headerBuf.Write(h.StatusMessage())
	headerBuf.Write(strCRLF)

	// Write all headers
	h.VisitAll(func(key, value []byte) {
		headerBuf.Write(key)
		headerBuf.Write(strColonSpace)
		headerBuf.Write(value)
		headerBuf.Write(strCRLF)
	})

	headerBuf.Write(strCRLF)

	return append(dest[:0], headerBuf.B...)
}

Output:

go run .\main.go
--- Collected Response Details ---
Status Code: 200

Response Headers (268 bytes):
HTTP/1.1 200 OK
Content-Length: 31466742
Content-Type: video/mp4
Server: Apache/2.4.6 (CentOS) OpenSSL/1.0.2k-fips PHP/7.3.23
Date: Wed, 09 Apr 2025 09:06:19 GMT
Last-Modified: Tue, 20 Oct 2020 10:45:05 GMT
Etag: "1e024f6-5b217ec8840b6"
Accept-Ranges: bytes


Content-Type: video/mp4
Content-Length Header: 31466742
Server: Apache/2.4.6 (CentOS) OpenSSL/1.0.2k-fips PHP/7.3.23
Total Bytes Received (Headers + Body Preview/Truncated Body): 31466749

Response Body Preview (1024 bytes):
ftypisomisomiso2mp4freeދ_mdat�&
�!���0
��@�dJa=w�����k��
�}y�jT�H��hr��}���G�+�EY��e�����&��.�T�Ӻ�O�^��Kʩ�Yz�:��Og���;��K�o��6�`�v��w�̈́v�׌��h&�F�����-�t"����aUY�b��[����
�o��$����ظ1ޑ���(/�w����v
���`�XN ��p�D��H�>�������g����~zխy�JUX��v��u��ґ��=�)��n]�4�`_��/L)]���j��{��C���r'*�"��w��u�f�l�m-��U�?�9�+�-�@=�8�4��������@�PP&

B�aXPN�Ap�DF    �B��D/�x�_3�U�}7Ϸ���Ǚ��RU�\`սW�����?ֈ��0u2#�ۯD��u�%���J4]ɛ��g�X���k+r�t�P���HSW���>�0�  ZS��b���u8�=$
�`)�0S��^����^ ������݂�`�YP␦
.�u��:J�]�S,�O��␦Ah]o*�M�ӓJTPM�w._~␦q���#���O�h$���FXo�crǖ�Eu���#H˞����[}��w@� Lh�@�`,$)�&��g����n�s[�VY.M(���ya�������y������ɑ}?��v��o�r�9��k:�����C��������s��Z��[�[�KNq ������ �tM�C�Ike�7�����ݡ]+��?���wn��nS�/4a_U`�"��EGA����Mh��J����zĤ ��b騭��+2������mf�0␦��"��w����q

Note: Full response body (31466742 bytes reported by Content-Length) is larger than the preview size (1024 bytes).

Using StreamResponsebody = true and reading the response using resp.Body() looked like the working approach; it worked for most of the requests. However, it also failed under high concurrency, and I'm still unsure of the exact cause.

I suspect that CloseBodyStream closes the stream too early, failing to retrieve the response: https://github.com/valyala/fasthttp/blob/4c71125994a1a67c8c6cb979142ae4269c5d89f1/http.go#L415 https://github.com/valyala/fasthttp/blob/4c71125994a1a67c8c6cb979142ae4269c5d89f1/http.go#L416

Approach 4 - Finally working

I managed to get a final working PoC, this version worked in all test cases. I've successfully sent and retrieved more than 1 million requests/responses (concurrent) on many different servers.

const (
	maxBodySize         = 12288 // Limit the maximum body size (headers + body)
	rwBufferSize        = maxBodySize + 4096
	RespBodyPreviewSize = 1024 // I want a sample preview of 1024 bytes from the response body
)

StreamResponseBody:  true

I found that resp.BodyWriteTo() calls Write() internally, so I ended up using a LimitedWriter implementation: https://github.com/valyala/fasthttp/blob/4c71125994a1a67c8c6cb979142ae4269c5d89f1/http.go#L637

package main

import (
	"crypto/tls"
	"fmt"
	"io"
	"log"

	"github.com/VictoriaMetrics/VictoriaMetrics/lib/bytesutil"
	"github.com/valyala/fasthttp"
)

var (
	strSpace      = []byte(" ")
	strCRLF       = []byte("\r\n")
	strColonSpace = []byte(": ")
)

var headerBufPool bytesutil.ByteBufferPool
var respPreviewBufPool bytesutil.ByteBufferPool

type RawHTTPResponseDetails struct {
	StatusCode      int
	ResponsePreview []byte // I want a sample preview bytes from the response body
	ResponseHeaders []byte
	ContentType     []byte
	ContentLength   int64
	ServerInfo      []byte
	ResponseBytes   int // Total bytes of the response received (Headers + Body up to maxBodySize)
}

type LimitedWriter struct {
	W io.Writer // Underlying writer
	N int64     // Max bytes remaining
}

func (l *LimitedWriter) Write(p []byte) (n int, err error) {
	if l.N <= 0 {
		return 0, io.EOF
	}
	if int64(len(p)) > l.N {
		p = p[0:l.N]
	}
	n, err = l.W.Write(p)
	l.N -= int64(n)
	return
}

const (
	maxBodySize         = 12288 // Limit the maximum body size (headers + body)
	rwBufferSize        = maxBodySize + 4096
	RespBodyPreviewSize = 1024 // I want a sample preview of 1024 bytes from the response body
)

func main() {
	url := "https://www.sample-videos.com/video321/mp4/360/big_buck_bunny_360p_30mb.mp4" // 30 mb video

	tlsConfig := &tls.Config{
		InsecureSkipVerify: true,
	}

	client := &fasthttp.Client{
		MaxResponseBodySize: maxBodySize,
		ReadBufferSize:      rwBufferSize,
		WriteBufferSize:     rwBufferSize,
		StreamResponseBody:  true, // Enabling streaming mode is crucial!
		TLSConfig:           tlsConfig,
	}

	req := fasthttp.AcquireRequest()
	defer fasthttp.ReleaseRequest(req)
	req.SetRequestURI(url)
	req.Header.SetMethod(fasthttp.MethodGet)

	resp := fasthttp.AcquireResponse()
	defer fasthttp.ReleaseResponse(resp)

	err := client.Do(req, resp)

	if err != nil && err != fasthttp.ErrBodyTooLarge {
		log.Fatalf("Error fetching URL %s: %v", url, err)
	}

	respDetails := RawHTTPResponseDetails{}

	respDetails.StatusCode = resp.StatusCode()
	respDetails.ContentType = resp.Header.ContentType()
	respDetails.ContentLength = int64(resp.Header.ContentLength())
	respDetails.ServerInfo = resp.Header.Server()
	respDetails.ResponseHeaders = GetResponseHeaders(&resp.Header, respDetails.StatusCode, respDetails.ResponseHeaders)

	previewBuf := respPreviewBufPool.Get()
	defer respPreviewBufPool.Put(previewBuf)

	// Create the LimitedWriter to control how much is read from the stream
	limitedWriter := &LimitedWriter{
		W: previewBuf,                 // Write into the pooled buffer
		N: int64(RespBodyPreviewSize), // Limit preview size
	}

	// Read stream into buffer via limitedWriter
	if err := resp.BodyWriteTo(limitedWriter); err != nil && err != io.EOF {
		log.Printf("Warning: Error reading response body stream: %v\n", err)
	}

	if len(previewBuf.B) > 0 {
		respDetails.ResponsePreview = append(respDetails.ResponsePreview[:0], previewBuf.B...)
		respDetails.ResponseBytes = resp.Header.Len() + len(previewBuf.B)
	}

	fmt.Println("--- Collected Response Details (Streaming Mode) ---")
	fmt.Printf("Status Code: %d\n", respDetails.StatusCode)
	fmt.Printf("\nResponse Headers (%d bytes):\n%s", len(respDetails.ResponseHeaders), string(respDetails.ResponseHeaders))
	fmt.Printf("Content-Type: %s\n", string(respDetails.ContentType))
	fmt.Printf("Content-Length Header: %d\n", respDetails.ContentLength)
	fmt.Printf("Server: %s\n", string(respDetails.ServerInfo))
	fmt.Printf("Total Bytes Received (Headers + Body Preview): %d\n", respDetails.ResponseBytes)

	fmt.Printf("\nResponse Body Preview (%d bytes):\n", len(respDetails.ResponsePreview))

	if len(respDetails.ResponsePreview) > 0 {
		fmt.Println(string(respDetails.ResponsePreview))
	} else {
		fmt.Println("<empty preview>")
	}
}

// GetResponseHeaders helper
func GetResponseHeaders(h *fasthttp.ResponseHeader, statusCode int, dest []byte) []byte {
	headerBuf := headerBufPool.Get()
	defer headerBufPool.Put(headerBuf)

	headerBuf.Write(h.Protocol())
	headerBuf.Write(strSpace)
	headerBuf.B = fasthttp.AppendUint(headerBuf.B, statusCode)
	headerBuf.Write(strSpace)
	headerBuf.Write(h.StatusMessage())
	headerBuf.Write(strCRLF)

	h.VisitAll(func(key, value []byte) {
		headerBuf.Write(key)
		headerBuf.Write(strColonSpace)
		headerBuf.Write(value)
		headerBuf.Write(strCRLF)
	})

	headerBuf.Write(strCRLF)

	return append(dest[:0], headerBuf.B...)
}

Output:

go run .\main.go
2025/04/09 13:33:21 Warning: Error reading response body stream: short write
--- Collected Response Details (Streaming Mode) ---
Status Code: 200

Response Headers (268 bytes):
HTTP/1.1 200 OK
Content-Length: 31466742
Content-Type: video/mp4
Server: Apache/2.4.6 (CentOS) OpenSSL/1.0.2k-fips PHP/7.3.23
Date: Wed, 09 Apr 2025 10:22:50 GMT
Last-Modified: Tue, 20 Oct 2020 10:45:05 GMT
Etag: "1e024f6-5b217ec8840b6"
Accept-Ranges: bytes

Content-Type: video/mp4
Content-Length Header: 31466742
Server: Apache/2.4.6 (CentOS) OpenSSL/1.0.2k-fips PHP/7.3.23
Total Bytes Received (Headers + Body Preview): 1031

Response Body Preview (1024 bytes):
ftypisomisomiso2mp4freeދ_mdat�&
�!���0
��@�dJa=w�����k��
�}y�jT�H��hr��}���G�+�EY��e�����&��.�T�Ӻ�O�^��Kʩ�Yz�:��Og���;��K�o��6�`�v��w�̈́v�׌��h&�F�����-�t"����aUY�b��[����
�o��$����ظ1ޑ���(/�w����v
���`�XN ��p�D��H�>�������g����~zխy�JUX��v��u��ґ��=�)��n]�4�`_��/L)]���j��{��C���r'*�"��w��u�f�l�m-��U�?�9�+�-�@=�8�4��������@�PP&

B�aXPN�Ap�DF    �B��D/�x�_3�U�}7Ϸ���Ǚ��RU�\`սW�����?ֈ��0u2#�ۯD��u�%���J4]ɛ��g�X���k+r�t�P���HSW���>�0�  ZS��b���u8�=$
�`)�0S��^����^ ������݂�`�YP␦
.�u��:J�]�S,�O��␦Ah]o*�M�ӓJTPM�w._~␦q���#���O�h$���FXo�crǖ�Eu���#H˞����[}��w@� Lh�@�`,$)�&��g����n�s[�VY.M(���ya�������y������ɑ}?��v��o�r�9��k:�����C��������s��Z��[�[�KNq ������ �tM�C�Ike�7�����ݡ]+��?���wn��nS�/4a_U`�"��EGA����Mh��J����zĤ ��b騭��+2������mf�0␦��"��w����q

Approach 4 seems to be working fine for me. I noticed that sometimes it returns io.ErrShortWrite, but I think this is expected based on my implementation.

My question now is, what would be the correct way to read up to MaxResponseBodySize and also capture a preview up to N bytes from the response? Or if I should stick to my approach.

Apr 09 '25 10:04 slicingmelon

I have never done this myself but your approach 4 seems to be a good approach. I'll keep this issue open for others who might want to do the same. Thanks for exploring this!

Apr 19 '25 05:04 erikdubbelboer

fasthttp fasthttp copied to clipboard

MaxResponseBodySize and the correct way to retrieve a sample preview of the HTTP response

Approach 1

Approach 2

Approach 3 - Almost working!

Approach 4 - Finally working

fasthttp
fasthttp copied to clipboard