Chunked responses length is not calculated correctly

[Mzack9999]

Proxify version:

dev

Current Behavior:

Miscalculation in dumped response body length

Expected Behavior:

Response body length calculated correctly

Steps To Reproduce:

1. run proxify in a terminal
2. in other terminal, run curl --insecure -x 127.0.0.1:8888 'https://pastebin.com/raw/TYsVwM0n' to ask proxify to write the HTTP request/response file
3. a file like ./logs/pastebin.com*.txt has now been created by proxify
4. observe the chunked length miscalculation:

GET /raw/TYsVwM0n HTTP/1.1
Host: pastebin.com
Accept: */*
User-Agent: curl/7.68.0

HTTP/1.1 200 OK
Transfer-Encoding: chunked
Cache-Control: max-age=1800, must-revalidate
Cf-Cache-Status: MISS
Cf-Ray: 710f4cc23fa97cc5-LAX
Connection: keep-alive
Content-Type: text/plain; charset=utf-8
Date: Wed, 25 May 2022 15:28:16 GMT
Expect-Ct: max-age=604800, report-uri="https://report-uri.cloudflare.com/cdn-cgi/beacon/expect-ct"
Expires: Sat, 26 Jul 1997 05:00:00 GMT
Last-Modified: Wed, 25 May 2022 15:28:16 GMT
Pragma: no-cache
Server: cloudflare
Vary: Accept-Encoding
X-Content-Type-Options: nosniff
X-Frame-Options: DENY
X-Xss-Protection: 1;mode=block

38 <===== Wrong
line1
this is a line containing HTTP/1.1 FOO BAR
line3
0

Anything else:

Reference: https://github.com/projectdiscovery/nuclei/issues/2068#issue-1248304030

[Mzack9999]

It seems like the issue is about the way go process chunked responses, apparently those containing HTTP/1.x markers are considered the end of current chunk, leading to offset miscalculation, example to reproduce:

package main

import (
	"crypto/tls"
	"log"
	"net/http"
	"net/http/httputil"
)

func main() {
	tr := http.DefaultTransport.(*http.Transport).Clone()
	tr.ForceAttemptHTTP2 = false
	tr.TLSNextProto = make(map[string]func(authority string, c *tls.Conn) http.RoundTripper)
	tr.TLSClientConfig = &tls.Config{}
	httpClient := &http.Client{
		Transport: tr,
	}
	req, _ := http.NewRequest(http.MethodGet, "https://pastebin.com/raw/TYsVwM0n", nil)
	resp, _ := httpClient.Do(req)
	dump, _ := httputil.DumpResponse(resp, true)
	log.Println(string(dump))
}

console:

$ go run .
2022/09/05 22:43:39 HTTP/1.1 200 OK
Transfer-Encoding: chunked
Age: 86
Cache-Control: public, max-age=1801
Cf-Cache-Status: HIT
Cf-Ray: 7461cc6418d1bac1-MXP
Connection: keep-alive
Content-Type: text/plain; charset=utf-8
Date: Mon, 05 Sep 2022 20:43:39 GMT
Last-Modified: Mon, 05 Sep 2022 20:42:13 GMT
Server: cloudflare
Vary: Accept-Encoding
X-Content-Type-Options: nosniff
X-Frame-Options: DENY
X-Xss-Protection: 1;mode=block

38
line1
this is a line containing HTTP/1.1 FOO BAR
line3
0

On Hold - Possible solutions to discuss:

• Post-process dumped responses to fix the chunked size calculation
• Investigate go standard library and implement a fix if possible

[ehsandeep]

Related - https://github.com/projectdiscovery/martian/issues/3

[tarunKoyalwar]

@ehsandeep @Mzack9999 , I think we can track this issue as seperate since the current martian issue was that it was returning a corrupted chunked response always . while in this case if i am not wrong this particular issue only happens when HTTP markers are present in chunked response body ^ apparently those containing HTTP/1.x markers