proxify
proxify copied to clipboard
Chunked responses length is not calculated correctly
Proxify version:
dev
Current Behavior:
Miscalculation in dumped response body length
Expected Behavior:
Response body length calculated correctly
Steps To Reproduce:
- run
proxify
in a terminal - in other terminal, run
curl --insecure -x 127.0.0.1:8888 'https://pastebin.com/raw/TYsVwM0n'
to ask proxify to write the HTTP request/response file - a file like
./logs/pastebin.com*.txt
has now been created by proxify - observe the chunked length miscalculation:
GET /raw/TYsVwM0n HTTP/1.1
Host: pastebin.com
Accept: */*
User-Agent: curl/7.68.0
HTTP/1.1 200 OK
Transfer-Encoding: chunked
Cache-Control: max-age=1800, must-revalidate
Cf-Cache-Status: MISS
Cf-Ray: 710f4cc23fa97cc5-LAX
Connection: keep-alive
Content-Type: text/plain; charset=utf-8
Date: Wed, 25 May 2022 15:28:16 GMT
Expect-Ct: max-age=604800, report-uri="https://report-uri.cloudflare.com/cdn-cgi/beacon/expect-ct"
Expires: Sat, 26 Jul 1997 05:00:00 GMT
Last-Modified: Wed, 25 May 2022 15:28:16 GMT
Pragma: no-cache
Server: cloudflare
Vary: Accept-Encoding
X-Content-Type-Options: nosniff
X-Frame-Options: DENY
X-Xss-Protection: 1;mode=block
38 <===== Wrong
line1
this is a line containing HTTP/1.1 FOO BAR
line3
0
Anything else:
Reference: https://github.com/projectdiscovery/nuclei/issues/2068#issue-1248304030
It seems like the issue is about the way go process chunked responses, apparently those containing HTTP/1.x markers are considered the end of current chunk, leading to offset miscalculation, example to reproduce:
package main
import (
"crypto/tls"
"log"
"net/http"
"net/http/httputil"
)
func main() {
tr := http.DefaultTransport.(*http.Transport).Clone()
tr.ForceAttemptHTTP2 = false
tr.TLSNextProto = make(map[string]func(authority string, c *tls.Conn) http.RoundTripper)
tr.TLSClientConfig = &tls.Config{}
httpClient := &http.Client{
Transport: tr,
}
req, _ := http.NewRequest(http.MethodGet, "https://pastebin.com/raw/TYsVwM0n", nil)
resp, _ := httpClient.Do(req)
dump, _ := httputil.DumpResponse(resp, true)
log.Println(string(dump))
}
console:
$ go run .
2022/09/05 22:43:39 HTTP/1.1 200 OK
Transfer-Encoding: chunked
Age: 86
Cache-Control: public, max-age=1801
Cf-Cache-Status: HIT
Cf-Ray: 7461cc6418d1bac1-MXP
Connection: keep-alive
Content-Type: text/plain; charset=utf-8
Date: Mon, 05 Sep 2022 20:43:39 GMT
Last-Modified: Mon, 05 Sep 2022 20:42:13 GMT
Server: cloudflare
Vary: Accept-Encoding
X-Content-Type-Options: nosniff
X-Frame-Options: DENY
X-Xss-Protection: 1;mode=block
38
line1
this is a line containing HTTP/1.1 FOO BAR
line3
0
On Hold - Possible solutions to discuss:
- Post-process dumped responses to fix the chunked size calculation
- Investigate go standard library and implement a fix if possible
Related - https://github.com/projectdiscovery/martian/issues/3
@ehsandeep @Mzack9999 , I think we can track this issue as seperate since the current martian
issue was that it was returning a corrupted chunked response always . while in this case if i am not wrong this particular issue only happens when HTTP markers are present in chunked response body ^ apparently those containing HTTP/1.x markers