retryablehttp-go
retryablehttp-go copied to clipboard
add one more http/2 edgecase
Proposed Changes
go std lib uses type assertion and then handles protocol escalation by checking the negotiated protocol from state of connection ( ex: http -> http2 ) but since utls.Conn is not of type tls.Conn this escalation will not work and server returns a tls http2 response but client thinks it is plain http
we already had such use case in retryablehttp and fixed by checking error and retrying with http2 client , adding one more error to this list solves the utls issue
Before
$ nuclei -u https://golang.org/robots.txt -t a.yaml -tlsi -debug
__ _
____ __ _______/ /__ (_)
/ __ \/ / / / ___/ / _ \/ /
/ / / / /_/ / /__/ / __/ /
/_/ /_/\__,_/\___/_/\___/_/ v3.2.9
projectdiscovery.io
[INF] Current nuclei version: v3.2.9 (latest)
[INF] Current nuclei-templates version: v9.9.0 (latest)
[WRN] Scan results upload to cloud is disabled.
[INF] New templates added in latest release: 164
[INF] Templates loaded for current scan: 1
[WRN] Loading 1 unsigned templates for scan. Use with caution.
[INF] Targets loaded for current scan: 1
[INF] [0cc49ffc-8b49-4eca-b015-7bd053743fe8] Dumped HTTP request for https://golang.org/robots.txt
GET /robots.txt HTTP/1.1
Host: golang.org
User-Agent: Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:89.0) Gecko/20100101 Firefox/89.0
Connection: close
Accept: */*
Accept-Language: en
Accept-Encoding: gzip
[WRN] [0cc49ffc-8b49-4eca-b015-7bd053743fe8] Could not execute request for https://golang.org/robots.txt: [:RUNTIME] got err while executing https://golang.org/robots.txt <- GET https://golang.org/robots.txt giving up after 2 attempts: Get "https://golang.org/robots.txt": net/http: HTTP/1.x transport connection broken: malformed HTTP response "\x00\x00\x12\x04\x00\x00\x00\x00\x00\x00\x03\x00\x00\x00d\x00\x04\x00\x10\x00\x00\x00\x06\x00\x01\x00\x00\x00\x00\x04\b\x00\x00\x00\x00\x00\x00\x0f\x00\x01\x00\x00\x1e\a\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x01http2_handshake_failed"
[INF] No results found. Better luck next time!
After
$ ./nuclei -u https://golang.org/robots.txt -t a.yaml -tlsi -debug
__ _
____ __ _______/ /__ (_)
/ __ \/ / / / ___/ / _ \/ /
/ / / / /_/ / /__/ / __/ /
/_/ /_/\__,_/\___/_/\___/_/ v3.3.0-dev
projectdiscovery.io
[INF] Current nuclei version: v3.3.0-dev (development)
[INF] Current nuclei-templates version: v9.9.0 (latest)
[WRN] Scan results upload to cloud is disabled.
[INF] New templates added in latest release: 164
[INF] Templates loaded for current scan: 1
[WRN] Loading 1 unsigned templates for scan. Use with caution.
[INF] Targets loaded for current scan: 1
[INF] [0cc49ffc-8b49-4eca-b015-7bd053743fe8] Dumped HTTP request for https://golang.org/robots.txt
GET /robots.txt HTTP/1.1
Host: golang.org
User-Agent: Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/95.0.4638.17 Safari/537.36 Edg/95.0.1020.5
Connection: close
Accept: */*
Accept-Language: en
Accept-Encoding: gzip
[DBG] [0cc49ffc-8b49-4eca-b015-7bd053743fe8] Dumped HTTP response https://golang.org/robots.txt
HTTP/1.1 200 OK
Connection: close
Transfer-Encoding: chunked
Cache-Control: private
Content-Security-Policy: connect-src 'self' www.google-analytics.com stats.g.doubleclick.net ; default-src 'self' ; font-src 'self' fonts.googleapis.com fonts.gstatic.com data: ; frame-ancestors 'self' ; frame-src 'self' www.google.com feedback.googleusercontent.com www.googletagmanager.com scone-pa.clients6.google.com www.youtube.com player.vimeo.com ; img-src 'self' www.google.com www.google-analytics.com ssl.gstatic.com www.gstatic.com gstatic.com data: * ; object-src 'none' ; script-src 'self' 'sha256-n6OdwTrm52KqKm6aHYgD0TFUdMgww4a0GQlIAVrMzck=' 'sha256-4ryYrf7Y5daLOBv0CpYtyBIcJPZkRD2eBPdfqsN3r1M=' 'sha256-sVKX08+SqOmnWhiySYk3xC7RDUgKyAkmbXV2GWts4fo=' www.google.com apis.google.com www.gstatic.com gstatic.com support.google.com www.googletagmanager.com www.google-analytics.com ssl.google-analytics.com tagmanager.google.com ; style-src 'self' 'unsafe-inline' fonts.googleapis.com feedback.googleusercontent.com www.gstatic.com gstatic.com tagmanager.google.com ;
Content-Type: text/plain; charset=utf-8
Date: Mon, 08 Jul 2024 15:27:09 GMT
Server: Google Frontend
Strict-Transport-Security: max-age=31536000; includeSubDomains; preload
Vary: Accept-Encoding
X-Cloud-Trace-Context: 2b2317c5420c146b618f8f869546e13f
User-agent: *
Allow: /
[DBG] [0cc49ffc-8b49-4eca-b015-7bd053743fe8] Dumped HTTP response https://golang.org/robots.txt
HTTP/1.1 301 Moved Permanently
Connection: close
Content-Length: 60
Alt-Svc: h3=":443"; ma=2592000,h3-29=":443"; ma=2592000
Content-Security-Policy: connect-src 'self' www.google-analytics.com stats.g.doubleclick.net ; default-src 'self' ; font-src 'self' fonts.googleapis.com fonts.gstatic.com data: ; frame-ancestors 'self' ; frame-src 'self' www.google.com feedback.googleusercontent.com www.googletagmanager.com scone-pa.clients6.google.com www.youtube.com player.vimeo.com ; img-src 'self' www.google.com www.google-analytics.com ssl.gstatic.com www.gstatic.com gstatic.com data: * ; object-src 'none' ; script-src 'self' 'sha256-n6OdwTrm52KqKm6aHYgD0TFUdMgww4a0GQlIAVrMzck=' 'sha256-4ryYrf7Y5daLOBv0CpYtyBIcJPZkRD2eBPdfqsN3r1M=' 'sha256-sVKX08+SqOmnWhiySYk3xC7RDUgKyAkmbXV2GWts4fo=' www.google.com apis.google.com www.gstatic.com gstatic.com support.google.com www.googletagmanager.com www.google-analytics.com ssl.google-analytics.com tagmanager.google.com ; style-src 'self' 'unsafe-inline' fonts.googleapis.com feedback.googleusercontent.com www.gstatic.com gstatic.com tagmanager.google.com ;
Content-Type: text/html; charset=utf-8
Date: Mon, 08 Jul 2024 15:27:09 GMT
Location: https://go.dev/robots.txt
Server: Google Frontend
Strict-Transport-Security: max-age=31536000; includeSubDomains; preload
X-Cloud-Trace-Context: 7be65cdf0fbb4172acd9f95b7d8bd0ca
[INF] No results found. Better luck next time!