retryablehttp-go icon indicating copy to clipboard operation
retryablehttp-go copied to clipboard

add one more http/2 edgecase

Open tarunKoyalwar opened this issue 1 year ago • 0 comments

Proposed Changes

go std lib uses type assertion and then handles protocol escalation by checking the negotiated protocol from state of connection ( ex: http -> http2 ) but since utls.Conn is not of type tls.Conn this escalation will not work and server returns a tls http2 response but client thinks it is plain http

we already had such use case in retryablehttp and fixed by checking error and retrying with http2 client , adding one more error to this list solves the utls issue

Before

$ nuclei -u  https://golang.org/robots.txt -t a.yaml -tlsi -debug 

                     __     _
   ____  __  _______/ /__  (_)
  / __ \/ / / / ___/ / _ \/ /
 / / / / /_/ / /__/ /  __/ /
/_/ /_/\__,_/\___/_/\___/_/   v3.2.9

		projectdiscovery.io

[INF] Current nuclei version: v3.2.9 (latest)
[INF] Current nuclei-templates version: v9.9.0 (latest)
[WRN] Scan results upload to cloud is disabled.
[INF] New templates added in latest release: 164
[INF] Templates loaded for current scan: 1
[WRN] Loading 1 unsigned templates for scan. Use with caution.
[INF] Targets loaded for current scan: 1
[INF] [0cc49ffc-8b49-4eca-b015-7bd053743fe8] Dumped HTTP request for https://golang.org/robots.txt

GET /robots.txt HTTP/1.1
Host: golang.org
User-Agent: Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:89.0) Gecko/20100101 Firefox/89.0
Connection: close
Accept: */*
Accept-Language: en
Accept-Encoding: gzip

[WRN] [0cc49ffc-8b49-4eca-b015-7bd053743fe8] Could not execute request for https://golang.org/robots.txt: [:RUNTIME] got err while executing https://golang.org/robots.txt <- GET https://golang.org/robots.txt giving up after 2 attempts: Get "https://golang.org/robots.txt": net/http: HTTP/1.x transport connection broken: malformed HTTP response "\x00\x00\x12\x04\x00\x00\x00\x00\x00\x00\x03\x00\x00\x00d\x00\x04\x00\x10\x00\x00\x00\x06\x00\x01\x00\x00\x00\x00\x04\b\x00\x00\x00\x00\x00\x00\x0f\x00\x01\x00\x00\x1e\a\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x01http2_handshake_failed"
[INF] No results found. Better luck next time!

After

$ ./nuclei -u  https://golang.org/robots.txt -t a.yaml -tlsi -debug 

                     __     _
   ____  __  _______/ /__  (_)
  / __ \/ / / / ___/ / _ \/ /
 / / / / /_/ / /__/ /  __/ /
/_/ /_/\__,_/\___/_/\___/_/   v3.3.0-dev

		projectdiscovery.io

[INF] Current nuclei version: v3.3.0-dev (development)
[INF] Current nuclei-templates version: v9.9.0 (latest)
[WRN] Scan results upload to cloud is disabled.
[INF] New templates added in latest release: 164
[INF] Templates loaded for current scan: 1
[WRN] Loading 1 unsigned templates for scan. Use with caution.
[INF] Targets loaded for current scan: 1
[INF] [0cc49ffc-8b49-4eca-b015-7bd053743fe8] Dumped HTTP request for https://golang.org/robots.txt

GET /robots.txt HTTP/1.1
Host: golang.org
User-Agent: Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/95.0.4638.17 Safari/537.36 Edg/95.0.1020.5
Connection: close
Accept: */*
Accept-Language: en
Accept-Encoding: gzip

[DBG] [0cc49ffc-8b49-4eca-b015-7bd053743fe8] Dumped HTTP response https://golang.org/robots.txt

HTTP/1.1 200 OK
Connection: close
Transfer-Encoding: chunked
Cache-Control: private
Content-Security-Policy: connect-src 'self' www.google-analytics.com stats.g.doubleclick.net ; default-src 'self' ; font-src 'self' fonts.googleapis.com fonts.gstatic.com data: ; frame-ancestors 'self' ; frame-src 'self' www.google.com feedback.googleusercontent.com www.googletagmanager.com scone-pa.clients6.google.com www.youtube.com player.vimeo.com ; img-src 'self' www.google.com www.google-analytics.com ssl.gstatic.com www.gstatic.com gstatic.com data: * ; object-src 'none' ; script-src 'self' 'sha256-n6OdwTrm52KqKm6aHYgD0TFUdMgww4a0GQlIAVrMzck=' 'sha256-4ryYrf7Y5daLOBv0CpYtyBIcJPZkRD2eBPdfqsN3r1M=' 'sha256-sVKX08+SqOmnWhiySYk3xC7RDUgKyAkmbXV2GWts4fo=' www.google.com apis.google.com www.gstatic.com gstatic.com support.google.com www.googletagmanager.com www.google-analytics.com ssl.google-analytics.com tagmanager.google.com ; style-src 'self' 'unsafe-inline' fonts.googleapis.com feedback.googleusercontent.com www.gstatic.com gstatic.com tagmanager.google.com ;
Content-Type: text/plain; charset=utf-8
Date: Mon, 08 Jul 2024 15:27:09 GMT
Server: Google Frontend
Strict-Transport-Security: max-age=31536000; includeSubDomains; preload
Vary: Accept-Encoding
X-Cloud-Trace-Context: 2b2317c5420c146b618f8f869546e13f

User-agent: *
Allow: /
[DBG] [0cc49ffc-8b49-4eca-b015-7bd053743fe8] Dumped HTTP response https://golang.org/robots.txt

HTTP/1.1 301 Moved Permanently
Connection: close
Content-Length: 60
Alt-Svc: h3=":443"; ma=2592000,h3-29=":443"; ma=2592000
Content-Security-Policy: connect-src 'self' www.google-analytics.com stats.g.doubleclick.net ; default-src 'self' ; font-src 'self' fonts.googleapis.com fonts.gstatic.com data: ; frame-ancestors 'self' ; frame-src 'self' www.google.com feedback.googleusercontent.com www.googletagmanager.com scone-pa.clients6.google.com www.youtube.com player.vimeo.com ; img-src 'self' www.google.com www.google-analytics.com ssl.gstatic.com www.gstatic.com gstatic.com data: * ; object-src 'none' ; script-src 'self' 'sha256-n6OdwTrm52KqKm6aHYgD0TFUdMgww4a0GQlIAVrMzck=' 'sha256-4ryYrf7Y5daLOBv0CpYtyBIcJPZkRD2eBPdfqsN3r1M=' 'sha256-sVKX08+SqOmnWhiySYk3xC7RDUgKyAkmbXV2GWts4fo=' www.google.com apis.google.com www.gstatic.com gstatic.com support.google.com www.googletagmanager.com www.google-analytics.com ssl.google-analytics.com tagmanager.google.com ; style-src 'self' 'unsafe-inline' fonts.googleapis.com feedback.googleusercontent.com www.gstatic.com gstatic.com tagmanager.google.com ;
Content-Type: text/html; charset=utf-8
Date: Mon, 08 Jul 2024 15:27:09 GMT
Location: https://go.dev/robots.txt
Server: Google Frontend
Strict-Transport-Security: max-age=31536000; includeSubDomains; preload
X-Cloud-Trace-Context: 7be65cdf0fbb4172acd9f95b7d8bd0ca

[INF] No results found. Better luck next time!

tarunKoyalwar avatar Jul 08 '24 15:07 tarunKoyalwar