bbot Idiotic results from httpx

When BBOT is set to proxy through Burpsuite, it seems that httpx discovers HTTPS URLs but not HTTP ones.

Sep 12 '22 15:09 TheTechromancer

Having trouble replicating, do you have an example query?

Oct 31 '22 20:10 liquidsec

On hold until issue can be replicated

Feb 02 '23 21:02 liquidsec

Unable to replicate, closing.

Jul 10 '23 20:07 TheTechromancer

Was able to replicate this, there seems to be two separate, but related issues:

False reporting of port 80 on https sites: https://www.blacklanternsecurity.com:80/
HTTP sites not being detected at all when running through burp

Nov 14 '23 17:11 liquidsec

Running some tests this week, and starting to realize how bad this is. Httpx consistently misses URLs any time the web server issues a valid HTTP response on an HTTPS port (which practically every modern server and WAF does).

When given a target, it will output idiotic results:

[URL]               	http://aliexpress.com/	httpx	(dir, http-title-301-moved-permanently, in-scope, ip-47-246-173-237, status-301)
[URL]               	http://aliexpress.com:443/	httpx	(dir, http-title-301-moved-permanently, in-scope, ip-47-246-173-237, status-301)

http://aliexpress.com:443/?? There is no excuse for this. We should prioritize replacing httpx.

@liquidsec

Apr 11 '24 00:04 TheTechromancer