hakrawler icon indicating copy to clipboard operation
hakrawler copied to clipboard

add basic redirect of input

Open marcelo321 opened this issue 5 months ago • 0 comments

echo https://paypal.com | hakrawler gives no output.

we need to make it follow the basic redirect for hakrawler to work.. and could be prevented if we just make it follow the initials redirects

# curl https://paypal.com -IL
HTTP/2 301 
date: Tue, 17 Sep 2024 23:03:42 GMT
location: https://www.paypal.com/
server: ECAcc (agc/7F3B)
content-length: 0

HTTP/2 302 
server: Varnish
retry-after: 0
location: https://www.paypal.com/us/home
accept-ranges: bytes
date: Tue, 17 Sep 2024 23:03:42 GMT
via: 1.1 varnish
strict-transport-security: max-age=63072000; includeSubDomains; preload
x-served-by: cache-pdk-kfty2130050-PDK
x-cache: HIT
x-cache-hits: 0
server-timing: content-encoding;desc="",x-cdn;desc="fastly"
content-length: 0

can we implement this so we don't have to do a curl request before doing hakrawler???

marcelo321 avatar Sep 17 '24 23:09 marcelo321