hakrawler
hakrawler copied to clipboard
add basic redirect of input
echo https://paypal.com | hakrawler
gives no output.
we need to make it follow the basic redirect for hakrawler to work.. and could be prevented if we just make it follow the initials redirects
# curl https://paypal.com -IL
HTTP/2 301
date: Tue, 17 Sep 2024 23:03:42 GMT
location: https://www.paypal.com/
server: ECAcc (agc/7F3B)
content-length: 0
HTTP/2 302
server: Varnish
retry-after: 0
location: https://www.paypal.com/us/home
accept-ranges: bytes
date: Tue, 17 Sep 2024 23:03:42 GMT
via: 1.1 varnish
strict-transport-security: max-age=63072000; includeSubDomains; preload
x-served-by: cache-pdk-kfty2130050-PDK
x-cache: HIT
x-cache-hits: 0
server-timing: content-encoding;desc="",x-cdn;desc="fastly"
content-length: 0
can we implement this so we don't have to do a curl request before doing hakrawler???