katana
katana copied to clipboard
Don't crawl same path when the page outputs visited url in response
Some websites append the current url which is being requested in the page content/either in an tag or something else, What happens here is that katana gets stuck in a loop where it tries to recrawl the same url.
E.g :
katana -u https://www.blackrock.com/corporate/search/ -d 5 -cs searchText
Response :
https://www.blackrock.com/corporate/search/summary-search-results?doTickerSearch=true&searchText=katana
https://www.blackrock.com/corporate/search/summary-search-results?searchText=katana
https://www.blackrock.com/corporate/search/summary-search-results?doTickerSearch=true&searchText=katana?searchText=katana
https://www.blackrock.com/corporate/search/summary-search-results?doTickerSearch=true&searchText=katana?
https://www.blackrock.com/corporate/search/summary-search-results?doTickerSearch=true&searchText=katana?searchText=katana&searchType=generalSearch
https://www.blackrock.com/corporate/search/summary-search-results?searchText=katana%3FsearchText%3Dkatana
https://www.blackrock.com/corporate/search/summary-search-results?doTickerSearch=true&searchText=katana?searchText=katana?searchText=katana%3FsearchText%3Dkatana
https://www.blackrock.com/corporate/search/summary-search-results?doTickerSearch=true&searchText=katana?searchText=katana?
https://www.blackrock.com/corporate/search/summary-search-results?searchText=katana?searchText=katana
https://www.blackrock.com/corporate/search/summary-search-results?searchText=katana?
https://www.blackrock.com/corporate/search/summary-search-results?searchText=katana?searchText=katana&searchType=generalSearch
https://www.blackrock.com/corporate/search/summary-search-results?searchText=katana%3F
https://www.blackrock.com/corporate/search/summary-search-results?doTickerSearch=true&searchText=katana??searchText=katana%3F
https://www.blackrock.com/corporate/search/summary-search-results?doTickerSearch=true&searchText=katana??
https://www.blackrock.com/corporate/search/summary-search-results?doTickerSearch=true&searchText=katana?searchText=katana&searchType=generalSearch?searchText=katana%3FsearchText%3Dkatana&searchType=generalSearch
https://www.blackrock.com/corporate/search/summary-search-results?doTickerSearch=true&searchText=katana?searchText=katana&searchType=generalSearch?
https://www.blackrock.com/corporate/search/summary-search-results?searchText=katana%3FsearchText%3Dkatana?searchText=katana%3FsearchText%3Dkatana
https://www.blackrock.com/corporate/search/summary-search-results?searchText=katana%3FsearchText%3Dkatana?
https://www.blackrock.com/corporate/search/summary-search-results?searchText=katana%3FsearchText%3Dkatana%3FsearchText%3Dkatana%3FsearchText%3Dkatana
https://www.blackrock.com/corporate/search/summary-search-results?doTickerSearch=true&searchText=katana?searchText=katana?searchText=katana%3FsearchText%3Dkatana?searchText=katana%3FsearchText%3Dkatana%3FsearchText%3Dkatana%3FsearchText%3Dkatana
https://www.blackrock.com/corporate/search/summary-search-results?doTickerSearch=true&searchText=katana?searchText=katana?searchText=katana%3FsearchText%3Dkatana?
https://www.blackrock.com/corporate/search/summary-search-results?searchText=katana%3FsearchText%3Dkatana%3F
https://www.blackrock.com/corporate/search/summary-search-results?doTickerSearch=true&searchText=katana?searchText=katana??searchText=katana%3FsearchText%3Dkatana%3F
https://www.blackrock.com/corporate/search/summary-search-results?doTickerSearch=true&searchText=katana?searchText=katana??
https://www.blackrock.com/corporate/search/summary-search-results?searchText=katana?searchText=katana?searchText=katana%3FsearchText%3Dkatana
https://www.blackrock.com/corporate/search/summary-search-results?searchText=katana?searchText=katana?
https://www.blackrock.com/corporate/search/summary-search-results?searchText=katana??searchText=katana%3F
https://www.blackrock.com/corporate/search/summary-search-results?searchText=katana??
https://www.blackrock.com/corporate/search/summary-search-results?searchText=katana?searchText=katana&searchType=generalSearch?searchText=katana%3FsearchText%3Dkatana&searchType=generalSearch
https://www.blackrock.com/corporate/search/summary-search-results?searchText=katana?searchText=katana&searchType=generalSearch?
https://www.blackrock.com/corporate/search/summary-search-results?searchText=katana%3F?searchText=katana%3F
https://www.blackrock.com/corporate/search/summary-search-results?searchText=katana%3F?
https://www.blackrock.com/corporate/search/summary-search-results?searchText=katana%3F%3FsearchText%3Dkatana%3F
https://www.blackrock.com/corporate/search/summary-search-results?doTickerSearch=true&searchText=katana??searchText=katana%3F?searchText=katana%3F%3FsearchText%3Dkatana%3F
https://www.blackrock.com/corporate/search/summary-search-results?doTickerSearch=true&searchText=katana??searchText=katana%3F?
https://www.blackrock.com/corporate/search/summary-search-results?searchText=katana%3F%3F
https://www.blackrock.com/corporate/search/summary-search-results?doTickerSearch=true&searchText=katana???searchText=katana%3F%3F
https://www.blackrock.com/corporate/search/summary-search-results?doTickerSearch=true&searchText=katana???
https://www.blackrock.com/corporate/search/summary-search-results?searchText=katana%3FsearchText%3Dkatana??searchText=katana%3FsearchText%3Dkatana%3F
https://www.blackrock.com/corporate/search/summary-search-results?searchText=katana%3FsearchText%3Dkatana??
https://www.blackrock.com/corporate/search/summary-search-results?searchText=katana%3FsearchText%3Dkatana%3FsearchText%3Dkatana%3FsearchText%3Dkatana%3F
https://www.blackrock.com/corporate/search/summary-search-results?doTickerSearch=true&searchText=katana?searchText=katana?searchText=katana%3FsearchText%3Dkatana??searchText=katana%3FsearchText%3Dkatana%3FsearchText%3Dkatana%3FsearchText%3Dkatana%3F
https://www.blackrock.com/corporate/search/summary-search-results?doTickerSearch=true&searchText=katana?searchText=katana?searchText=katana%3FsearchText%3Dkatana??
https://www.blackrock.com/corporate/search/summary-search-results?searchText=katana%3FsearchText%3Dkatana%3F%3F
https://www.blackrock.com/corporate/search/summary-search-results?doTickerSearch=true&searchText=katana?searchText=katana???searchText=katana%3FsearchText%3Dkatana%3F%3F
https://www.blackrock.com/corporate/search/summary-search-results?doTickerSearch=true&searchText=katana?searchText=katana???
https://www.blackrock.com/corporate/search/summary-search-results?searchText=katana?searchText=katana??searchText=katana%3FsearchText%3Dkatana%3F
https://www.blackrock.com/corporate/search/summary-search-results?searchText=katana?searchText=katana??
https://www.blackrock.com/corporate/search/summary-search-results?searchText=katana???searchText=katana%3F%3F
https://www.blackrock.com/corporate/search/summary-search-results?searchText=katana???
https://www.blackrock.com/corporate/search/summary-search-results?searchText=katana%3F??searchText=katana%3F%3F
https://www.blackrock.com/corporate/search/summary-search-results?searchText=katana%3F??
https://www.blackrock.com/corporate/search/summary-search-results?searchText=katana%3F%3FsearchText%3Dkatana%3F%3F
https://www.blackrock.com/corporate/search/summary-search-results?doTickerSearch=true&searchText=katana??searchText=katana%3F??searchText=katana%3F%3FsearchText%3Dkatana%3F%3F
https://www.blackrock.com/corporate/search/summary-search-results?doTickerSearch=true&searchText=katana??searchText=katana%3F??
https://www.blackrock.com/corporate/search/summary-search-results?searchText=katana%3F%3F%3F
https://www.blackrock.com/corporate/search/summary-search-results?doTickerSearch=true&searchText=katana????searchText=katana%3F%3F%3F
https://www.blackrock.com/corporate/search/summary-search-results?doTickerSearch=true&searchText=katana????
https://www.blackrock.com/corporate/search/summary-search-results?searchText=katana%3F%3F%3FsearchText%3Dkatana%3F%3F
https://www.blackrock.com/corporate/search/summary-search-results?doTickerSearch=true&searchText=katana???searchText=katana%3F%3F?searchText=katana%3F%3F%3FsearchText%3Dkatana%3F%3F
https://www.blackrock.com/corporate/search/summary-search-results?doTickerSearch=true&searchText=katana???searchText=katana%3F%3F?
https://www.blackrock.com/corporate/search/summary-search-results?searchText=katana%3F%3F?searchText=katana%3F%3F
https://www.blackrock.com/corporate/search/summary-search-results?searchText=katana%3F%3F?
https://www.blackrock.com/corporate/search/summary-search-results?searchText=katana%3F%3FsearchText%3Dkatana%3F%3FsearchText%3Dkatana%3F%3FsearchText%3Dkatana%3F
https://www.blackrock.com/corporate/search/summary-search-results?doTickerSearch=true&searchText=katana??searchText=katana%3F?searchText=katana%3F%3FsearchText%3Dkatana%3F?searchText=katana%3F%3FsearchText%3Dkatana%3F%3FsearchText%3Dkatana%3F%3FsearchText%3Dkatana%3F
https://www.blackrock.com/corporate/search/summary-search-results?doTickerSearch=true&searchText=katana??searchText=katana%3F?searchText=katana%3F%3FsearchText%3Dkatana%3F?
https://www.blackrock.com/corporate/search/summary-search-results?searchText=katana%3F%3FsearchText%3Dkatana%3F?searchText=katana%3F%3FsearchText%3Dkatana%3F
https://www.blackrock.com/corporate/search/summary-search-results?searchText=katana%3F%3FsearchText%3Dkatana%3F?
https://www.blackrock.com/corporate/search/summary-search-results?searchText=katana%3F?searchText=katana%3F?searchText=katana%3F%3FsearchText%3Dkatana%3F
https://www.blackrock.com/corporate/search/summary-search-results?searchText=katana%3F?searchText=katana%3F?
https://www.blackrock.com/corporate/search/summary-search-results?searchText=katana??searchText=katana%3F?searchText=katana%3F%3FsearchText%3Dkatana%3F
https://www.blackrock.com/corporate/search/summary-search-results?searchText=katana??searchText=katana%3F?
https://www.blackrock.com/corporate/search/summary-search-results?searchText=katana?searchText=katana?searchText=katana%3FsearchText%3Dkatana?searchText=katana%3FsearchText%3Dkatana%3FsearchText%3Dkatana%3FsearchText%3Dkatana
https://www.blackrock.com/corporate/search/summary-search-results?searchText=katana?searchText=katana?searchText=katana%3FsearchText%3Dkatana?
https://www.blackrock.com/corporate/search/summary-search-results?searchText=katana%3FsearchText%3Dkatana%3F%3FsearchText%3Dkatana%3FsearchText%3Dkatana%3F
https://www.blackrock.com/corporate/search/summary-search-results?doTickerSearch=true&searchText=katana?searchText=katana??searchText=katana%3FsearchText%3Dkatana%3F?searchText=katana%3FsearchText%3Dkatana%3F%3FsearchText%3Dkatana%3FsearchText%3Dkatana%3F
https://www.blackrock.com/corporate/search/summary-search-results?doTickerSearch=true&searchText=katana?searchText=katana??searchText=katana%3FsearchText%3Dkatana%3F?
https://www.blackrock.com/corporate/search/summary-search-results?searchText=katana%3FsearchText%3Dkatana%3F?searchText=katana%3FsearchText%3Dkatana%3F
https://www.blackrock.com/corporate/search/summary-search-results?searchText=katana%3FsearchText%3Dkatana%3F?
https://www.blackrock.com/corporate/search/summary-search-results?searchText=katana%3FsearchText%3Dkatana%3FsearchText%3Dkatana%3FsearchText%3Dkatana%3FsearchText%3Dkatana%3FsearchText%3Dkatana%3FsearchText%3Dkatana%3FsearchText%3Dkatana
https://www.blackrock.com/corporate/search/summary-search-results?doTickerSearch=true&searchText=katana?searchText=katana?searchText=katana%3FsearchText%3Dkatana?searchText=katana%3FsearchText%3Dkatana%3FsearchText%3Dkatana%3FsearchText%3Dkatana?searchText=katana%3FsearchText%3Dkatana%3FsearchText%3Dkatana%3FsearchText%3Dkatana%3FsearchText%3Dkatana%3FsearchText%3Dkatana%3FsearchText%3Dkatana%3FsearchText%3Dkatana
https://www.blackrock.com/corporate/search/summary-search-results?doTickerSearch=true&searchText=katana?searchText=katana?searchText=katana%3FsearchText%3Dkatana?searchText=katana%3FsearchText%3Dkatana%3FsearchText%3Dkatana%3FsearchText%3Dkatana?
https://www.blackrock.com/corporate/search/summary-search-results?searchText=katana%3FsearchText%3Dkatana%3FsearchText%3Dkatana%3FsearchText%3Dkatana?searchText=katana%3FsearchText%3Dkatana%3FsearchText%3Dkatana%3FsearchText%3Dkatana
https://www.blackrock.com/corporate/search/summary-search-results?searchText=katana%3FsearchText%3Dkatana%3FsearchText%3Dkatana%3FsearchText%3Dkatana?
https://www.blackrock.com/corporate/search/summary-search-results?searchText=katana%3FsearchText%3Dkatana?searchText=katana%3FsearchText%3Dkatana?searchText=katana%3FsearchText%3Dkatana%3FsearchText%3Dkatana%3FsearchText%3Dkatana
https://www.blackrock.com/corporate/search/summary-search-results?searchText=katana%3FsearchText%3Dkatana?searchText=katana%3FsearchText%3Dkatana?
Notice katana keeps on appending searchText=katana? / searchText=katana??? to the already requested URL, This should only happen only when a new previously undiscovered parameter is found via spidering, Only then it should be crawled.
Note: it seems like a bug in the deduplication mechanism in place (probably the value used as key in the map)
@Mzack9999 : Also i thing there is a bug in automatic request construction in case of auto form filling enabled that ?
is added to the url not respecting wether it already exists or not. This will result in server not processing the parameters crawled by katana. There should be a check that there should be only one ?
when katana constructs a url.
I've added a generic loop breakout mechanism, but I'm unable to reproduce the behavior in the dev
branch:
$ go run . -u https://www.blackrock.com/corporate/search/ -d 5 -cs searchText -v
__ __
/ /_____ _/ /____ ____ ___ _
/ '_/ _ / __/ _ / _ \/ _ /
/_/\_\\_,_/\__/\_,_/_//_/\_,_/ v0.0.2
projectdiscovery.io
[WRN] Use with caution. You are responsible for your actions.
[WRN] Developers assume no liability and are not responsible for any misuse or damage.
@parthmalhotra Do I need any particular configuration file?
Hello @Mzack9999 : As -aff was disabled by default , this should be reproducible by providing the aff flag:
katana -u https://www.blackrock.com/corporate/search/ -d 5 -cs searchText -aff