katana
katana copied to clipboard
Published
20 hours ago •
projectdiscovery
projectdiscovery
Don't crawl same path when the page outputs visited url in response
Some websites append the current url which is being requested in the page content/either in an tag or something else, What happens here is that katana gets stuck in a loop where it tries to recrawl the same url.
E.g :
katana -u https://www.blackrock.com/corporate/search/ -d 5 -cs searchText
Response :
https://www.blackrock.com/corporate/search/summary-search-results?doTickerSearch=true&searchText=katana
https://www.blackrock.com/corporate/search/summary-search-results?searchText=katana
https://www.blackrock.com/corporate/search/summary-search-results?doTickerSearch=true&searchText=katana?searchText=katana
https://www.blackrock.com/corporate/search/summary-search-results?doTickerSearch=true&searchText=katana?
https://www.blackrock.com/corporate/search/summary-search-results?doTickerSearch=true&searchText=katana?searchText=katana&searchType=generalSearch
https://www.blackrock.com/corporate/search/summary-search-results?searchText=katana%3FsearchText%3Dkatana
https://www.blackrock.com/corporate/search/summary-search-results?doTickerSearch=true&searchText=katana?searchText=katana?searchText=katana%3FsearchText%3Dkatana
https://www.blackrock.com/corporate/search/summary-search-results?doTickerSearch=true&searchText=katana?searchText=katana?
https://www.blackrock.com/corporate/search/summary-search-results?searchText=katana?searchText=katana
https://www.blackrock.com/corporate/search/summary-search-results?searchText=katana?
https://www.blackrock.com/corporate/search/summary-search-results?searchText=katana?searchText=katana&searchType=generalSearch
https://www.blackrock.com/corporate/search/summary-search-results?searchText=katana%3F
https://www.blackrock.com/corporate/search/summary-search-results?doTickerSearch=true&searchText=katana??searchText=katana%3F
https://www.blackrock.com/corporate/search/summary-search-results?doTickerSearch=true&searchText=katana??
https://www.blackrock.com/corporate/search/summary-search-results?doTickerSearch=true&searchText=katana?searchText=katana&searchType=generalSearch?searchText=katana%3FsearchText%3Dkatana&searchType=generalSearch
https://www.blackrock.com/corporate/search/summary-search-results?doTickerSearch=true&searchText=katana?searchText=katana&searchType=generalSearch?
https://www.blackrock.com/corporate/search/summary-search-results?searchText=katana%3FsearchText%3Dkatana?searchText=katana%3FsearchText%3Dkatana
https://www.blackrock.com/corporate/search/summary-search-results?searchText=katana%3FsearchText%3Dkatana?
https://www.blackrock.com/corporate/search/summary-search-results?searchText=katana%3FsearchText%3Dkatana%3FsearchText%3Dkatana%3FsearchText%3Dkatana
https://www.blackrock.com/corporate/search/summary-search-results?doTickerSearch=true&searchText=katana?searchText=katana?searchText=katana%3FsearchText%3Dkatana?searchText=katana%3FsearchText%3Dkatana%3FsearchText%3Dkatana%3FsearchText%3Dkatana
https://www.blackrock.com/corporate/search/summary-search-results?doTickerSearch=true&searchText=katana?searchText=katana?searchText=katana%3FsearchText%3Dkatana?
https://www.blackrock.com/corporate/search/summary-search-results?searchText=katana%3FsearchText%3Dkatana%3F
https://www.blackrock.com/corporate/search/summary-search-results?doTickerSearch=true&searchText=katana?searchText=katana??searchText=katana%3FsearchText%3Dkatana%3F
https://www.blackrock.com/corporate/search/summary-search-results?doTickerSearch=true&searchText=katana?searchText=katana??
https://www.blackrock.com/corporate/search/summary-search-results?searchText=katana?searchText=katana?searchText=katana%3FsearchText%3Dkatana
https://www.blackrock.com/corporate/search/summary-search-results?searchText=katana?searchText=katana?
https://www.blackrock.com/corporate/search/summary-search-results?searchText=katana??searchText=katana%3F
https://www.blackrock.com/corporate/search/summary-search-results?searchText=katana??
https://www.blackrock.com/corporate/search/summary-search-results?searchText=katana?searchText=katana&searchType=generalSearch?searchText=katana%3FsearchText%3Dkatana&searchType=generalSearch
https://www.blackrock.com/corporate/search/summary-search-results?searchText=katana?searchText=katana&searchType=generalSearch?
https://www.blackrock.com/corporate/search/summary-search-results?searchText=katana%3F?searchText=katana%3F
https://www.blackrock.com/corporate/search/summary-search-results?searchText=katana%3F?
https://www.blackrock.com/corporate/search/summary-search-results?searchText=katana%3F%3FsearchText%3Dkatana%3F
https://www.blackrock.com/corporate/search/summary-search-results?doTickerSearch=true&searchText=katana??searchText=katana%3F?searchText=katana%3F%3FsearchText%3Dkatana%3F
https://www.blackrock.com/corporate/search/summary-search-results?doTickerSearch=true&searchText=katana??searchText=katana%3F?
https://www.blackrock.com/corporate/search/summary-search-results?searchText=katana%3F%3F
https://www.blackrock.com/corporate/search/summary-search-results?doTickerSearch=true&searchText=katana???searchText=katana%3F%3F
https://www.blackrock.com/corporate/search/summary-search-results?doTickerSearch=true&searchText=katana???
https://www.blackrock.com/corporate/search/summary-search-results?searchText=katana%3FsearchText%3Dkatana??searchText=katana%3FsearchText%3Dkatana%3F
https://www.blackrock.com/corporate/search/summary-search-results?searchText=katana%3FsearchText%3Dkatana??
https://www.blackrock.com/corporate/search/summary-search-results?searchText=katana%3FsearchText%3Dkatana%3FsearchText%3Dkatana%3FsearchText%3Dkatana%3F
https://www.blackrock.com/corporate/search/summary-search-results?doTickerSearch=true&searchText=katana?searchText=katana?searchText=katana%3FsearchText%3Dkatana??searchText=katana%3FsearchText%3Dkatana%3FsearchText%3Dkatana%3FsearchText%3Dkatana%3F
https://www.blackrock.com/corporate/search/summary-search-results?doTickerSearch=true&searchText=katana?searchText=katana?searchText=katana%3FsearchText%3Dkatana??
https://www.blackrock.com/corporate/search/summary-search-results?searchText=katana%3FsearchText%3Dkatana%3F%3F
https://www.blackrock.com/corporate/search/summary-search-results?doTickerSearch=true&searchText=katana?searchText=katana???searchText=katana%3FsearchText%3Dkatana%3F%3F
https://www.blackrock.com/corporate/search/summary-search-results?doTickerSearch=true&searchText=katana?searchText=katana???
https://www.blackrock.com/corporate/search/summary-search-results?searchText=katana?searchText=katana??searchText=katana%3FsearchText%3Dkatana%3F
https://www.blackrock.com/corporate/search/summary-search-results?searchText=katana?searchText=katana??
https://www.blackrock.com/corporate/search/summary-search-results?searchText=katana???searchText=katana%3F%3F
https://www.blackrock.com/corporate/search/summary-search-results?searchText=katana???
https://www.blackrock.com/corporate/search/summary-search-results?searchText=katana%3F??searchText=katana%3F%3F
https://www.blackrock.com/corporate/search/summary-search-results?searchText=katana%3F??
https://www.blackrock.com/corporate/search/summary-search-results?searchText=katana%3F%3FsearchText%3Dkatana%3F%3F
https://www.blackrock.com/corporate/search/summary-search-results?doTickerSearch=true&searchText=katana??searchText=katana%3F??searchText=katana%3F%3FsearchText%3Dkatana%3F%3F
https://www.blackrock.com/corporate/search/summary-search-results?doTickerSearch=true&searchText=katana??searchText=katana%3F??
https://www.blackrock.com/corporate/search/summary-search-results?searchText=katana%3F%3F%3F
https://www.blackrock.com/corporate/search/summary-search-results?doTickerSearch=true&searchText=katana????searchText=katana%3F%3F%3F
https://www.blackrock.com/corporate/search/summary-search-results?doTickerSearch=true&searchText=katana????
https://www.blackrock.com/corporate/search/summary-search-results?searchText=katana%3F%3F%3FsearchText%3Dkatana%3F%3F
https://www.blackrock.com/corporate/search/summary-search-results?doTickerSearch=true&searchText=katana???searchText=katana%3F%3F?searchText=katana%3F%3F%3FsearchText%3Dkatana%3F%3F
https://www.blackrock.com/corporate/search/summary-search-results?doTickerSearch=true&searchText=katana???searchText=katana%3F%3F?
https://www.blackrock.com/corporate/search/summary-search-results?searchText=katana%3F%3F?searchText=katana%3F%3F
https://www.blackrock.com/corporate/search/summary-search-results?searchText=katana%3F%3F?
https://www.blackrock.com/corporate/search/summary-search-results?searchText=katana%3F%3FsearchText%3Dkatana%3F%3FsearchText%3Dkatana%3F%3FsearchText%3Dkatana%3F
https://www.blackrock.com/corporate/search/summary-search-results?doTickerSearch=true&searchText=katana??searchText=katana%3F?searchText=katana%3F%3FsearchText%3Dkatana%3F?searchText=katana%3F%3FsearchText%3Dkatana%3F%3FsearchText%3Dkatana%3F%3FsearchText%3Dkatana%3F
https://www.blackrock.com/corporate/search/summary-search-results?doTickerSearch=true&searchText=katana??searchText=katana%3F?searchText=katana%3F%3FsearchText%3Dkatana%3F?
https://www.blackrock.com/corporate/search/summary-search-results?searchText=katana%3F%3FsearchText%3Dkatana%3F?searchText=katana%3F%3FsearchText%3Dkatana%3F
https://www.blackrock.com/corporate/search/summary-search-results?searchText=katana%3F%3FsearchText%3Dkatana%3F?
https://www.blackrock.com/corporate/search/summary-search-results?searchText=katana%3F?searchText=katana%3F?searchText=katana%3F%3FsearchText%3Dkatana%3F
https://www.blackrock.com/corporate/search/summary-search-results?searchText=katana%3F?searchText=katana%3F?
https://www.blackrock.com/corporate/search/summary-search-results?searchText=katana??searchText=katana%3F?searchText=katana%3F%3FsearchText%3Dkatana%3F
https://www.blackrock.com/corporate/search/summary-search-results?searchText=katana??searchText=katana%3F?
https://www.blackrock.com/corporate/search/summary-search-results?searchText=katana?searchText=katana?searchText=katana%3FsearchText%3Dkatana?searchText=katana%3FsearchText%3Dkatana%3FsearchText%3Dkatana%3FsearchText%3Dkatana
https://www.blackrock.com/corporate/search/summary-search-results?searchText=katana?searchText=katana?searchText=katana%3FsearchText%3Dkatana?
https://www.blackrock.com/corporate/search/summary-search-results?searchText=katana%3FsearchText%3Dkatana%3F%3FsearchText%3Dkatana%3FsearchText%3Dkatana%3F
https://www.blackrock.com/corporate/search/summary-search-results?doTickerSearch=true&searchText=katana?searchText=katana??searchText=katana%3FsearchText%3Dkatana%3F?searchText=katana%3FsearchText%3Dkatana%3F%3FsearchText%3Dkatana%3FsearchText%3Dkatana%3F
https://www.blackrock.com/corporate/search/summary-search-results?doTickerSearch=true&searchText=katana?searchText=katana??searchText=katana%3FsearchText%3Dkatana%3F?
https://www.blackrock.com/corporate/search/summary-search-results?searchText=katana%3FsearchText%3Dkatana%3F?searchText=katana%3FsearchText%3Dkatana%3F
https://www.blackrock.com/corporate/search/summary-search-results?searchText=katana%3FsearchText%3Dkatana%3F?
https://www.blackrock.com/corporate/search/summary-search-results?searchText=katana%3FsearchText%3Dkatana%3FsearchText%3Dkatana%3FsearchText%3Dkatana%3FsearchText%3Dkatana%3FsearchText%3Dkatana%3FsearchText%3Dkatana%3FsearchText%3Dkatana
https://www.blackrock.com/corporate/search/summary-search-results?doTickerSearch=true&searchText=katana?searchText=katana?searchText=katana%3FsearchText%3Dkatana?searchText=katana%3FsearchText%3Dkatana%3FsearchText%3Dkatana%3FsearchText%3Dkatana?searchText=katana%3FsearchText%3Dkatana%3FsearchText%3Dkatana%3FsearchText%3Dkatana%3FsearchText%3Dkatana%3FsearchText%3Dkatana%3FsearchText%3Dkatana%3FsearchText%3Dkatana
https://www.blackrock.com/corporate/search/summary-search-results?doTickerSearch=true&searchText=katana?searchText=katana?searchText=katana%3FsearchText%3Dkatana?searchText=katana%3FsearchText%3Dkatana%3FsearchText%3Dkatana%3FsearchText%3Dkatana?
https://www.blackrock.com/corporate/search/summary-search-results?searchText=katana%3FsearchText%3Dkatana%3FsearchText%3Dkatana%3FsearchText%3Dkatana?searchText=katana%3FsearchText%3Dkatana%3FsearchText%3Dkatana%3FsearchText%3Dkatana
https://www.blackrock.com/corporate/search/summary-search-results?searchText=katana%3FsearchText%3Dkatana%3FsearchText%3Dkatana%3FsearchText%3Dkatana?
https://www.blackrock.com/corporate/search/summary-search-results?searchText=katana%3FsearchText%3Dkatana?searchText=katana%3FsearchText%3Dkatana?searchText=katana%3FsearchText%3Dkatana%3FsearchText%3Dkatana%3FsearchText%3Dkatana
https://www.blackrock.com/corporate/search/summary-search-results?searchText=katana%3FsearchText%3Dkatana?searchText=katana%3FsearchText%3Dkatana?
Notice katana keeps on appending searchText=katana? / searchText=katana??? to the already requested URL, This should only happen only when a new previously undiscovered parameter is found via spidering, Only then it should be crawled.
Note: it seems like a bug in the deduplication mechanism in place (probably the value used as key in the map)
@Mzack9999 : Also i thing there is a bug in automatic request construction in case of auto form filling enabled that ? is added to the url not respecting wether it already exists or not. This will result in server not processing the parameters crawled by katana. There should be a check that there should be only one ? when katana constructs a url.
I've added a generic loop breakout mechanism, but I'm unable to reproduce the behavior in the dev branch:
$ go run . -u https://www.blackrock.com/corporate/search/ -d 5 -cs searchText -v
__ __
/ /_____ _/ /____ ____ ___ _
/ '_/ _ / __/ _ / _ \/ _ /
/_/\_\\_,_/\__/\_,_/_//_/\_,_/ v0.0.2
projectdiscovery.io
[WRN] Use with caution. You are responsible for your actions.
[WRN] Developers assume no liability and are not responsible for any misuse or damage.
@parthmalhotra Do I need any particular configuration file?
Hello @Mzack9999 : As -aff was disabled by default , this should be reproducible by providing the aff flag:
katana -u https://www.blackrock.com/corporate/search/ -d 5 -cs searchText -aff