katana
katana copied to clipboard
A next-generation crawling and spidering framework.
I tried to add the detection of textearea and select in a form, respecting what was already implemented. This is the first time I've worked on a GO project. I...
### katana version: Katana version: v1.1.0 ### Current Behavior: If cookies are added to customHeaders, they are not integrated directly into the browser. As a result, they are used when...
"Dynamic Scope" integration, cuts back on data usage while crawling by utilizing a TF-IDF machine learning model to discard pages which might be too similar to pages already crawled. 👍🏻...
### katana version: v1.1.0 ### Current Behavior: When crawling a site with the `-jsonl` flag, the response headers are lowercased, and `-` characters are converted to `_`. This is misleading,...
### Current Behavior: I'm currently crawling a headless site in order to detect as many XHR requests as possible. While debugging, I noticed that some pages are well loaded in...
It seems like we should review the proposed solution based upon https://github.com/projectdiscovery/katana/issues/876#issuecomment-2151797550. Probably form filling should be reconsidered/disabled as a whole for headless navigation as the way we are doing...
使用的命令 -headless -no-sandbox -list url.txt -jc -kf robotstxt,sitemapxml -aff,使用的时候经常会卡住不动,一直不动的那种,查看是存在大量chrme线程没有结束导致,需要自己结束所有chrome线程 pkill -9 chrome 才能继续运行 ```[tasklist] ### Tasks ```
1. Error Handling: Centralized error handling with the handleError function to reduce code repetition. 2. Function Decomposition: Split the main function and readFlags function into smaller functions for better readability...
### katana version: `main|dev` ### Current Behavior: The function `WebUserAgent()` in `pkg.utils.utils.go` uses an hard-coded user agent: ``` "Mozilla/5.0 (Macintosh; Intel Mac OS X 11_1) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/87.0.4280.88...
For form extraction, the "value" attribute of an element is used. When the value is not present, katana uses hardcoded placeholders. E.g. the value "katana". This patch favors using the...