Inability to List Both Normal URLs and Extensions in a Single Operation

[swdbo]

Katana Version:

v1.0.5

Current Behavior:

Currently, Katana does not provide a way to list both normal URLs and specific file extensions in one operation. Users can either obtain a list of URLs using the default settings or apply the extension match option (-em) to filter for specific file extensions. However, when using -em, URLs without an extension are omitted from the final output.

Desired Behavior:

Modify Katana's behavior to list all URLs and file extensions by default. Introduce functionality for the -em option to serve as a filter that includes only the specified file extensions in the output. This change would allow users to see the complete set of resources initially and have the option to narrow down the results based on specific extension criteria, enhancing usability and flexibility.

Steps To Reproduce (Current Behavior):

1. Run Katana with the command: katana -u https://chaos.projectdiscovery.io -headless -depth 2
2. Observe that only URLs are listed, and file extensions are not included.
3. Execute Katana with extension filtering: katana -u https://chaos.projectdiscovery.io -headless -depth 2 -em css,js,ico,jpg,png,html
4. Notice the inclusion of specified extensions in the results, but URLs without an extension are missing.

Results

katana -u https://chaos.projectdiscovery.io -headless -depth 2 

https://chaos.projectdiscovery.io
https://chaos.projectdiscovery.io/app.bundle.css
https://chaos.projectdiscovery.io/app.js

katana -u https://chaos.projectdiscovery.io -headless -depth 2 -em css,js,ico,jpg,png,html 

https://chaos.projectdiscovery.io/fevicon.png
https://chaos.projectdiscovery.io/app.bundle.css
https://chaos.projectdiscovery.io/app.js
https://chaos.projectdiscovery.io/361bc8b680f5b7c8f0bd7fb587ea7666.png
https://chaos.projectdiscovery.io/326b684b7243f6148e7ec7dcd3ba1d5b.png
https://chaos.projectdiscovery.io/e9b61c5e5a0c43cdcd96fcc568af8e36.png

Proposed Fix:

Implement changes to the crawling and listing mechanism to display all accessible URLs and assets by default. Adjust the -em flag functionality to act as a post-crawl filter that refines the output to include only the assets with the specified extensions. This approach ensures a comprehensive view of the site's resources is available by default, with the flexibility to focus on specific types of files as needed.

Benefits:

• Provides a complete overview of all site resources without the need to run multiple commands.
• Enhances user efficiency by simplifying the process of targeting specific file types.
• Improves Katana's flexibility and adaptability to different use cases.

[olearycrew]

Thanks for this issue @swdbo - I do think this is intended behavior but your ideas could be great enhancements.

[olearycrew]

cc @Mzack9999