OWASP
AI Crawlers
What would you like added? Insert https://techcrunch.com/2025/03/27/open-source-devs-are-fighting-ai-crawlers-with-cleverness-and-vengeance/ into https://github.com/OWASP/wstg/blob/master/document/4-Web_Application_Security_Testing/01-Information_Gathering/03-Review_Webserver_Metafiles_for_Information_Leakage.md#robots
Would you like to be assigned to this issue? Check the box if you will submit a PR to add the proposed content. Please read CONTRIBUTING.md.
- [X] Assign me, please!
Do you want to put the techcrunch content directly (unchanged) into owasp or do you want to write a new section with the same idea?
Hello @kingthorin , I will like to contribute to this issue. I can draft a short section explaining how some AI crawlers ignore robots.txt dissalowed files&folders and why it’s a security concern, along with safer alternatives. Please assign this issue to me if it’s still open. Thankyou !!
@theAnuragMishra wrote:
Do you want to put the techcrunch content directly (unchanged) into owasp or do you want to write a new section with the same idea?
A new subsection that references TechCrunch and others.
@vishantgawali1811
I will like to contribute to this issue. I can draft a short section explaining how some AI crawlers ignore robots.txt dissalowed files&folders and why it’s a security concern, along with safer alternatives. Please assign this issue to me if it’s still open. Thankyou !!
You can still submit a Pull Request.