robots-txt topic
List
robots-txt repositories
the-great-gpt-firewall
76
Stars
4
Forks
Watchers
🤖 A curated list of websites that restrict access to AI Agents, AI crawlers and GPTs
scrapeGPT
67
Stars
9
Forks
Watchers
ScrapeGPT is a RAG-based Telegram bot designed to scrape and analyze websites, then answer questions based on the scraped content. The bot utilizes Retrieval Augmented Generation and webscraping to re...
Site-Scanner
49
Stars
14
Forks
Watchers
Site-Scanner - Web application vulnerability assessment tool.