Farooq Sheikh issues

Repositories
Issues
Comments

Results 1 issues of


                                            Farooq Sheikh

Does FESS supports Crawl rate limiting in robots.txt

Hi @marevol I have checked FESS respects Disallow for robots.txt but i am unable to verify Crawl-delay and Request-rate. Can you please confirm is it implemented? https://www.promptcloud.com/blog/how-to-read-and-respect-robots-file 4. Crawl rate...

contributions welcome