impler.io icon indicating copy to clipboard operation
impler.io copied to clipboard

Published 20 hours ago • verified publisher icon implerhq

Add robots.txt file for API, Embed, Widget & Web Applications

Open chavda-bhavik opened this issue 1 year ago • 0 comments

Describe the bug Due to API routes getting indexed in the Google search console, it's harming the application's reputation. Also, API, Embed, Widget & Web applications are not meant to be crawled by Google or any other search engines.

Expected behavior API, Embed, Widget & Web applications should have robots.txt files where it should block crawlers from crawling. Expect these paths in web applications,

  • https://web.impler.io/auth/signin
  • https://web.impler.io/auth/signup

Additional context Add any other context about the problem here.

chavda-bhavik avatar Aug 09 '24 06:08 chavda-bhavik