Allow customizing robots.txt via config

[APB9785]

[leandrocp]

Should we leave docs recommending to disallow /admin or whatever path you chose for the admin interface? If the admin was mounted in the same app. It doesn't block bots to accessing /admin but that's a good practice.