Add dynamic robots.txt from API

[bookernath]

What/Why?

Use new robots.txt node to dynamically populate robots.txt details from control panel, replacing previous hardcoded approach.

We still dynamically insert the sitemap URL based on the env.

Adding do not merge label until the API is fully rolled out.

Testing

Preview deployment