More Astro work

[jacobsandersen]

Good evening folks.

This PR does some more stuff to further the Astro agenda.

• Allow indexing in robots.txt
• ~~Update domain in astro.config.mjs~~
• Use web component in SoftwarePreview to bring web requests clientside
• Use web component in Terminal to bring web request clientside

The use of web components both

1. Avoids importing a bulky UI framework
2. Massively speeds up the following pages:

• /downloads
• /javadocs
• /

So far...

Once again, you can merge or leave it here. Either way, more to come later.