Improve SEO for docs pages

[carlosgmartin]

When I Google a Flax module to pull up its API documentation, I get broken or irrelevant pages. Examples:

• flax dense | top result | desired page
• flax layernorm | top result | desired page
• flax multi head attention | top result | desired page
• flax conv | top result | desired page

Is there something that can be done on the website side to improve this?

[chiamp]

Waiting to see if Google search will stop indexing the deprecated _autosummary links after adding robots.txt

[carlosgmartin]

@chiamp Not sure if the current changes take care of this, but it would be nice to not only remove the dead links, but to also bring the correct API pages to the fore.