Revert search engine mitigation when it isn't needed, or make it permanent

[sjahl]

[sjahl]

We left the search engine user-agent ban in place on the prod website until we can get the robots.txt configuration deployed. We should remember to remove the mitigation once we don't think it's needed anymore or ensure that it is encoded in our deployment if we want to keep it permanently.

[rileyhgrant]

Robots.txt is deployed on prod, https://gnomad.broadinstitute.org/robots.txt

Next step is to disable cloud armor rules banning petalbot and see what the results are.