datasette
datasette copied to clipboard
Block or rate limit based on User Agent?
I'm getting traffic from facebookexternalhit
user agents -- it's not a huge amount (2req/s) but the bill starts to add up. From what I can tell, this is the facebook crawler which does not mention robots.txt
(vs FacebookBot which seems to respect it. This SO thread claims that the Crawler doesn't respect robots.txt
, so datasette-block-robots
doesn't seem to solve this.
Is there another way to block or rate limit a given user agent in datasette? I'm deploying on Google Cloud if that's relevant. Thanks!