datasette icon indicating copy to clipboard operation
datasette copied to clipboard

Block or rate limit based on User Agent?

Open louispotok opened this issue 8 months ago • 1 comments

I'm getting traffic from facebookexternalhit user agents -- it's not a huge amount (2req/s) but the bill starts to add up. From what I can tell, this is the facebook crawler which does not mention robots.txt (vs FacebookBot which seems to respect it. This SO thread claims that the Crawler doesn't respect robots.txt, so datasette-block-robots doesn't seem to solve this.

Is there another way to block or rate limit a given user agent in datasette? I'm deploying on Google Cloud if that's relevant. Thanks!

louispotok avatar Jun 28 '24 04:06 louispotok