Simon Willison

One thing that's clear is I need to add a demo to the fixtures that illustrates the feature, ideally with a thousand generated rows in it to give a better...

My reply on Twitter: https://twitter.com/simonw/status/1148254258900262913 > Huh I hadn't thought about order-by-date - that's really useful. Not sure what I'll do for data that's spread over e.g. 1,000 days (other...

Having a "more link" is very interesting. I could do that by generating a custom SQL query link, but that wouldn't play well in instances that disable custom SQL. This...

A few options for how this would work: - `datasette ... --robots allow` - `datasette ... --setting robots allow` Options could be: - `allow` - allow all crawling - `deny`...

I could try out the `X-Robots` HTTP header too: https://developers.google.com/search/docs/advanced/robots/robots_meta_tag#xrobotstag

https://twitter.com/mal/status/1424825895139876870 > True pinging google should be part of the build process on a static site :) That's another aspect of this: if you DO want your site crawled, teaching...

At the very least Datasette should serve a blank `/robots.txt` by default - I'm seeing a ton of 404s for it in the logs.

I think this feature is out-of-scope for Datasette core, but it could make a great plugin. It would need to use the ASGI middleware plugin, then implement its own state...

Actually it looks like you can send a `sitemap.xml` to Google using an unauthenticated GET request to: https://www.google.com/ping?sitemap=FULL_URL_OF_SITEMAP According to https://developers.google.com/search/docs/advanced/sitemaps/build-sitemap

Bing's equivalent is: https://www.bing.com/webmasters/help/Sitemaps-3b5cf6ed http://www.bing.com/ping?sitemap=FULL_URL_OF_SITEMAP