Simon Willison

Keeping this issue open until I've proven the new plugin hook works by releasing a plugin that uses it.

Documentation for the new hook: https://docs.datasette.io/en/latest/plugin_hooks.html#handle-exception-datasette-request-exception

I've tested this with `datasette-sentry` and it works well.

This will probably happen as part of turning this into an officially documented API that serves the template context for the homepage: - #1510

Wrote more about that here: https://simonwillison.net/2022/Apr/27/parallel-queries/ Compare https://latest-with-plugins.datasette.io/github/commits?_facet=repo&_facet=committer&_trace=1  With the same thing but with parallel execution disabled: https://latest-with-plugins.datasette.io/github/commits?_facet=repo&_facet=committer&_trace=1&_noparallel=1  Those total page load time numbers are very similar. Is...

I just remembered the `--setting num_sql_threads` option... which defaults to 3! https://github.com/simonw/datasette/blob/942411ef946e9a34a2094944d3423cddad27efd3/datasette/app.py#L109-L113 Would explain why the first trace never seems to show more than three SQL queries executing at once.

One weird thing: I noticed that in the parallel trace above the SQL query bars are wider. Mousover shows duration in ms, and I got 13ms for this query: select...

Relevant: here's the code that sets up a Datasette SQLite connection: https://github.com/simonw/datasette/blob/7a6654a253dee243518dc542ce4c06dbb0d0801d/datasette/database.py#L73-L96 It's using `check_same_thread=False` - here's [the Python docs on that](https://docs.python.org/3/library/sqlite3.html#sqlite3.connect): > By default, *check_same_thread* is [`True`](https://docs.python.org/3/library/constants.html#True "True") and...

I think I need some much more in-depth tracing tricks for this. https://www.maartenbreddels.com/perf/jupyter/python/tracing/gil/2021/01/14/Tracing-the-Python-GIL.html looks relevant - uses the `perf` tool on Linux.

Also useful: https://avi.im/blag/2021/fast-sqlite-inserts/ - from a tip on Twitter: https://twitter.com/ricardoanderegg/status/1519402047556235264