Support static assets where file length may change, e.g. logs

[broccolihighkicks]

This is a bit of an oxymoron.

I am serving a log.txt file for a background process using the Datasette --static CLI. This is useful as I can observe a background process from the web UI to see any errors that occur (instead of spelunking the logs via docker exec/ssh etc).

I get this error, which I think is because Datasette assumes that the size of the content does not change (but appending new log lines means the content length changes).

Traceback (most recent call last):
  File "/usr/local/lib/python3.9/site-packages/datasette/app.py", line 1181, in route_path
    response = await view(request, send)
  File "/usr/local/lib/python3.9/site-packages/datasette/utils/asgi.py", line 305, in inner_static
    await asgi_send_file(send, full_path, chunk_size=chunk_size)
  File "/usr/local/lib/python3.9/site-packages/datasette/utils/asgi.py", line 280, in asgi_send_file
    await send(
  File "/usr/local/lib/python3.9/site-packages/asgi_csrf.py", line 104, in wrapped_send
    await send(event)
  File "/usr/local/lib/python3.9/site-packages/uvicorn/protocols/http/h11_impl.py", line 460, in send
    output = self.conn.send(event)
  File "/usr/local/lib/python3.9/site-packages/h11/_connection.py", line 468, in send
    data_list = self.send_with_data_passthrough(event)
  File "/usr/local/lib/python3.9/site-packages/h11/_connection.py", line 501, in send_with_data_passthrough
    writer(event, data_list.append)
  File "/usr/local/lib/python3.9/site-packages/h11/_writers.py", line 58, in __call__
    self.send_data(event.data, write)
  File "/usr/local/lib/python3.9/site-packages/h11/_writers.py", line 78, in send_data
    raise LocalProtocolError("Too much data for declared Content-Length")
h11._util.LocalProtocolError: Too much data for declared Content-Length
ERROR:    Exception in ASGI application
Traceback (most recent call last):
  File "/usr/local/lib/python3.9/site-packages/datasette/app.py", line 1181, in route_path
    response = await view(request, send)
  File "/usr/local/lib/python3.9/site-packages/datasette/utils/asgi.py", line 305, in inner_static
    await asgi_send_file(send, full_path, chunk_size=chunk_size)
  File "/usr/local/lib/python3.9/site-packages/datasette/utils/asgi.py", line 280, in asgi_send_file
    await send(
  File "/usr/local/lib/python3.9/site-packages/asgi_csrf.py", line 104, in wrapped_send
    await send(event)
  File "/usr/local/lib/python3.9/site-packages/uvicorn/protocols/http/h11_impl.py", line 460, in send
    output = self.conn.send(event)
  File "/usr/local/lib/python3.9/site-packages/h11/_connection.py", line 468, in send
    data_list = self.send_with_data_passthrough(event)
  File "/usr/local/lib/python3.9/site-packages/h11/_connection.py", line 501, in send_with_data_passthrough
    writer(event, data_list.append)
  File "/usr/local/lib/python3.9/site-packages/h11/_writers.py", line 58, in __call__
    self.send_data(event.data, write)
  File "/usr/local/lib/python3.9/site-packages/h11/_writers.py", line 78, in send_data
    raise LocalProtocolError("Too much data for declared Content-Length")
h11._util.LocalProtocolError: Too much data for declared Content-Length

Thanks, I am finding Datasette very useful.

[simonw]

Hah, this is certainly unexpected.

It looks like this is the code in question: https://github.com/simonw/datasette/blob/a6ff123de5464806441f6a6f95145c9a83b7f20b/datasette/utils/asgi.py#L259-L266

You're right: it assumes that the file it is serving won't change length while it is serving it.

[simonw]

The reason I implemented it like this was to support things like the curl progress bar if users decide to serve up large files using the --static mechanism.

Here's the code that hooks it up to the URL resolver:

https://github.com/simonw/datasette/blob/458f03ad3a454d271f47a643f4530bd8b60ddb76/datasette/app.py#L1001-L1005

Which uses this function:

https://github.com/simonw/datasette/blob/a6ff123de5464806441f6a6f95145c9a83b7f20b/datasette/utils/asgi.py#L285-L310

One option here would be to support a workaround that looks something like this:

http://localhost:8001/my-static/log.txt?_unknown_size=1`

The URL routing code could then look out for that ?_unknown_size=1 option and, if it's present, omit the content-length header entirely.

It's a bit of a cludge, but it would be pretty straight-forward to implement.

Would that work for you @broccolihighkicks?