500 error if sorted by a column not in the ?_col= list

[simonw]

For example: https://latest.datasette.io/fixtures/sortable?_sort_desc=sortable&_col=sortable_with_nulls

That's ?_sort_desc=sortable&_col=sortable_with_nulls

[simonw]

Traceback:

Traceback (most recent call last):
  File "/Users/simon/Dropbox/Development/datasette/datasette/app.py", line 1264, in route_path
    response = await view(request, send)
  File "/Users/simon/Dropbox/Development/datasette/datasette/views/base.py", line 134, in view
    return await self.dispatch_request(request)
  File "/Users/simon/Dropbox/Development/datasette/datasette/views/base.py", line 91, in dispatch_request
    return await handler(request)
  File "/Users/simon/Dropbox/Development/datasette/datasette/views/base.py", line 361, in get
    response_or_template_contexts = await self.data(request, **data_kwargs)
  File "/Users/simon/Dropbox/Development/datasette/datasette/views/table.py", line 157, in data
    return await self._data_traced(request, default_labels, _next, _size)
  File "/Users/simon/Dropbox/Development/datasette/datasette/views/table.py", line 633, in _data_traced
    prefix = rows[-2][sort or sort_desc]
IndexError: No item with that key

That's this code here: https://github.com/simonw/datasette/blob/7af67b54b7d9bca43e948510fc62f6db2b748fa8/datasette/views/table.py#L631-L645

[simonw]

So the problem here is that in generating the ?_next= next page link we need the value from the specified sort column - but we're not selecting it any more.

Possible fixes:

• Always include the sort column in the list of columns that are selected, then filter that out before they are displayed
• Use a second query to figure out the _sort or _sort_desc value for that last row, since we know its primary key (we always select primary keys)

Not sure which solution is more elegant. I think it might be the second one.

[simonw]

So code would look something like this:

try:
    prefix = rows[-2][sort or sort_desc]
except KeyError:
    # Didn't select sort/sort_desc column - look up value by primary key instead
    primary_key = rows[-2]["pk"] # But more complex than this
    prefix = (await db.execute("select * from {table} where pk = ?", [primary_key])).first_value()

[simonw]

https://latest.datasette.io/fixtures/sortable?_sort_desc=sortable&_col=sortable_with_nulls works now.