quickwit
quickwit copied to clipboard
consider not retuning all columns from metastore when listing splits
splits has 12 columns, while we filter based on most of these, we rarely need to fetch more than split_id index_uid and possibly split_metadata_json. We might reduce some postgres + metastore load by not fetching other columns, especially tags which can grow fairly large and often uses toast when it's not empty
this seems to have little (if at all) impact on actual performance. I suspect this would have a larger impact if split_metadata_json and tags where toasted, which is generally not the case unless you have a lot of tags.
It's unclear if we'd want to lower the toast limit, that would reduce the number of blocks to read when selecting, but add new reads when fetching (which is very often for split_metadata_json). Toasting tags could also have a very adverse effect on workload heavily relying on tags (for which we still do seq scans). We'd need to go through on #4658 before we can toast it.