quickwit icon indicating copy to clipboard operation
quickwit copied to clipboard

consider not retuning all columns from metastore when listing splits

Open trinity-1686a opened this issue 1 year ago • 1 comments

splits has 12 columns, while we filter based on most of these, we rarely need to fetch more than split_id index_uid and possibly split_metadata_json. We might reduce some postgres + metastore load by not fetching other columns, especially tags which can grow fairly large and often uses toast when it's not empty

trinity-1686a avatar Sep 12 '24 13:09 trinity-1686a

this seems to have little (if at all) impact on actual performance. I suspect this would have a larger impact if split_metadata_json and tags where toasted, which is generally not the case unless you have a lot of tags. It's unclear if we'd want to lower the toast limit, that would reduce the number of blocks to read when selecting, but add new reads when fetching (which is very often for split_metadata_json). Toasting tags could also have a very adverse effect on workload heavily relying on tags (for which we still do seq scans). We'd need to go through on #4658 before we can toast it.

trinity-1686a avatar Sep 19 '24 08:09 trinity-1686a