datalust
Show progress of long running queries
I ran the pre-built query for storage space by event and running it took a few minutes to run. Please add some message like "scanning range x-y" or better yet some progress bar. I had no idea how long this run would take and just waiting to see the results without knowing when they are expected is wrong UX
Hi @DanAvni
Thank you for the suggestion.
Because filtering and querying happens in parallel it is not linear. Seq is searching the whole log range at once. How quickly the log is searched depends on indexes and the composition of those indexes, as well as IO and CPU capacity, which are not constant. We could perhaps show progress based on a worst case scenario (if your server tends to scan about 5GB/s and you've selected a date range with 50GB then your query should complete in < 10s) however this will still be affected by other queries running concurrently and also won't be very accurate.
Is there a database that does something similar to what you have in mind that we could look at for ideas?
The most high level thing I can think of is Seq counting before starting that it needs to scan 150 span files and showing a progress on how many files were scanned. Better than nothing. A finer grained idea would be to know it needs to scan the 150 files and they take 10GB in total and to count based on file sizes scanned (because files are not of the same size this might be better)