quickwit
quickwit copied to clipboard
Fix the ingest rate displayed in the CLI.
The CLI was showing the average rate since the CLI was started.
This PR changes it to an estimation over the last 2 seconds.
I tried it and it seems somehow broken. After a while, I get this:
⠋ [00:20:00] 20.82 GiB/20.82 GiB (91.4 MiB/s)
Ingestion rate should be in the order of 17MiB/s, but the number I got never seemed to go below 40MiB/s
oh really? let me check! Thanks for testing it!
if that helps, it seems to only take a few discrete values, a few of which are 51.4, 45.7, 40, 62.8, 57.1, 74.2, that seems to be increments of 5.7 from each other
@trinity-1686a There was a bug in the rate estimator indeed. I added @guilload to the review.
the estimator always seems to give a multiple of 5.71 for this workload. This is because it essentially count batches of (quasi-)fixed size, and there aren't that many batches processed over a 2s window. In my case, there are almost always 3 batches processed, so I get 17.1MiB most of the time, and 11.4 or 22.8 when 2 or 4 batches got processed. Maybe for that estimator, it would be better to count the time it took to send the last X batches, rather than to count how many batches were sent in X seconds.
I think there are different ways to improve the estimation, but I'll merge this for the bugfix. @trinity-1686a feel free to revisit it.