quickwit icon indicating copy to clipboard operation
quickwit copied to clipboard

Fix the ingest rate displayed in the CLI.

Open fulmicoton opened this issue 1 year ago • 4 comments

The CLI was showing the average rate since the CLI was started.

This PR changes it to an estimation over the last 2 seconds.

fulmicoton avatar Mar 06 '24 04:03 fulmicoton

I tried it and it seems somehow broken. After a while, I get this:

⠋ [00:20:00] 20.82 GiB/20.82 GiB (91.4 MiB/s)

Ingestion rate should be in the order of 17MiB/s, but the number I got never seemed to go below 40MiB/s

trinity-1686a avatar Mar 06 '24 13:03 trinity-1686a

oh really? let me check! Thanks for testing it!

fulmicoton avatar Mar 06 '24 13:03 fulmicoton

if that helps, it seems to only take a few discrete values, a few of which are 51.4, 45.7, 40, 62.8, 57.1, 74.2, that seems to be increments of 5.7 from each other

trinity-1686a avatar Mar 06 '24 13:03 trinity-1686a

@trinity-1686a There was a bug in the rate estimator indeed. I added @guilload to the review.

fulmicoton avatar Mar 08 '24 07:03 fulmicoton

the estimator always seems to give a multiple of 5.71 for this workload. This is because it essentially count batches of (quasi-)fixed size, and there aren't that many batches processed over a 2s window. In my case, there are almost always 3 batches processed, so I get 17.1MiB most of the time, and 11.4 or 22.8 when 2 or 4 batches got processed. Maybe for that estimator, it would be better to count the time it took to send the last X batches, rather than to count how many batches were sent in X seconds.

trinity-1686a avatar Mar 22 '24 11:03 trinity-1686a

I think there are different ways to improve the estimation, but I'll merge this for the bugfix. @trinity-1686a feel free to revisit it.

fulmicoton avatar May 10 '24 02:05 fulmicoton