Smoothing window

[jscheid]

Thanks for this gem, we're using it for all our long-running (data migration) Rake tasks to get a better idea of progress.

Current Problem

The ETA and rate display take all past observations into account but there are many situations where looking only at recent observations would paint a more accurate picture. A few examples:

• A loop where the first iteration takes significantly longer than the rest ("cold start")
• A process that is paused and unpaused (perhaps using Ctrl+S/Ctrl+Q in the shell)
• A process that is temporarily slowed down by external activity, eg. higher database load

In all of these cases there is a temporary slow-down which doesn't necessarily affect later progress, and therefore would ideally affect ETA/rate calculation only temporarily.

Potential Solution or New Behavior

I thought that perhaps smoothing could be used to help with this but after spending some time looking into it, I don't think it can because it's applied to the running total of items.

This could be made to work with a window, something like the following:

• When progress.increment is called, add a timestamp to the tail of an array
• Delete all timestamps from the head of the array that are more than N seconds ago, N being the window size. Keep at least two entries.
• Divide the time range (newest timestamp minus oldest) by number of timestamps (minus one) to yield the current (windowed) rate

This will break down for situations where progress is incremented by multiple steps or set to an absolute value so perhaps this could be opt-in?

Mockups

N/A

[jfelchner]

@jscheid I'm absolutely open to this idea. However, the biggest caveat here (other than the other one you mentioned) is speed. ruby-progressbar must be, above all other things: light and fast. If we can determine a solution for this that is smart enough to handle arbitrary movements in progress, and is both light and fast, I'm interested in this.

[jfelchner]

Additionally: https://github.com/jfelchner/ruby-progressbar/issues/168 also is having an issue with arbitrary progress changes messing things up. Perhaps you two can work on a common solution?

[jfelchner]

@jscheid sorry for the (extremely long) delay on this. I added some code for you to get you started.

This code allows for the ability to add additional calculators to be used when the bar determines the running average of the amount of time each subsequent progress tick will take. This running average is then used to calculate every other estimated metric for the bar.

Right now it uses the standard smoothing calculator that we've been using for over a decade now. If you would like to take a stab at implementing a new calculator, I'd be open to looking at the PR and working with you to try to figure out a solution.

I'm open to additional parameters being passed to the calculator but I don't want to go overboard.

Let me know if there's anything else you need. I'm going to close this issue and feel free to open a PR and reference it.

Thanks!

[jfelchner]

Actually I went even further with this. Check out this PR. It should now be extremely extensible for someone to submit a PR to be able to include a different projector to estimate the remaining time.