Ticking/Updating elapsed time

[jonzarecki]

• [x] I have marked all applicable categories:
  • [x] new feature request

I'm tqdm'ing with DB calls which take some time to compute, so I'm interested in the calls % progress as well as the total time that has passed.

I was wondering if it's possible to add a ticking "elapsed time" in some way to the tqdm bar, as well as I could see tqdm only updates when an iteration is finished. I think it would be very nice for all the people staring at tqdm bars for a living.

[casperdcl]

Do you mean miniters=1, mininterval=0?

[jonzarecki]

I also thought this should produce this behavior, however, at least for me, it doesn't.

import time
from tqdm import tqdm

for i in tqdm(range(10), miniters=1, mininterval=0, maxinterval=1):
    time.sleep(20)

This code for example only updates every 20s, and not continuously as I wished.

[jonzarecki]

Can I help with that ? Can you point me to the module that's not working as expected ?

[casperdcl]

I think the issue is with https://github.com/tqdm/tqdm/blob/master/tqdm/_monitor.py. The update logic is https://github.com/tqdm/tqdm/blob/cc53d86f86cef63ba8dbbfb8e9cb27d4f247fd20/tqdm/_monitor.py#L85-L92 which may not be triggered properly.

[jonzarecki]

Alright, I'll look into it 👍

[casperdcl]

Maybe related: #543, #320

[hagenw]

I think #717 is related as well. That one I found first when I was looking exactly for this issue.

[reox]

As I wrote in #717, one solution is to remove the if instance.miniters > 1. I do not quite understand why it is in there in the first place and why miniters is set to 1 then.

However, there is also another problem. tqdm.tqdm.monitor_interval is set by default to 10 seconds: https://github.com/tqdm/tqdm/blob/ee651fd8556d3c3aaa5d7aadb5c7af018d46d9bf/tqdm/std.py#L194 But unless this is not less than the value which is configured for the mininterval, the constant ticking will also not work as intended. Regardless, this can be adjusted by overwriting the class variable before using tqdm, i.e.:

import tqdm
tqdm.tqdm.monitor_interval = 1

I also find the docstring for miniters a little bit confusing: https://github.com/tqdm/tqdm/blob/ee651fd8556d3c3aaa5d7aadb5c7af018d46d9bf/tqdm/std.py#L841-L848

It says if it is zero and dynamic_miniters. But when looking at the code, dynamic_miniters is only used if miniters is set to None. That on the other hand means, if we do not set miniters, we enable dynamic_miniters by default, which leads to the problem that miniters will never be greater than 1 until the first dynamic_miniters kicks in. Probably it is a good idea to change the docstring to reflect the current behaviour:

miniters  : int or float, optional
            Minimum progress display update interval, in iterations.
            If `None`, will automatically adjust to equal
            `mininterval` (more CPU efficient, good for tight loops).
            If > 0, will skip display of specified number of iterations.
            Tweak this and `mininterval` to get very efficient loops.
            If your progress is erratic with both fast and slow iterations
            (network, skipping items, etc) you should set miniters=1.

So if you want the progressbar to tick the time when starting, there is probably a complete other logic required for the monitor thread. In my opinion you have three usecases:

1. you want the progressbar to update only on iterations
2. you want the progressbar to update only on time ticks
3. you want the progressbar to update on time ticks or iterations, depending on what comes first but never faster than some limit

IMO right now only 1) is correctly implemented. No. 2) depends on the monitor thread to tick correctly, which it cant because of the miniter blocking it. No 3) seems to work if and only if the iteration is fast, however if there comes a very slow iteration, the timer thread will only update the progressbar if miniters happens to be >1 at that instance. But then, as no more iterations come, it will not update until the next iteration. So it is also kind of broken.

I think a very easy (but ugly) solution would be to have some flag for tqdm which disables all iteration magic checks and calculations and simply ticks at a constant pace. The better solution is probably to think about how no 2) can be implemented in the current setup without breaking the other things...

[reox]

I'm not sure, but what if the and is exchanged for an or?

diff --git a/tqdm/_monitor.py b/tqdm/_monitor.py
index e1e2570..a8f3d28 100644
--- a/tqdm/_monitor.py
+++ b/tqdm/_monitor.py
@@ -82,7 +82,7 @@ class TMonitor(Thread):
                         return
                     # Only if mininterval > 1 (else iterations are just slow)
                     # and last refresh exceeded maxinterval
-                    if instance.miniters > 1 and \
+                    if instance.miniters > 1 or \
                             (cur_t - instance.last_print_t) >= \
                             instance.maxinterval:
                         # force bypassing miniters on next iteration

In that case the timer works correctly, however if an iteration and a maxinterval coincide at the right time instance, the timer needs two seconds before it ticks over (if maxinterval is set to 1)

Not sure what it breaks instead though :D

[casperdcl]

I'm not sure, but what if the and is exchanged for an or?

Hah, not likely an option.

You earlier comment is valid though - it's a bit of a mess with min/maxinterval, miniters, and minitor_interval. I think has been raised before in similar detail but I haven't found where (yet)...

[reox]

Hah, not likely an option.

that's understandable :)

I stepped a little bit through the code to understand the logic, but it needs some time... So that is all I can provide right now :/

[dreamflasher]

I can confirm this bug. What I observe is for slow-running tasks, that the bar is only updated when a new iteration is started, and not every X seconds.

Unfortunately, this also doesn't work:

import time

from tqdm import tqdm

tqdm.monitor_interval = 1

for i in tqdm(range(10), miniters=1, mininterval=0, maxinterval=1):
	time.sleep(10)

The resolution of the bar is in seconds. Why would a higher update frequency be desirable? I suggest to only and fully support use case (2) from @reox – with an update interval of 1 second for all bars. That would improve the usability and simplify the code base a lot. Having miniters, mininterval, maxinterval, monitor_interval looks like premature optimization?

[TheHardew]

So the only solution for now is probably to run async tasks and manually refresh the bar? And if I want to update the remaining time and the rate, I have to also calculate that myself and pass it as a custom bar_format?

[kwilinsi]

For those (like me) looking for a relatively simple workaround, you can refresh the bar from a separate thread. This can be done via a wrapper function acting as a drop-in replacement for tqdm.tqdm.

import random
from threading import Thread
import time
from tqdm import tqdm

def tqdm_wrapper(iterable, update_delay: float = 1, *args, **kwargs):
    t = tqdm(iterable, *args, **kwargs)

    def refresh():
        while not t.disable:
            time.sleep(update_delay)
            t.refresh()
    thread = Thread(target=refresh, daemon=True)
    thread.start()

    return t

And used as follows:

for _ in tqdm_wrapper(range(50)):
    time.sleep(random.uniform(1, 5))

[radar3301]

~~Awesome! tqdm now updates the seconds every second, but how can one solve the issue of tqdm not updating the iteration count until every multiple of 20 seconds?~~

import time
from threading import Thread
from tqdm import tqdm

def tqdm_wrapper(iterable, update_delay: float = 1, *args, **kwargs):
    t = tqdm(iterable, *args, **kwargs)

    def refresh():
        while not t.disable:
            time.sleep(update_delay)
            t.refresh()
    thread = Thread(target=refresh, daemon=True)
    thread.start()

    return t

for i in tqdm_wrapper(range(1, 2000)):
    if i < 1900:
        continue
    time.sleep(1)

Edit: nevermind, figured out I needed to add miniters=1

[hagenw]

Yes, in your example, you don't need the tqdm_wrapper:

for i in tqdm(range(1, 2000), miniters=1):
    if i < 1900:
        continue
    time.sleep(1)

works as well.