[RFC] add support for multiple simultaneous progressbars

[MarcMush]

following my answer in #156 , this adds two types of progressbars that mimic Progress:

ParallelProgress is pretty much the example in the readme and works like that :

using Distributed
addprocs()
@everywhere using ProgressMeter
prog = ParallelProgress(10; desc="test ")
pmap(1:10) do i
    sleep(rand())
    next!(prog)
    return myid()
end

MultipleProgress spawns multiple progress bars using offset

using Distributed
addprocs(2)
@everywhere using ProgressMeter
p = MultipleProgress([Progress(10; desc="task $i ") for i in 1:5], Progress(50; desc="global "))
pmap(1:5) do i
    for _ in 1:10
        sleep(rand())
        next!(p[i])
    end
    sleep(0.01)
    myid()
end

global 100%|██████████████████████████████████████████████████████████| Time: 0:00:24
task 4 100%|██████████████████████████████████████████████████████████| Time: 0:00:05
task 5 100%|██████████████████████████████████████████████████████████| Time: 0:00:05

It is also possible to add progressbars from another worker

p = MultipleProgress(Progress(10, "tasks done "); count_finishes=true)
sleep(0.1)
pmap(1:10) do i
    L = rand(20:50)
    p[i] = Progress(L, desc=" task $i ")
    for _ in 1:L
        sleep(0.05)
        next!(p[i])
    end
end

the main progress can be a Progress, either counting all individual steps, or counting finished tasks. It can also be a ProgressUnknown

the other progresses can be Progress, UnknownProgress or ProgressTresh

feel free to try this PR with ]add ProgressMeter#5c21f5d and give feedback and suggestions Everything should be explained in ?MultipleProgress

closes #9, closes #96, closes #97, closes #125, closes #156, closes #253

[IanButterworth]

This is great! It's exactly what I was just looking for, and seems stable on a few trials across remote distributed workers.

One minor thought, I noticed that the color field of the added update!() method isn't used. It might be nice to be able to change that on the fly. I couldn't see an easy way to implement that with this approach. I've also in the past updated the progress bar description during runs by changing prog.desc directly. It might be nice to add that functionality, but that could definitely wait for a future PR.

[IanButterworth]

Master now has the option to update the desc from update!() via https://github.com/timholy/ProgressMeter.jl/pull/159. It would be good to think about whether it could be incorporated here. I did try a few attempts, but couldn't figure out a simple modification.

[MarcMush]

I added tests and fixed some bugs due to overshooting

@ianshmean it wouldn't be difficult to change it, by passing NamedTuples instead of Ints in the RemoteChannel

[MarcMush]

i'm working on it here https://github.com/MarcMush/ProgressMeter.jl/tree/parallel_update

[jtrakk]

This looks really nice. Does it work for tasks distributed over threads as well?

[MarcMush]

yes, it should work, you can even use it on a single core I didn't work on this much since I did this PR, but I can continue working on it if there is demand for it (or if someone wants to continue it, feel free to do it)

[jtrakk]

I think it would be really useful for parallel long-running tasks. For example, it would be a nice improvement in the UX of parallel algorithms in Turing.jl.

[stevengj]

It would be nicer if you could encapsulate any other progress bars, e.g. ProgressUnknown etcetera, in this way. That is, if you could just pass an array of progress meters:

p = MultipleProgress([Progress(...), ProgressUnknown(...), ProgressThresh(...)])

[bdeonovic]

Whats the status of this?

[MarcMush]

I did some work on this PR I implemented @IanButterworth's suggestion https://github.com/timholy/ProgressMeter.jl/pull/157#issuecomment-638330463

[MarcMush]

It would be nicer if you could encapsulate any other progress bars, e.g. ProgressUnknown etcetera, in this way. That is, if you could just pass an array of progress meters:

p = MultipleProgress([Progress(...), ProgressUnknown(...), ProgressThresh(...)])

I really like the idea, and I enabled it for ParallelProgress, but for MultipleProgress, there will be a problem for tasks that are not started immediatly, since the start time will report when the whole object was created There is several solutions to this, I don't know which one is better:

• just don't care, the timings will be wrong in some cases
• pass functions that will be executed at first call of the progressmeter eg p = MultipleProgress([()->Progress(...), ()->ProgressUnknown(...), ()->ProgressThresh(...)])
• reset the p.tinit at first call

note that right now, and with one these solutions (except the first one) the first step won't be included in the timing except if initialized with update!(p,0) for example

[codecov[bot]]

Codecov Report

Attention: Patch coverage is 98.19277% with 3 lines in your changes missing coverage. Please review.

Project coverage is 97.21%. Comparing base (807496a) to head (5c21f5d). Report is 15 commits behind head on master.

Files	Patch %	Lines
src/parallel_progress.jl	98.19%	3 Missing :warning:

Additional details and impacted files

@@            Coverage Diff             @@
##           master     #157      +/-   ##
==========================================
+ Coverage   93.48%   97.21%   +3.72%     
==========================================
  Files           1        2       +1     
  Lines         399      717     +318     
==========================================
+ Hits          373      697     +324     
+ Misses         26       20       -6     

:umbrella: View full report in Codecov by Sentry.
:loudspeaker: Have feedback on the report? Share it here.

[stevengj]

but for MultipleProgress, there will be a problem for tasks that are not started immediately, since the start time will report when the whole object was created

How about have a Channel object that you can put! progress bars to asynchronously? That way you don't need to know in advance how many progress bars you need (e.g. if you are recursively processing a directory tree).

[jtrakk]

It would be cool to have a full tree of hierarchical progress bars for hierarchical task trees, matching up with the move toward structured concurrency in https://github.com/JuliaLang/julia/issues/33248.

[MarcMush]

again, a lot of changes this includes #215, which is of course necessary for this

Now, there are 2 main ways to use a MultipleProgress, with a few options

MultipleProgress([Progress(10), ProgressUnknown(), ProgressTresh(0.1)...], [mainprogressbar])

p = MultipleProgress(ProgressUnknown())
addprogress!(p[1], Progress, 10, desc="task 1")

I tried to detail the different possibilities in the docstring and in the tests

[HiramTheHero]

I apologize if I am resurrecting this thread inappropriately. What is the current status of this feature?

[MarcMush]

this is basically working but there are some decisions to make regarding syntax and functionnalities

[MarcMush]

This PR is about to celebrate its third anniversary, and I would love to merge this one day. Here is an attempt to list what is currently blocking to do so:

• [ ] are we satisfied with the current addprogress!(mp[i], Progress, args...)? This can be confusing and the index i needs to be known and managed correctly. See discussion https://github.com/timholy/ProgressMeter.jl/pull/157#pullrequestreview-758944656
• [ ] mp[0] is currently the main progressmeter, and mp[1:end] all the other ones. It is not an AbstractArray, no firstindex. Should this be changed?
• [ ] is everything ok with the approach? in particular the Channels
• [ ] documentation/readme

I encourage everyone interested in having multiple progressbars to try this PR with ]add ProgressMeter#3156d04 and give feedback and suggestions. Everything should be explained in ?MultipleProgress.

[timholy]

Sorry I've not come around to reviewing this impressive PR. My main thought is that if progress bars might be added or removed at various times, integer-indexing seems like a bad idea. I'd rather use a Dict-style of indexing, where each progress bar gets a unique key (and trying to create a new progress bar using an existing key throws an error). Keys should live for the entire lifetime of the main progress bar, even if they've finished.

[MarcMush]

I've changed to a Dict for storing the progressbars and throws an error when trying to re-add a progess with the same id. It is possible to use Ints or any other type for indexing. The main progress can be specified with the kwarg main and recalled with p.main

[Moelf]

this seems actually ready? gentle bump for a nod from main devs and we can do documentation etc. and ship it?

[MarcMush]

have you tested it? I'd appreciate other people trying it and giving feedback, see if it is actually what is wanted you can use it with ]add ProgressMeter#3156d04

[Moelf]

I read the examples here and I'm exactly using Distributed pmap so I think yes?

[MarcMush]

@distributed pmap() was already implemented before

this PR adds next!() accross workers with ParallelProgress, as well as multiple progressbars at once with MultipleProgress

[Moelf]

@distributed pmap() was already implemented before

I have multiple sub tasks and also pmap() is not good enough because workers are unstable

[MarcMush]

great, you can use this PR with ]add ProgressMeter#3156d04 and see if it works as expected for your usecase

[MarcMush]

I changed the syntax for adding new progressbars, now it simply works with setindex!:

mp = MultipleProgress()
mp[1] = Progress(10)
next!(p[1])