No progress update with @spawn use case?

[paulmelis]

This MWE is a slight variation on one of the examples from the readme. Instead of doing a sleep(3*rand()) it does some (useless) computation for a random period of time, mimicking an actual compute task. It also spawns double the number of tasks compared to the example.

What I observe with 1.5.3 and 1.6RC1 is that the progress bar does not update at all but only prints out at 100% when all tasks are done. It doesn't even show an initial Progress:.

using ProgressMeter
n = 20
p = Progress(n)
tasks = Vector{Task}(undef, n)
for i in 1:n
    tasks[i] = Threads.@spawn begin
        #sleep(3*rand())
        w = 3*rand()
        count = 0
        t0 = time()        
        while time() - t0 < w
            count += 1
        end
        
        ProgressMeter.next!(p)
    end
end
wait.(tasks)

julia> versioninfo()
Julia Version 1.6.0-rc1
Commit a58bdd9010 (2021-02-06 15:49 UTC)
Platform Info:
  OS: Linux (x86_64-pc-linux-gnu)
  CPU: Intel(R) Core(TM) i5-4460  CPU @ 3.20GHz
  WORD_SIZE: 64
  LIBM: libopenlibm
  LLVM: libLLVM-11.0.1 (ORCJIT, haswell)

(@v1.6) pkg> st ProgressMeter
      Status `~/.julia/environments/v1.6/Project.toml`
  [92933f4c] ProgressMeter v1.5.0

[IanButterworth]

Perhaps try adding a yield() after the next!?

[paulmelis]

When running with -t4 this only improves things a bit for some of the runs, showing an initial 5% step but then still jumping to 100% after that. Even when I increase the amount of "work" done in the tasks to be 10*rand() it shows the same behaviour.

[paulmelis]

Revisiting this issue and I still don't fully understand what's going wrong. With the MWE below I get really funky progress output when running with -t1:

using Random, ProgressMeter

function go(N)

    tasks = Task[]    
    progress = Progress(N)

    for i in 1:N        
        t = Threads.@spawn begin                    
            t0 = time()
            sleep(rand()*0.5)
            #println("$(i) starting")
            sleep(5.0)
            t1 = time()
            #println("$(i) ran from $(t0) to $(t1) $(t1-t0)")
            next!(progress)
            yield() 
        end
        push!(tasks, t)
        
    end

    wait.(tasks)

end

Random.seed!(123456)
t0 = time()
go(5)
println("total $(time()-t0)")

Seems there's at least cursor-back-to-start and/or newline missing. If I uncomment both println statements it at least makes some sense:

When I change the number of tasks to, say, 50 and run with -t4 the output is also broken and I would have expected more progress updates:

A second example with 200 tasks, in the range 0.02 to 2 seconds, when running with -t4 I see no progress updates at all, except the final 100% one, on a total run time of 50+ seconds (using yield() makes no difference):

using Random, ProgressMeter

function spin(t)
    c = 0
    t0 = time()
    while time() - t0 < t
        c += 1
    end
    return c
end

function go(N)

    tasks = Task[]    
    progress = Progress(N)

    for i in 1:N        
        t = Threads.@spawn begin                    
            t0 = time()
            #println("$(i) starting")
            spin(i*0.01)
            t1 = time()
            #println("$(i) ran from $(t0) to $(t1) $(t1-t0)")
            next!(progress)
            #yield()
        end
        push!(tasks, t)
        
    end

    wait.(tasks)

end

Random.seed!(123456)
t0 = time()
go(200)
println("total $(time()-t0)")

julia> versioninfo()
Julia Version 1.7.0
Commit 3bf9d17731* (2021-11-30 12:12 UTC)
Platform Info:
  OS: Linux (x86_64-pc-linux-gnu)
  CPU: Intel(R) Core(TM) i5-4460  CPU @ 3.20GHz
  WORD_SIZE: 64
  LIBM: libopenlibm
  LLVM: libLLVM-13.0.0 (ORCJIT, haswell)

(sahara-julia) pkg> st
      Status `~/concepts/sahara-julia/Project.toml`
...
  [92933f4c] ProgressMeter v1.7.1
...

[miguelbiron]

Just run into this issue and what worked for my use case---which does not require storing the tasks in a vector---simply fetching the task makes it work with any nthreads()>=1

using ProgressMeter
n = 20
p = Progress(n)
for i in 1:n
    tsk = Threads.@spawn begin
        w = 3*rand()
        count = 0
        t0 = time()        
        while time() - t0 < w
            count += 1
        end
        ProgressMeter.next!(p)
    end
    fetch(tsk)
end

julia> versioninfo()
Julia Version 1.7.3
Commit 742b9abb4d (2022-05-06 12:58 UTC)
Platform Info:
  OS: Linux (x86_64-linux-gnu)
  CPU: Intel(R) Core(TM) i7-6820HQ CPU @ 2.70GHz
  WORD_SIZE: 64
  LIBM: libopenlibm
  LLVM: libLLVM-12.0.1 (ORCJIT, skylake)

[hsgg]

I see the same behavior if I just println() something instead of next!(). For example, use -t 3, fix w=2, then one would expect the output to come in batches of 3 every ~2 seconds. Instead I see the first output, then the rest all at once at the end.

So not a ProgressMeter bug, but more in the core of Julia?

simply fetching the task

That, unfortunately, serializes the computation.

julia> versioninfo()
Julia Version 1.7.3
Commit 742b9abb4d (2022-05-06 12:58 UTC)
Platform Info:
  OS: Linux (x86_64-pc-linux-gnu)
  CPU: 11th Gen Intel(R) Core(TM) i5-1135G7 @ 2.40GHz
  WORD_SIZE: 64
  LIBM: libopenlibm
  LLVM: libLLVM-12.0.1 (ORCJIT, tigerlake)