FileTrees.jl icon indicating copy to clipboard operation
FileTrees.jl copied to clipboard

Use Dagger's eager API

Open jpsamaroo opened this issue 4 years ago • 1 comments

After https://github.com/JuliaParallel/Dagger.jl/pull/223 gets merged, Dagger's eager API (Dagger.@spawn) should be suitable for use in packages. I would recommend we use it for non-lazy computations in FileTrees so that we can get the benefits of Dagger 100% of the time.

jpsamaroo avatar Jul 04 '21 02:07 jpsamaroo

I was thinking on how to add this in the best way. I assume we want it to be opt-in to begin with?

Would it suffice with some convenience which basically does this:

julia> tt = maketree("root" => ["next" => [(name=string(x), value=1:10) for x in 'a':'k']]);

julia> function myvcat(x, y)
           sleep(1)
           vcat(x,y)
       end
myvcat (generic function with 1 method)

julia> @time fetch(reducevalues((v1,v2) -> Dagger.@spawn(myvcat(v1, v2)), tt));
  4.159940 seconds (110.59 k allocations: 5.883 MiB, 2.32% compilation time)

For example, letting exec also fetch EagerThunks and wrapping @spawn in some utility function, e.g. parallel(f) = (args...) -> Dagger.@spawn(f(args...))?

DrChainsaw avatar May 22 '22 22:05 DrChainsaw