MappedArrays.jl icon indicating copy to clipboard operation
MappedArrays.jl copied to clipboard

Add mappedarrayreduce

Open jishnub opened this issue 5 years ago • 3 comments

A lazy version of mapreduce using MappedArrays to get around intermediate allocations (see julia/issues/38558). I'm not sure what to name the function, open to suggestions. The performance improvement is visible:

julia> x = randn(10240); y = similar(x);

julia> @btime mapreduce(==, +, $x, $y);
  22.145 μs (1 allocation: 10.19 KiB)

julia> @btime mappedarrayreduce(==, +, $x, $y);
  5.072 μs (0 allocations: 0 bytes)

jishnub avatar Nov 24 '20 07:11 jishnub

looks good, about the name, perhaps mapreducedarray?

Would be helpful to throw an explicit error for mappedarrayreduce(f, finv, op, A...) since there's a writable version mappedarray(f, finv, A...).

btw, we need to update docs and README.md, too.

johnnychen94 avatar Nov 24 '20 10:11 johnnychen94

If we can solve this by doing in https://github.com/JuliaLang/julia/issues/38558#issuecomment-732752166, I'd very much prefer it to adding a new export here. But if that's not really feasible, then sure, let's do this. But I'd like to hold off a bit and see if someone tackles Generator performance, or at least digs deep enough to understand why it's slower.

timholy avatar Nov 24 '20 13:11 timholy

I don't think "lazy" is accurate here; it just computes the reduction. Whether there is an intermediate array is not observable.

JeffBezanson avatar Nov 24 '20 18:11 JeffBezanson