julia icon indicating copy to clipboard operation
julia copied to clipboard

speed-up `randperm` by using our current `rand(1:n)`

Open rfourquet opened this issue 2 years ago • 5 comments

And similarly for randcycle and shuffle.

We had a custom version of range generation for randperm, which was based on the ideas of our previous default range sampler SamplerRangeFast (generate k-bits integers using masking and reject out-of-range ones) and took advantage of the fact that randperm needs to generate rand(1:i) for i = 2:n.

But our current range sampler ("Nearly Division Less") is usually better than this hack, and makes these functions more readable. Typically, for array lengths < 2^20, the new version is faster, but gets slightly slower beyond 2^22.

Here are some speedups: randperm-ndl-speedup

The slow down for big arrays seems fine to me, but I will see if I can find an easy workaround.

Fix #57771.

rfourquet avatar Jul 11 '23 16:07 rfourquet

Revisiting this: I lost the original benchmark code, so I don't remember if it was using an explicit rng, or the implicit one. Here is a more comprehensive graph, which shows the same tendencies:

new-shuffle

However, if #58089 would be merged, the changes here would seem to always be an improvement:

new-shuffle-ndl

rfourquet avatar Apr 12 '25 14:04 rfourquet

Does this close https://github.com/JuliaLang/julia/issues/57771 ?

jakobnissen avatar Apr 12 '25 16:04 jakobnissen

Does this close https://github.com/JuliaLang/julia/issues/57771 ?

Yes.

rfourquet avatar Apr 13 '25 10:04 rfourquet

The slow down for big arrays seems fine to me

me as well, given that the larger the shuffled array gets the more likely that the user will want to do it with a distributed algorithm anyway, like the hypothetical https://github.com/JuliaFolds2/OhMyThreads.jl/issues/139 (Rust impl here https://github.com/manpen/rip_shuffle)

adienes avatar Apr 14 '25 14:04 adienes

Nice reference! But also, with #58089, I believe there won't actually be slowdowns anyway.

rfourquet avatar Apr 14 '25 17:04 rfourquet

since https://github.com/JuliaLang/julia/pull/58089 is merged, is this basically ready besides the doctests?

adienes avatar Aug 30 '25 22:08 adienes

The test failure looks unrelated, if that's the case I will merge in a couple of days.

rfourquet avatar Oct 25 '25 21:10 rfourquet