Probably.jl icon indicating copy to clipboard operation
Probably.jl copied to clipboard

HyperLogLog example using threads

Open SaiCharanMarrivada opened this issue 6 months ago • 0 comments

Hi, Thank you for such wonderful library. I have one simple example using HyperLogLog with multiple threads.

using Probably
using Base.Threads

function example(n::Int) 
    lock_ = ReentrantLock()
    hll = HyperLogLog()
    nthreads = Threads.nthreads()
    Threads.@threads for id in 1:nthreads
        temp_hll = HyperLogLog()
        for i in id:nthreads:n
            push!(temp_hll, i)
        end
        lock(lock_) do
            union!(hll, temp_hll)
        end
    end
    return length(hll)
end

@time @show example(Int(1e10))

This code takes 14.34s with 1 thread, 3.73s with 16 threads on my intel ™ 15-1240P machine

JULIA_NUM_THREADS=16 julia hll_parallel.jl 
# output
example(Int(1.0e10)) = 10022247613
  3.731444 seconds (35.67 k allocations: 2.804 MiB, 24.87% compilation time)
JULIA_NUM_THREADS=1 julia hll_parallel.jl 
# output
example(Int(1.0e10)) = 10022247613
 14.344354 seconds (35.21 k allocations: 2.542 MiB, 0.30% compilation time)

Is it possible to add this example in documentation?

SaiCharanMarrivada avatar Aug 17 '24 23:08 SaiCharanMarrivada