Probably.jl
Probably.jl copied to clipboard
HyperLogLog example using threads
Hi, Thank you for such wonderful library. I have one simple example using HyperLogLog
with multiple threads.
using Probably
using Base.Threads
function example(n::Int)
lock_ = ReentrantLock()
hll = HyperLogLog()
nthreads = Threads.nthreads()
Threads.@threads for id in 1:nthreads
temp_hll = HyperLogLog()
for i in id:nthreads:n
push!(temp_hll, i)
end
lock(lock_) do
union!(hll, temp_hll)
end
end
return length(hll)
end
@time @show example(Int(1e10))
This code takes 14.34s with 1 thread, 3.73s with 16 threads on my intel ™ 15-1240P machine
JULIA_NUM_THREADS=16 julia hll_parallel.jl
# output
example(Int(1.0e10)) = 10022247613
3.731444 seconds (35.67 k allocations: 2.804 MiB, 24.87% compilation time)
JULIA_NUM_THREADS=1 julia hll_parallel.jl
# output
example(Int(1.0e10)) = 10022247613
14.344354 seconds (35.21 k allocations: 2.542 MiB, 0.30% compilation time)
Is it possible to add this example in documentation?