Multithreading issue

[jaehongpark-neuro]

Hi,

I am trying to apply multithreading approach to your codes It looks like your KDE codes are not compatible with multithreading. For some reasons, it even took more time than serial version when the length of observations is relatively big. the length of the grid is about 10000

here is the length of observations in 3d coordinates

for i in 1:12 println(length(cell_coor[i])); end

1349 1582 652 1915 2585 4256 2416 3632 2722 5782 5276 5345

single thread version function single_KDE_(cs::Vector{Vector{Float64}},cs_grid::Vector{Vector{Float64}}, dims::Vector{ContinuousDim},bws::Vector{Float64}) kde = KDEMulti(dims, bws, cs) projection = zeros(Float64,length(cs_grid))

    for i in 1:length(cs_grid)
        projection[i]= MultiKDE.pdf(kde,cs_grid[i])
    end
projection

end

for i in 1:12 @time single_KDE_(cell_coor[i],wbg,dims,bws); end

7.664427 seconds (91.29 M allocations: 3.383 GiB, 5.24% gc time, 16.62% compilation time) 7.908790 seconds (103.59 M allocations: 3.761 GiB, 4.39% gc time, 8.71% compilation time) 3.376405 seconds (43.29 M allocations: 1.582 GiB, 4.39% gc time, 11.66% compilation time) 9.388097 seconds (125.17 M allocations: 4.546 GiB, 3.92% gc time, 9.63% compilation time) 13.182955 seconds (168.77 M allocations: 6.113 GiB, 4.04% gc time, 11.63% compilation time) 19.032727 seconds (277.10 M allocations: 10.025 GiB, 4.48% gc time, 0.15% compilation time) 12.208769 seconds (157.81 M allocations: 5.708 GiB, 4.04% gc time, 11.19% compilation time) 18.432195 seconds (236.65 M allocations: 8.561 GiB, 3.51% gc time, 13.57% compilation time) 13.631460 seconds (177.65 M allocations: 6.433 GiB, 3.68% gc time, 11.73% compilation time) 25.260157 seconds (376.03 M allocations: 13.608 GiB, 3.95% gc time, 0.09% compilation time) 22.995075 seconds (343.23 M allocations: 12.423 GiB, 3.88% gc time, 0.10% compilation time) 23.216559 seconds (347.70 M allocations: 12.588 GiB, 3.81% gc time, 0.17% compilation time)

multi thread version function single_KDE_th(cs::Vector{Vector{Float64}},cs_grid::Vector{Vector{Float64}}, dims::Vector{ContinuousDim},bws::Vector{Float64}) kde = KDEMulti(dims, bws, cs) projection = zeros(Float64,length(cs_grid))

    @threads for i in 1:length(cs_grid)
        projection[i]= MultiKDE.pdf(kde,cs_grid[i])
    end
projection

end

for i in 1:12 @time single_KDE_th(cell_coor[i],wbg,dims,bws); end

5.042971 seconds (88.41 M allocations: 3.211 GiB, 87.86% gc time) 2.400278 seconds (103.51 M allocations: 3.757 GiB, 66.58% gc time) 0.299331 seconds (43.21 M allocations: 1.578 GiB) 3.085649 seconds (125.10 M allocations: 4.542 GiB, 72.25% gc time) 16.860719 seconds (168.69 M allocations: 6.109 GiB, 88.83% gc time) 54.170797 seconds (277.01 M allocations: 10.020 GiB, 93.90% gc time) 54.794265 seconds (157.73 M allocations: 5.704 GiB, 97.01% gc time) 36.269803 seconds (236.56 M allocations: 8.556 GiB, 93.03% gc time) 51.954300 seconds (177.57 M allocations: 6.429 GiB, 96.32% gc time) 29.745716 seconds (375.94 M allocations: 13.603 GiB, 85.52% gc time) 33.432218 seconds (343.14 M allocations: 12.418 GiB, 87.29% gc time) 34.831675 seconds (347.61 M allocations: 12.584 GiB, 87.29% gc time)

---

I am not familiar with multithreading and just getting started to learn, so I don't know what's going on here.

I use AMD 5950x (16 cores, so 32 threads) and tested the simple multithreading with mandelbrot set, and it works well.