Julian Samaroo

You can give https://github.com/JuliaData/MemPool.jl/pull/60 a try, which is my new WIP approach to swap-to-disk (just set the env. var. `JULIA_MEMPOOL_EXPERIMENTAL_FANCY_ALLOCATOR=1` to enable it). I will warn you that it's not...

Do you have a reproducer for this one, just to help me debug it reliably?

@StevenWhitaker when this occurs, how much does `Base.format_bytes(MemPool.GLOBAL_DEVICE[].device_size)` report, and what amount of memory is reported by the OS as being used by Julia (out of the total system RAM)?...

Can you try throwing in some `@everywhere GC.gc()` calls every iteration and see if that delays or eliminates the OOM situation? If so, then it means that we should start...

@StevenWhitaker can you try out https://github.com/JuliaData/MemPool.jl/pull/75 and see if it delays the OOM further or eliminates it? You'll probably want to fiddle with the `MemPool.MEM_RESERVED[]` value to see which avoids...

The error shown should be fixed by https://github.com/JuliaParallel/Dagger.jl/issues/450#issuecomment-1812894331, you need to be using Julia 1.11 with my Distributed PR at all times for reliable distributed computing. And yes, it doesn't...

Happy Holidays! > Is my assumption wrong, i.e., even with one thread we need that Distributed.jl fix? Probably yes, as it may still be a race with multiple async tasks....

Odd, can you provide a stacktrace of the segfault?

Yeah I'm dealing with these errors too now that I've opened the PR. I'll get those fixed and then let you know when you can try it.

Huh, I hadn't thought of that, but now that you mention it, every Dagger task does rely on `fetch`ing the result of a Julia `Task` to get the Dagger task's...