Matthew Rocklin

> we would likely first hit other known scaling limitations due to the number of tasks Which limitation did you run into, I'm curious? (hearing about these again is useful...

If it's a good idea then I don't mind anyone doing it. I don't think we need to do anything in response to this particular CVE though (except complain about...

I've pinged @ethansilvas by e-mail. Tom and Jacob, you're cc'ed. (@aftersnows didn't have a listed e-mail)

Thanks @adam-nygate ! That's great to hear, both on our specific case, and on reviewing current processes.

Hrm, interesting. I set `export MALLOC_ARENA_MAX=1`, then start a jupyter notebook and work from there. Sometimes I get no leak, sometimes I get a much bigger leak at 1.4GB.

cc @stefanv and @mattip

Also, as suggested by @stuartarchibald ``` mrocklin@carbon:~/workspace/play$ python memory-sharding.py 52.20 MB [########################################] | 100% Completed | 7.4s 568.71 MB mrocklin@carbon:~/workspace/play$ export MALLOC_MMAP_THRESHOLD_=0 mrocklin@carbon:~/workspace/play$ python memory-sharding.py 142.88 MB [########################################] | 100%...

@pitrou , long term what do you think should happen here? Can this issue be globally resolved through changes in CPython? Should Numpy change the way it calls `malloc`? Or...

Also, for comparison with `MALLOC_ARENA_MAX=1` ``` mrocklin@carbon:~/workspace/play$ python memory-sharding.py 52.40 MB [########################################] | 100% Completed | 6.9s [########################################] | 100% Completed | 0.1s 760.98 MB mrocklin@carbon:~/workspace/play$ export MALLOC_ARENA_MAX=1 mrocklin@carbon:~/workspace/play$ python...

Yes, this seems nicer. ``` mrocklin@carbon:~/workspace/play$ python memory-sharding.py 52.22 MB [########################################] | 100% Completed | 6.5s [########################################] | 100% Completed | 0.1s 697.00 MB mrocklin@carbon:~/workspace/play$ export MALLOC_MMAP_THRESHOLD_=16384 mrocklin@carbon:~/workspace/play$ python memory-sharding.py...