How to limit memory consumption?

[Veritogen]

Hello! First of all, thank you very much for the awesome work! swifter is a big help to me and also very easy to use. I have a question concerning pandas.DataFrame.swifter.set_ray_compute(num_cpus=None, memory=None, **kwds).apply. I managed to render my machines unusable when just applying swifter.apply without the limits. Unfortunately its not clear to me, if the amount of memory specified in memory is per process/cpu, per partition or per worker? Is the amount of workers specified by the number of cpus? The example for point 8 seems to be the same as for point 7, unfortunately. Help/information would be much appreciated! Greetings from Germany Hannes

[juls858]

Yes I would love to have a feature like this as well for Dask. I could not find a way to set a cpu limit or memory limit. I tried using Dask.distributed to setup a local cluster, but swifter doesn't check for existing local cluster. Would it be possible to have an option to set number of CPU to be used or use a local cluster if one exists?