Shared memory issues with parallelization

[shubhamagarwal92]

Hi @kdexd

I am running into all kinds of shared memory errors after this commit 9c1ee36b85c2c63d554471cac2825cf0b9cf2efd

https://github.com/pytorch/pytorch/issues/8976 https://github.com/pytorch/pytorch/issues/973

I guess this parallelization is not stable; sometimes it run while sometimes it breaks (even though after trying possible solutions) such as:

torch.multiprocessing.set_sharing_strategy('file_system')

# https://github.com/pytorch/pytorch/issues/973
import resource
rlimit = resource.getrlimit(resource.RLIMIT_NOFILE)
resource.setrlimit(resource.RLIMIT_NOFILE, (2048*4, rlimit[1]))

Is there a leak somewhere? Might be best to have a look.

[kdexd]

Thanks for reporting this! I will try to reproduce this on my end and see where it breaks.

[shubhamagarwal92]

My hunch was that if we use parallelization in pytorch's data loader and still do multiprocess tokenization in one process, it is giving those errors. Basically, it tries to do in tokenization in (cpu_workers * cpu_workers) processes (?) and thus eating up shared memory (?)

I have removed this multiprocess tokenization and running some experiments. Will let you know how it goes. Your suggestion is also appreciated.

[kdexd]

My hunch was that if we use parallelization in pytorch's data loader and still do multiprocess tokenization in one process, it is giving those errors. Basically, it tries to do in tokenization in (cpu_workers * cpu_workers) processes (?) and thus eating up shared memory (?)

Both of these happen at different times. All the tokenization happens in the reader even before the training starts (when the dataloader makes batches).

I am unable to reproduce this so far. Could you check if it works well with 1 worker?

[shubhamagarwal92]

Yeah I did try with 1 worker. Had the same errors. (Cant use 0 because this requires at least one worker :D )

Have removed multiprocess tokenization in my code and it works fine.

Just to let you know it doesn't happen at starting iterations or epochs. I guess it was after 3-5 epochs.

[lucmos]

I think I'm hitting this.

In my setup I'm doing independent runs in parallel threads (not processes, since I'm using LevelDB and it does not support multiprocessing). Sometimes it breaks with the error:

RuntimeError: received 0 items of ancdata

Even though I'm using the workaround suggested here: https://github.com/pytorch/pytorch/issues/973#issuecomment-346405667