DeepLearningProject icon indicating copy to clipboard operation
DeepLearningProject copied to clipboard

Got IOError half way through learning

Open bobbleoxs opened this issue 7 years ago • 0 comments

When I got to the last session on model textual, the model went through 5 epochs then threw this error:

IOError: [Errno 24] Too many open files

I went ahead trying to change $ulimit -n but realized the easiest way is to just change numb_workers to 4. It's arbitrary but someone suggested 4*num of GPU is a good approximation for num_workers.

It's not a specific issue per se but I think it may be beneficial for people to know this is one of the nuances in building ML pipeline which is not necessarily apparent.

bobbleoxs avatar Nov 30 '18 15:11 bobbleoxs