Mehul Jain
Mehul Jain
i'm getting the same error.
Hello everyone! Can someone please help here? Does anyone know what could've caused this?
I'm using python. Maybe because I'm reading files from HDFS and using Yarn as the resource manager? (it is a Yarn cluster)
thanks for quick response @jameslamb . Earlier, I did not think it is out-of-memory issue, I tried with 64GB per worker as well and I still saw the same issues....
total size of data is just 2GB so I think it can't be out-of-memory issue.
i meant raw data. I will try out with more memory but if 2GB of raw data is going to take that much memory then I probably should try different...
sure, let me try will Dask Arrays. Machines aren't removed during training. These machines are hadoop datanodes but they are not consuming as much memory.