Cdk29
Cdk29
Hi again, I set up pages on github, but the problem still persist. Cheers.
I am trying without the $to_fp16() to see if the error came from there.
Weird. You don't have a GPU ? I tried without the $to_fp16() but I just got a classic out of memory error. A first glance it does seems to came...
@henry090 : I wonder if it is not a more general memory leak issue as the GitHub issue I linked in the first post suggest. My technical background is light...
Same without fp16. Memory error but only after few epoch (!) and I got the same traceback/error if I look into the log. Looks like a memory leak. I will...
Yes. To me it happen only after some epochs, whether it is on my local computer or kaggle.
Yes. I just got suspicious because of the only report saying that it was because the guy was making everything into a function. Actually it might have triggered the leak...
@henry090 Hi! Yes and no. The issue cannot be solved per se directly. But I managed to train such a big network by splitting it across a lot of notebooks...
Sorry I did not had the opportunity to train a big model for the last competition on kaggle, next time if I am not struggling too much with other aspect...
Session info : > > R version 4.0.2 (2020-06-22) > Platform: x86_64-pc-linux-gnu (64-bit) > Running under: Ubuntu 20.10 > > Matrix products: default > BLAS: /usr/lib/x86_64-linux-gnu/blas/libblas.so.3.9.0 > LAPACK: /usr/lib/x86_64-linux-gnu/lapack/liblapack.so.3.9.0 >...