textgenrnn icon indicating copy to clipboard operation
textgenrnn copied to clipboard

Published 20 hours ago • verified publisher icon minimaxir

Training With A Big File

Open nikhilanayak opened this issue 4 years ago • 1 comments

I was wondering what's the biggest file textgenrnn can support well. I have a huge file and was wondering whether it would be worth it to just split it up because I keep running into the same memory error when I train with 1 single file.

nikhilanayak avatar Apr 18 '20 17:04 nikhilanayak

That might have to be the solution; unfortunately all texts have to be loaded into memory first.

You may have to use a server GPU w/ more VRAM.

minimaxir avatar Apr 28 '20 03:04 minimaxir