haystack icon indicating copy to clipboard operation
haystack copied to clipboard
verified publisher icon deepset-ai

Support dataset streaming

Open Koenlaermans opened this issue 2 years ago • 0 comments

Is your feature request related to a problem? Please describe. When I want to train a reader, my dataset uses 40Gb RAM. I only have 12Gb available. Streaming (loading only necessary part of dataset into memory for each batch) would fix this.

Koenlaermans avatar May 15 '23 13:05 Koenlaermans