nanoGPT icon indicating copy to clipboard operation
nanoGPT copied to clipboard

batch and multiprocess file write

Open LaihoE opened this issue 1 year ago • 0 comments

Now write takes ~40s on my machine (~40x improvement, ~500MB/s write). Uses similar logic to the first implementation but with batching, so offsets are also calculated per batch not per sample.

LaihoE avatar Jan 01 '23 15:01 LaihoE