nanoGPT
nanoGPT copied to clipboard
batch and multiprocess file write
Now write takes ~40s on my machine (~40x improvement, ~500MB/s write). Uses similar logic to the first implementation but with batching, so offsets are also calculated per batch not per sample.