llm-foundry icon indicating copy to clipboard operation
llm-foundry copied to clipboard

Published 20 hours ago • verified publisher icon mosaicml

Handle large files during text to mds conversion

Open irenedea opened this issue 1 year ago • 0 comments

Previously, large files are read entirely at once via file.read(). This reads the file and tokenizes in chunks.

irenedea avatar Jan 08 '24 03:01 irenedea