datatrove icon indicating copy to clipboard operation
datatrove copied to clipboard

Published 20 hours ago • verified publisher icon huggingface

Speedup json writer

Open its5Q opened this issue 9 months ago • 2 comments

This pull request replaces the json standard library with a much more faster orjson library, speeding up JsonlWriter by about 5x on my machine, and about 2x for JsonlReader, roughly matching the numbers given in the orjson's README.

its5Q avatar May 05 '24 11:05 its5Q