datatrove icon indicating copy to clipboard operation
datatrove copied to clipboard

Published 20 hours ago • verified publisher icon huggingface

Add shuffle option on huggingface reader

Open justHungryMan opened this issue 8 months ago • 1 comments

Added shuffle option on huggingface reader and also test code for shuffle.

Before merge this commit, plz check the fixed seed value and also buffer size

justHungryMan avatar Jun 18 '24 01:06 justHungryMan