datatrove icon indicating copy to clipboard operation
datatrove copied to clipboard

Add Dataset type parameters

Open aiqwe opened this issue 7 months ago • 0 comments

https://github.com/huggingface/datatrove/blob/734990228d305bdd38c2c3bab4e697d988c9ae68/src/datatrove/pipeline/readers/huggingface.py#L94

How about adding Dataset type parameter? To handle the case of the dataset that is processed at runtime and passed as a Dataset object. 😀

aiqwe avatar Jul 17 '24 12:07 aiqwe