datatrove
datatrove copied to clipboard
Add Dataset type parameters
https://github.com/huggingface/datatrove/blob/734990228d305bdd38c2c3bab4e697d988c9ae68/src/datatrove/pipeline/readers/huggingface.py#L94
How about adding Dataset type parameter? To handle the case of the dataset that is processed at runtime and passed as a Dataset object. 😀