Li Tong
Add Examples of Common Preprocessing Steps with IterDataPipe (such as splitting a data set into two)
Is there a general method? I currently implement an IterDataPipe that splits the dataset by index: ```python @functional_datapipe("index_split") class IndexSpliterIterDataPipe(IterDataPipe): def __init__(self, source_dp, start_idx, end_idx) -> None: super().__init__() self.source_dp =...
Add Examples of Common Preprocessing Steps with IterDataPipe (such as splitting a data set into two)
In fact, it is not advisable to split the dataset by demux. Because the training set, validation set, and test set are often very large, it is easy to exceed...
Add Examples of Common Preprocessing Steps with IterDataPipe (such as splitting a data set into two)
@ejguan It does take some time to skip some elements, a little trick is to put the small dataset in the front, such as the validation set and test set,...
或者可以从文件导入也可以，太多复制不了。
And when processing other types of elements, it should be iterated. For example, when **output_hidden_states=True**, the tuple object will be returned. For example: ```python out = {} for k, element...
作者可以解答一下吗？当通过pip安装的时候应该如何自定义Dataset和Evaluator？