Xiaoyang Chen
Results
1
issues of
Xiaoyang Chen
According to the document [MaxTokenBucketizer](https://pytorch.org/data/main/generated/torchdata.datapipes.iter.MaxTokenBucketizer.html#torchdata.datapipes.iter.MaxTokenBucketizer) buffer_size – This restricts how many **tokens** are taken from prior DataPipe to bucketize However, in the code, [bucketbatcher.py#L277](https://github.com/pytorch/data/blob/84587ff57575fd47fcae61635a3f4ffc1e639941/torchdata/datapipes/iter/transform/bucketbatcher.py#L277) The unit of buffer_size is **sample**...
documentation