The error of test the Dataloader

[sysusicily]

when I run "python ../../llmfoundry/data/text_data.py --local_path ./my-copy-c4 --split val_small", I get the error

(mpt-env) root@autodl-container-645911b4fa-161dd8b6:~/autodl-tmp/llm-foundry/scripts/train# python ../../llmfoundry/data/text_data.py --local_path ./my-copy-c4 --split val_small

Traceback (most recent call last): File "../../llmfoundry/data/text_data.py", line 308, in from llmfoundry.utils.builders import build_tokenizer File "/root/autodl-tmp/llm-foundry/llmfoundry/init.py", line 8, in from llmfoundry.data import (ConcatTokensDataset, File "/root/autodl-tmp/llm-foundry/llmfoundry/data/init.py", line 4, in from llmfoundry.data.datasets import ConcatTokensDataset, NoConcatDataset File "/root/autodl-tmp/llm-foundry/llmfoundry/data/datasets.py", line 9, in import datasets as hf_datasets File "/root/autodl-tmp/llm-foundry/llmfoundry/data/datasets.py", line 15, in class NoConcatDataset(IterableDataset): File "/root/autodl-tmp/llm-foundry/llmfoundry/data/datasets.py", line 22, in NoConcatDataset hf_datasets.Dataset]): AttributeError: partially initialized module 'datasets' has no attribute 'Dataset' (most likely due to a circular import)

[tginart]

I ran into this as well. You can fix it by just getting rid of the typing hints. You'll probably run into more bugs after that though. Let me know how that goes.

[codestar12]

I'll take a look at this. Thanks for bringing it to our attention.

[hanlint]

Hello, we pushed a fix in #175 . Closing for now, please re-open if you run into the same issue again!