ludwig icon indicating copy to clipboard operation
ludwig copied to clipboard

Torchtext undefined module when using gpt2bpe tokenizer

Open jhudsy opened this issue 1 year ago • 2 comments

When trying to use the gpt2bpe tokenizer, I get an error that "module torchtext has no attribute util".

Fixed by adding

import torchtext.utils
import torchtext.transforms

to ludwig/utils/tokenizers.py

jhudsy avatar Jun 06 '24 18:06 jhudsy

@jhudsy Would you like to create a pull request with your fix? Thanks a lot.

alexsherstinsky avatar Jul 26 '24 21:07 alexsherstinsky

Hi @jhudsy could you try again with torchtext 0.17.2?

mhabedank avatar Oct 23 '24 21:10 mhabedank