ludwig icon indicating copy to clipboard operation
ludwig copied to clipboard
verified publisher icon ludwig-ai

Refactor GPT2BPETokenizer

Open mhabedank opened this issue 1 year ago • 2 comments

The GPT2BPETokenizer is using torchtext. We want to remove torchtext as a dependency so this Tokenizer has to be refactored not using it.

mhabedank avatar Oct 21 '24 20:10 mhabedank

@mhabedank is this issue still open?

Satarupa22-SD avatar Aug 28 '25 09:08 Satarupa22-SD

Hi @Satarupa22-SD it's still open. Yet no one is currently working on it, I fear.

mhabedank avatar Sep 04 '25 06:09 mhabedank