Karan Jhanjee

Hi, Can someone help me with code customization for input size 1920x1080 image sizes. Working on a passion project right now, this would be really helpful. I do not wish...

Very slow training on AMD CPU

6

Hi All, I've been trying to train a BPE tokenizer with the following options. The data is around 300+ GB of text. `spm.SentencePieceTrainer.Train('--input={} --model_prefix=tokener --unk_piece=[UNK] --pad_piece=[PAD] --bos_piece=[CLS] --eos_piece=[SEP] --user_defined_symbols=[PAD],[MASK],[END_GEN] --vocab_size=110000...