FastChat icon indicating copy to clipboard operation
FastChat copied to clipboard

Published 20 hours ago • verified publisher icon lm-sys

Using nproc multithreading for mapping in tokenizer

Open lucasjinreal opened this issue 2 years ago • 0 comments

Currently the speed is extremly slow, is there a way to provide multi threading using all nproc num for mapping Just like datasets ?

lucasjinreal avatar May 31 '23 12:05 lucasjinreal