Recoder icon indicating copy to clipboard operation
Recoder copied to clipboard

Published 20 hours ago • verified publisher icon pkuzqh

Which pkl is used for tokenization?

Open guoweijun137 opened this issue 1 year ago • 1 comments

Hello！I found your work to be exceptionally insightful and engaging. I noticed that there are three pkls in your project, namely char_ voc.pkl, code_ voc.pkl and nl_ voc.pkl, so which file is used for tokenization of code readers?

guoweijun137 avatar Jan 16 '24 08:01 guoweijun137

char_voc.pkl and nl_voc.pkl are used to tokenize code for code readers.

pkuzqh avatar Mar 07 '24 06:03 pkuzqh