SmartEmbed icon indicating copy to clipboard operation
SmartEmbed copied to clipboard

Published 20 hours ago • verified publisher icon beyondacm

Function level tokenizer

Open minhhn2910 opened this issue 2 years ago • 0 comments

Hi, great repo. Could you share the Normalizer and Java Parser code for function_level to regenerate token list similar to your function_normalized_tokens and function_tokens in the original processed dataset ?

minhhn2910 avatar Jan 12 '23 05:01 minhhn2910