Yunhu Ye

################################################ if(int(correctness[j]) == 0): label_index = problem_id else: label_index = problem_id + num_skills ############################################## It means coding a single skill separately by its correctness?

Could you please provide some details about tokenizer between Megatron-lm and HF tokenizer?

2

1. There are some different about megatron-lm tokenizer and HF tokenizer. ``` python llama/tools/preprocess_data.py \ --input /mnt/workspace/{}.json \ --output-prefix \ --vocab-file **gpt2-vocab.json** \ --dataset-impl mmap \ --tokenizer-type **GPT2BPETokenizer** \ --merge-file...