Pretrained Models for generator no tokenizer files

[ghosthamlet]

Thanks for open source code and model for paper 'Acquiring and Modelling Abstract Commonsense Knowledge via Conceptualization', the paper is a wonderful read. There is just model and config file in download folder, where to download the tokenizer files?

[mutiann]

Ah, it is simply the standard GPT2 tokenizer on Huggingface transformers.

[ghosthamlet]

But you have added special tokens: https://github.com/HKUST-KnowComp/atomic-conceptualization/blob/main/generator/models/comet_atomic2020_gpt2/comet_gpt2.py#L190 , if you are convenient to upload the tokenizer files, it will be more convenient to use it.