is there a vocabulary for xlnet

[cotitan]

I wonder if there is a vocabulary for xlnet, so that giving a sentence, I could generate input_ids according to this vocab, instead of getting it from prepro_utils.encode_ids()

[lukemelas]

prepro_utils.encode_ids() essentially just wraps SentencePiece, so this question is more appropriate for the SentencePiece repo: https://github.com/google/sentencepiece. The "vocab" is really in the spiece.model file.

Let me know if you have any more questions or if you have a specific use-case where you need a vocab file.

[cotitan]

I'm doing some research on text summarization where a vocabulary file is important. On decode stage, we also need to generate a sequence word by word, and each word is from a vocabulary.

[guddu0007]

prepro_utils.encode_ids() essentially just wraps SentencePiece, so this question is more appropriate for the SentencePiece repo: https://github.com/google/sentencepiece. The "vocab" is really in the spiece.model file.

Let me know if you have any more questions or if you have a specific use-case where you need a vocab file.

prepro_utils.encode_ids() essentially just wraps SentencePiece, so this question is more appropriate for the SentencePiece repo: https://github.com/google/sentencepiece. The "vocab" is really in the spiece.model file.

Let me know if you have any more questions or if you have a specific use-case where you need a vocab file.

I need vocab file while running run_squad.py i need it.