self.tokenizer.save error due to tokenizers version incompatibility

[wyshi]

Bug description when tokenizers.__version__==0.5.2 the code works fine; but if tokenizers.__version__==0.8.0, it reports the following error

Output

Traceback (most recent call last):
  File "examples/train_model.py", line 15, in <module>
    TrainModel.main()
  File "/private/home/wyshi/ParlAI/parlai/scripts/script.py", line 79, in main
    return cls._run_args(None)
  File "/private/home/wyshi/ParlAI/parlai/scripts/script.py", line 66, in _run_args
    return script.run()
  File "/private/home/wyshi/ParlAI/parlai/scripts/train_model.py", line 761, in run
    return self.train_loop.train()
  File "/private/home/wyshi/ParlAI/parlai/scripts/train_model.py", line 713, in train
    self.save_model('.checkpoint')
  File "/private/home/wyshi/ParlAI/parlai/scripts/train_model.py", line 375, in save_model
    self.agent.save(fn)
  File "/private/home/wyshi/ParlAI/parlai/core/torch_agent.py", line 1795, in save
    self.dict.save(model_dict_path, sort=False)
  File "/private/home/wyshi/ParlAI/parlai/core/dict.py", line 628, in save
    self.bpe.save(os.path.dirname(filename), os.path.basename(filename))
  File "/private/home/wyshi/ParlAI/parlai/utils/bpe.py", line 804, in save
    self.tokenizer.save(dir_name, file_name)
  File "/private/home/wyshi/.conda/envs/conda_parlai/lib/python3.7/site-packages/tokenizers/implementations/base_tokenizer.py", line 332, in save
    return self._tokenizer.save(path, pretty)
TypeError

[stephenroller]

Thanks. For now I recommend downgrading, but we will have to fix this.

That error message tho :/