Implement custom tokenizer trouble

[zhy605420954]

Hi I want to implement use textattack to attack my lstm model for detect vulnerability in python code. So I need to use custom tokenizer to tokenize python source code.Could you please tell me how to implement a custom tokenizer? Really thank you!

[jxmorris12]

Hi! Textattack relies on an abstraction we call model wrappers which take text and output the model predictions. So you need to write a custom model wrapper that calls your custom tokenizer. It should look very similar to the existing model wrappers.