Could you share us the C/Cython code in tokenizer?

[xuanqing94]

I really appreciate your code, it helps me a lot to reproduce the squad results, very good job!

One thing I am wondering is that how you write the tokenizer part. Currently, you uploaded the binary as .so files, but could you also upload the Cython code to compile this? I am very curious about how you encode the sentence.

Thanks in advance!