transformers-js
transformers-js copied to clipboard
Use WASM tokenizers
To simply your implementation, might be better to use the tokenizers through WASM https://blog.mithrilsecurity.io/porting-tokenizers-to-wasm/