feat: update all tokenizers to latest versions

[niieani]

I've done some heavy optimization of gpt-tokenizer, and was curious about the speed of the latest version. I've upgraded all the tokenizers and devDependencies, and the result in the README.md.

I've also created some of my own benchmarks to confirm the results - it's now the fastest, and most memory efficient tokenizer, and it's pure JavaScript. Happy to say that it's even faster than WASM or node bindings to Rust. 😄

[socket-security[bot]]

New and removed dependencies detected. Learn more about Socket for GitHub ↗︎

Package	New capabilities	Transitives	Size	Publisher
npm/@dqbd/[email protected]	filesystem	0	23.5 MB	davidduong
npm/@types/[email protected]	None	+1	2.29 MB	types
npm/[email protected]	None	0	40.5 MB	niieani
npm/[email protected]	environment, filesystem, shell	0	4.03 kB	typicode
npm/[email protected]	None	+1	22.4 MB	davidduong
npm/[email protected]	Transitive: environment, filesystem, shell	+54	2.24 MB	okonet
npm/[email protected]	environment, filesystem, unsafe	0	7.7 MB	prettier-bot
npm/[email protected]	None	0	33.1 MB	ceifa
npm/[email protected]	filesystem	0	23.5 MB	davidduong
npm/[email protected]	None	0	62.9 kB	aslemammad
npm/[email protected]	Transitive: environment, filesystem, network, shell, unsafe	+97	9.95 MB	egoist, sxzz
npm/[email protected]	Transitive: filesystem	+3	680 kB	hirokiosame
npm/[email protected]	None	0	22.4 MB	typescript-bot

🚮 Removed packages: npm/@dqbd/[email protected], npm/@types/[email protected], npm/[email protected], npm/[email protected], npm/[email protected], npm/[email protected], npm/[email protected], npm/[email protected], npm/[email protected], npm/[email protected], npm/[email protected], npm/[email protected], npm/[email protected]

View full report↗︎