tokenizr

tokenizr copied to clipboard

Typescript

1

I didn't make a direct fork so I guess it didn't show on here, but, what do you think of this? I saw you just posted a types file and...

kevinkhill

Numerous Tokens Performance

1

Performance on large files is atrocious. You can look at code here https://github.com/Cpecific/twwh2_ancillary/blob/master/node_modules/tokenizr/lib/tokenizr.js Mainly added `Tokenizr._pendingPos`. Switched from concatting and shifting arrays to simply iterating over them. In my case...

Cpecific

Hello: It seems the output blows up at the first error, even though other tokens may still be usable. I couldn't find this type of use case in the example...

mauriciomateus