tokenize.jl: don‘t hardcode unicode ops range

[pthariensflame]

There are very nice (BEGIN|END)_UNICODE_OPS kinds; they should be used!

[codecov[bot]]

Codecov Report

All modified and coverable lines are covered by tests :white_check_mark:

Comparison is base (a6f2d15) 94.61% compared to head (7ace8d7) 96.54%.

Additional details and impacted files

@@            Coverage Diff             @@
##             main     #404      +/-   ##
==========================================
+ Coverage   94.61%   96.54%   +1.92%     
==========================================
  Files          14       14              
  Lines        4178     4195      +17     
==========================================
+ Hits         3953     4050      +97     
+ Misses        225      145      -80     

:umbrella: View full report in Codecov by Sentry.
:loudspeaker: Have feedback on the report? Share it here.

[c42f]

Heya, we just moved JuliaSyntax into Base in https://github.com/JuliaLang/julia/pull/59870 - if you want to re-propose this PR over there, I've created a branch https://github.com/JuliaLang/JuliaSyntax.jl/tree/pr-for-Base/404 to make this easier.

To make use of the branch, you can use the following steps:

• Clone JuliaLang/julia
• Add JuliaSyntax as a git remote
• Check out pr-for-Base/404
• Rebase on top of master

For example:

# git clone [email protected]:JuliaLang/julia julia_dir
# cd julia_dir

git remote add JuliaSyntax [email protected]:JuliaLang/JuliaSyntax.jl
git fetch JuliaSyntax
git checkout pr-for-Base/404
git rebase origin/master

Sorry this wasn't dealt with prior to the big move!