Tokenize.jl

Tokenize.jl copied to clipboard

Since recently, ``tokenize`` fails on a file containing double quotes. Example ``` using Tokenize # Create test file open("test.txt", "w") do out print(out, "f(\"data\")") end # Display test file open("test.txt")...

MichelJuillard

Use Documenter

7

I did not add a badge ([](https://JuliaLang.github.io/Tokenize.jl/dev)) to the README since there isn't much documentation yet. But with this scaffolding in place, it will be easier to add some documentation...

goretkin

Minor documentation

2

goretkin

```julia using Tokenize collect(tokenize("0x.")) ```

ZacLN

Don't error on invalid chars

1

ZacLN

KristofferC

KristofferC

Hexadecimal ints

1

https://github.com/JuliaLang/julia/issues/16356#issuecomment-312267173 Cc @ZacLN

KristofferC

Get to 100% coverage

1

Important that all branches are covered since we do some "ugly" coding for performance reasons.

KristofferC

It is unnecessary for all tokens to store this. Instead just make a few error kinds and have a lookup table for their meaning.

KristofferC