Layout21
Layout21 copied to clipboard
Numeric looking token fix
Fixes #55 for numeric looking tokens.
I have some concerns over performance (relative to the previous implementation), as I constructed a String
to capture relevant part of the chars iterator. This can probably be improved, but it wasnt obvious to me that I had the right data to do it. Also it is checking i32 || f64 conversion to str as a means of deciding. It's possible a custom int/float lexer could shortcut by identifying the more trivial cases before resorting to the conversion routines.
Some tests were added with positive and integers, numeric floats and exponential floats, plus a token that starts with a number and has letters.