Refactor lexer

[andrewchambers]

I feel like the lexer could be simplified by dumbing the code down a bit then using a regex to validate floats and numbers.

[dhamidi]

Hey!

I just skimmed the lexer code; it doesn't look too complicated. There is a talk by Rob Pike about writing lexers in Go, which details how the lexer for text/template is implemented. Maybe that proves useful.

[andrewchambers]

Thanks, I'll check out the code there. It's probably pretty good to compare a few techniques for implementing this.

Compared with https://github.com/rui314/8cc/blob/master/lex.c , The state machines in the number/float/string lexing take up a bunch of space, even if it is fairly simple.

[andrewchambers]

I think the performance of the lexer is fairly critical, currently i think most of the time is spent allocating new objects. I want to emphasize clean code over performance for now though.