refactor: on demand lexing

[mhanberg]

Description

Recursive Descent parsers tend to lex the document as they go, rather than all up front. This allows you to know what to do in the case a bad token is lexed in the context of the current parsing state.

The existing lexer (elixir_tokenizer) is designed to work with a parser generator (yecc), so it parses it all up front, and if it reaches a bad token, it bails and returns an error.

Solution

Refactor the existing :elixir_tokenizer (vendored as :spitfire_tokenzier to enable on demand lexing.

The API of the module should basically consist of

• new - creates a new lexer state instance from a source code string
• next_token - returns the next token in the document and the new lexer state

The token structure should stay the same and contain the same semantics.

Considerations

• the existing implementation has a special interpolation module
• conversion from Erlang to Elixir is not necessary, but might eventually be done