C23-compliant lexer

[ForNeVeR]

Currently, we're using a C lexer from Yoakke, a wonderful language toolkit.

Though, it's not currently known how good the lexer is regarding the actual standard.

We should write tests and ensure the lexer is good enough to be the backbone of our parser, with relation to the following sections of the C standard:

• [ ] 6.4 Lexical elements
  • [ ] 6.4.1 Keywords
  • [ ] 6.4.2 Identifiers
  • [ ] 6.4.3 Universal character names
  • [ ] 6.4.4 Constants
    • [ ] 6.4.4.1 Integer constants
    • [ ] 6.4.4.2 Floating constants
    • [ ] 6.4.4.3 Enumeration constants
    • [ ] 6.4.4.4 Character constants
  • [ ] 6.4.5 String literals
  • [ ] 6.4.6 Punctuators
  • [ ] 6.4.7 Header names
  • [ ] 6.4.8 Preprocessing numbers
  • [ ] 6.4.9 Comments

Note that certain features of the C standard may be hard to implement in the actual lexer, so they may involve parser support. This is okay, but still have to be thoroughly tested.