Unit tests for tokenization

[scripthunter7]

Based on the TMLanguage, it is possible to tokenize various inputs and then compare them with a token list. This makes test_rules.txt unnecessary in practice (good for a demo at most), since unit tests can be used to ensure that there is no breakage and that tokenization always happens as expected.

See: https://www.npmjs.com/package/vscode-textmate