Suggestion: Unit tests

[de-code]

I believe there aren't currently any unit tests (apologies if there are)...

It would be good to have some unit tests, e.g. for the functionality that decides when to start a new token.

Would libcheck be a good tool for that?