code_tokenize

code_tokenize copied to clipboard

Unexpected token with no matched text at the end of tokenstream

4

Consider the following code: ```python3 text = """private void unlockMap(Player player) { TowerData towerData = player.getTowerData(); if (!towerData.getClass().equals(TowerData.class)) { CommandHandler.sendTranslatedMessage(player, "commands.generic.no_permissions"); } else { if (towerData.""" import code_tokenize as ctok...

LakshyAAAgrawal

string literals lost during tokenization for C/C++ code.

2

For the following code ```python import code_tokenize as ctok sample = """ #include int main() { printf("hello world"); } """ ctok.tokenize(sample, lang = "cpp") ``` Output: ```txt [#include, , ,...

5c4lar