Tokenizer fails on expression ending in integer '0'

[benmaier]

Minimal example reproducing bug

In [1]: from javalang.tokenizer import tokenize

In [2]: list(tokenize("a == 0"))
---------------------------------------------------------------------------
TypeError                                 Traceback (most recent call last)
Cell In[3], line 1
----> 1 list(tokenize("a == 0"))

File .\env\Lib\site-packages\javalang\tokenizer.py:538, in JavaTokenizer.tokenize(self)
    535     self.read_string()
    537 elif c in '0123456789':
--> 538     token_type = self.read_integer_or_float(c, c_next)
    540 elif self.is_java_identifier_start(c):
    541     token_type = self.read_identifier()

File .\env\Lib\site-packages\javalang\tokenizer.py:370, in JavaTokenizer.read_integer_or_float(self, c, c_next)
    369 def read_integer_or_float(self, c, c_next):
--> 370     if c == '0' and c_next in 'xX':
    371         return self.read_hex_integer_or_float()
    372     elif c == '0' and c_next in 'bB':

TypeError: 'in <string>' requires string as left operand, not NoneType

Expected behavior

In [1]: from javalang.tokenizer import tokenize

In [2]: list(tokenize("a == 0"))
Out[2]: 
[Identifier "a" line 1, position 1,
 Operator "==" line 1, position 3,
 DecimalInteger "0" line 1, position 6]

Source of problem

Lies in the expectation that an integer of '0' would always be followed by another token (this is done to check for possible hexadecimal or binary notation), see https://github.com/c2nes/javalang/blob/master/javalang/tokenizer.py#L370 . But this does not have to be the case.

Suggested fix

catch None for c_next in read_integer_or_float

    def read_integer_or_float(self, c, c_next):
        if c == '0' and c_next is None:
            return self.read_decimal_float_or_integer()
        elif c == '0' and c_next in 'xX':
            return self.read_hex_integer_or_float()
        elif c == '0' and c_next in 'bB':
            self.read_bin_integer()
            return BinaryInteger
        elif c == '0' and c_next in '01234567':
            self.read_octal_integer()
            return OctalInteger
        else:
            return self.read_decimal_float_or_integer()