Uncaught TokenError while fuzzing

[correctmost]

Steps to reproduce

The following testcase from OSS-Fuzz raises a TokenError with Python 3.8.0 - 3.11.9:

import astroid

code="\rclass\x0cj:(\n)"
astroid.builder.parse(code)

With Python 3.12.0+, the code no longer raises an exception.

OSS-Fuzz is currently running 3.8.3 and reports the exception as a crash: https://bugs.chromium.org/p/oss-fuzz/issues/detail?id=67696 (report not public yet)

Given that Pylint catches TokenErrors, should the fuzzer on OSS-Fuzz also catch TokenErrors to avoid similar OSS-Fuzz reports? Or should astroid handle this case more gracefully?

Here are the current exceptions that the fuzzer catches:

  try:
    astroid.builder.parse(code)
  except (
      astroid.exceptions._NonDeducibleTypeHierarchy,
      astroid.modutils.NoSourceFile,
      astroid.exceptions.AstroidError,
      astroid.exceptions.UseInferenceDefault,
  ):
    pass

-- Source

Current behavior

Traceback (most recent call last):
  File "corpus/a.py", line 5, in <module>
    astroid.builder.parse(code)
  File "lib/python3.11/site-packages/astroid/builder.py", line 303, in parse
    return builder.string_build(code, modname=module_name, path=path)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "lib/python3.11/site-packages/astroid/builder.py", line 151, in string_build
    module, builder = self._data_build(data, modname, path)
                      ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "lib/python3.11/site-packages/astroid/builder.py", line 206, in _data_build
    module = builder.visit_module(node, modname, node_file, package)
             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "lib/python3.11/site-packages/astroid/rebuilder.py", line 194, in visit_module
    [self.visit(child, newnode) for child in node.body],
    ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "lib/python3.11/site-packages/astroid/rebuilder.py", line 194, in <listcomp>
    [self.visit(child, newnode) for child in node.body],
     ^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "lib/python3.11/site-packages/astroid/rebuilder.py", line 483, in visit
    return visit_method(node, parent)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "lib/python3.11/site-packages/astroid/rebuilder.py", line 853, in visit_classdef
    position=self._get_position_info(node, newnode),
             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "lib/python3.11/site-packages/astroid/rebuilder.py", line 136, in _get_position_info
    for t in generate_tokens(StringIO(data).readline):
  File "lib/python3.11/tokenize.py", line 525, in _tokenize
    raise TokenError("EOF in multi-line statement", (lnum, 0))
tokenize.TokenError: ('EOF in multi-line statement', (2, 0))

Expected behavior

Suppressed fuzzer failures

Version

d8dbc462243a65d72153ace2e7ac87daa9ffc1cf