lark
lark copied to clipboard
lark-parser
AttributeError: 'NoneType' object has no attribute 'char_pos'
Description
The following line seems to raise an error on None line:
https://github.com/lark-parser/lark/blob/f79772cd4c6d2076b5dc01f399dbb816cc484f77/lark/lexer.py#L615-L616
Running lark=1.3.1 produces the error, while it does not occur with lark==1.2.2.
Example
https://github.com/crim-ca/weaver/actions/runs/18957464025/job/54137521089
File "/home/runner/work/weaver/weaver/tests/wps_restapi/test_swagger_definitions.py", line 425, in test_collection_input_filter_parsing_error
sd.ExecuteCollectionInput().deserialize(input_data)
File "/home/runner/work/weaver/weaver/weaver/wps_restapi/swagger_definitions.py", line 1761, in deserialize
self.convert(filter_expr, filter_lang)
File "/home/runner/work/weaver/weaver/weaver/wps_restapi/swagger_definitions.py", line 1724, in convert
parsed_expr = self.validate(filter_expr, filter_lang)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/home/runner/work/weaver/weaver/weaver/wps_restapi/swagger_definitions.py", line 1713, in validate
return self.parse(filter_expr, filter_lang)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/home/runner/work/weaver/weaver/weaver/utils.py", line 401, in wrapped
return func(*args, **kwargs)
^^^^^^^^^^^^^^^^^^^^^
File "/home/runner/work/weaver/weaver/weaver/wps_restapi/swagger_definitions.py", line 1697, in parse
parsed_expr = ecql.parse(filter_expr)
^^^^^^^^^^^^^^^^^^^^^^^
File "/opt/hostedtoolcache/Python/3.11.13/x64/lib/python3.11/site-packages/pygeofilter/parsers/ecql/parser.py", line 211, in parse
return parser.parse(cql_text)
^^^^^^^^^^^^^^^^^^^^^^
File "/opt/hostedtoolcache/Python/3.11.13/x64/lib/python3.11/site-packages/lark/lark.py", line 677, in parse
return self.parser.parse(text, start=start, on_error=on_error)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/opt/hostedtoolcache/Python/3.11.13/x64/lib/python3.11/site-packages/lark/parser_frontends.py", line 131, in parse
return self.parser.parse(stream, chosen_start, **kw)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/opt/hostedtoolcache/Python/3.11.13/x64/lib/python3.11/site-packages/lark/parsers/lalr_parser.py", line 42, in parse
return self.parser.parse(lexer, start)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/opt/hostedtoolcache/Python/3.11.13/x64/lib/python3.11/site-packages/lark/parsers/lalr_parser.py", line 88, in parse
return self.parse_from_state(parser_state)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/opt/hostedtoolcache/Python/3.11.13/x64/lib/python3.11/site-packages/lark/parsers/lalr_parser.py", line 100, in parse_from_state
for token in state.lexer.lex(state):
File "/opt/hostedtoolcache/Python/3.11.13/x64/lib/python3.11/site-packages/lark/lexer.py", line 689, in lex
yield lexer.next_token(lexer_state, parser_state)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/opt/hostedtoolcache/Python/3.11.13/x64/lib/python3.11/site-packages/lark/lexer.py", line 616, in next_token
while line_ctr.char_pos < lex_state.text.end:
^^^^^^^^^^^^^^^^^
AttributeError: 'NoneType' object has no attribute 'char_pos'
@fmigneault Thanks for reporting. Are you using a custom lexer, by any chance?
I am relying on the lexer definitions found in https://github.com/geopython/pygeofilter/tree/main/pygeofilter/parsers
The test is defined here: https://github.com/crim-ca/weaver/blob/cba028a11d1bee42d5b7c81211ad6ec8b1a717bf/tests/wps_restapi/test_swagger_definitions.py#L403-L426
The error log displays pygeofilter/parsers/ecql/parser.py, so the code should be equivalent to:
from pygeofilter.parsers import ecql
filter_expr = {"test": "bad"}
parsed_expr = ecql.parse(filter_expr)
The operation is expected to fail due to invalid format (purpose of the test), but it was somewhat handled before with (TypeError, ValueError) being raised (https://github.com/crim-ca/weaver/blob/cba028a11d1bee42d5b7c81211ad6ec8b1a717bf/weaver/wps_restapi/swagger_definitions.py#L1710-L1719) rather than the unexpected AttributeError: 'NoneType' access.
@fmigneault Can you please try running it with PR #1567 and see if that restores the expected behavior?
@erezsh It did not fix it, but the error is different.
====================================================================================================================================== short test summary info ======================================================================================================================================
FAILED tests/wps_restapi/test_swagger_definitions.py::test_collection_input_filter_parsing_error[filter_data3-cql-text] - AttributeError: 'HashDict' object has no attribute 'end'
FAILED tests/wps_restapi/test_swagger_definitions.py::test_collection_input_filter_parsing_error[filter_data4-cql2-text] - AttributeError: 'HashDict' object has no attribute 'end'
FAILED tests/wps_restapi/test_swagger_definitions.py::test_collection_input_filter_parsing_error[filter_data5-ecql] - AttributeError: 'HashDict' object has no attribute 'end'
FAILED tests/wps_restapi/test_swagger_definitions.py::test_collection_input_filter_parsing_error[filter_data6-cql] - AttributeError: 'HashDict' object has no attribute 'end'
FAILED tests/wps_restapi/test_swagger_definitions.py::test_collection_input_filter_parsing_error[filter_data7-simple-cql] - AttributeError: 'HashDict' object has no attribute 'end'
============================================================================================= 5 failed, 2923 passed, 3 skipped, 516 deselected, 3 xfailed, 1 xpassed, 342 warnings in 235.52s (0:03:55) =============================================================================================
make: *** [Makefile:420: test-unit-only] Error 1
❯ pip show lark
Name: lark
Version: 1.3.1
Summary: a modern parsing library
Home-page: https://github.com/lark-parser/lark
Author:
Author-email: Erez Shinan <[email protected]>
License: MIT
Location: /home/francis/dev/conda/envs/weaver-py312/lib/python3.12/site-packages
Requires:
Required-by: pygeofilter
❯ cat /home/francis/dev/miniconda/envs/weaver-py312/lib/python3.12/site-packages/lark-1.3.1.dist-info/direct_url.json
{"dir_info": {}, "url": "file:///home/francis/dev/lark"}%
❯ cd /home/francis/dev/lark && git status
On branch issue1565
Your branch is up to date with 'origin/issue1565'.
Error for one of the failed test:
___________________________________________________________________________________________________________________ test_collection_input_filter_parsing_error[filter_data5-ecql] ___________________________________________________________________________________________________________________
Traceback (most recent call last):
File "/home/francis/dev/conda/envs/weaver-py312/lib/python3.12/site-packages/_pytest/runner.py", line 344, in from_call
result: TResult | None = func()
^^^^^^
File "/home/francis/dev/conda/envs/weaver-py312/lib/python3.12/site-packages/_pytest/runner.py", line 246, in <lambda>
lambda: runtest_hook(item=item, **kwds), when=when, reraise=reraise
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/home/francis/dev/conda/envs/weaver-py312/lib/python3.12/site-packages/pluggy/_hooks.py", line 512, in __call__
return self._hookexec(self.name, self._hookimpls.copy(), kwargs, firstresult)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/home/francis/dev/conda/envs/weaver-py312/lib/python3.12/site-packages/pluggy/_manager.py", line 120, in _hookexec
return self._inner_hookexec(hook_name, methods, kwargs, firstresult)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/home/francis/dev/conda/envs/weaver-py312/lib/python3.12/site-packages/pluggy/_callers.py", line 167, in _multicall
raise exception
File "/home/francis/dev/conda/envs/weaver-py312/lib/python3.12/site-packages/pluggy/_callers.py", line 139, in _multicall
teardown.throw(exception)
File "/home/francis/dev/conda/envs/weaver-py312/lib/python3.12/site-packages/_pytest/logging.py", line 850, in pytest_runtest_call
yield
File "/home/francis/dev/conda/envs/weaver-py312/lib/python3.12/site-packages/pluggy/_callers.py", line 139, in _multicall
teardown.throw(exception)
File "/home/francis/dev/conda/envs/weaver-py312/lib/python3.12/site-packages/_pytest/capture.py", line 900, in pytest_runtest_call
return (yield)
^^^^^
File "/home/francis/dev/conda/envs/weaver-py312/lib/python3.12/site-packages/pluggy/_callers.py", line 139, in _multicall
teardown.throw(exception)
File "/home/francis/dev/conda/envs/weaver-py312/lib/python3.12/site-packages/_pytest/skipping.py", line 263, in pytest_runtest_call
return (yield)
^^^^^
File "/home/francis/dev/conda/envs/weaver-py312/lib/python3.12/site-packages/pluggy/_callers.py", line 121, in _multicall
res = hook_impl.function(*args)
^^^^^^^^^^^^^^^^^^^^^^^^^
File "/home/francis/dev/conda/envs/weaver-py312/lib/python3.12/site-packages/_pytest/runner.py", line 178, in pytest_runtest_call
item.runtest()
File "/home/francis/dev/conda/envs/weaver-py312/lib/python3.12/site-packages/_pytest/python.py", line 1671, in runtest
self.ihook.pytest_pyfunc_call(pyfuncitem=self)
File "/home/francis/dev/conda/envs/weaver-py312/lib/python3.12/site-packages/pluggy/_hooks.py", line 512, in __call__
return self._hookexec(self.name, self._hookimpls.copy(), kwargs, firstresult)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/home/francis/dev/conda/envs/weaver-py312/lib/python3.12/site-packages/pluggy/_manager.py", line 120, in _hookexec
return self._inner_hookexec(hook_name, methods, kwargs, firstresult)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/home/francis/dev/conda/envs/weaver-py312/lib/python3.12/site-packages/pluggy/_callers.py", line 167, in _multicall
raise exception
File "/home/francis/dev/conda/envs/weaver-py312/lib/python3.12/site-packages/pluggy/_callers.py", line 121, in _multicall
res = hook_impl.function(*args)
^^^^^^^^^^^^^^^^^^^^^^^^^
File "/home/francis/dev/conda/envs/weaver-py312/lib/python3.12/site-packages/_pytest/python.py", line 157, in pytest_pyfunc_call
result = testfunction(**testargs)
^^^^^^^^^^^^^^^^^^^^^^^^
File "/home/francis/dev/weaver/tests/wps_restapi/test_swagger_definitions.py", line 425, in test_collection_input_filter_parsing_error
sd.ExecuteCollectionInput().deserialize(input_data)
File "/home/francis/dev/weaver/weaver/wps_restapi/swagger_definitions.py", line 1761, in deserialize
self.convert(filter_expr, filter_lang)
File "/home/francis/dev/weaver/weaver/wps_restapi/swagger_definitions.py", line 1724, in convert
parsed_expr = self.validate(filter_expr, filter_lang)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/home/francis/dev/weaver/weaver/wps_restapi/swagger_definitions.py", line 1713, in validate
return self.parse(filter_expr, filter_lang)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/home/francis/dev/weaver/weaver/utils.py", line 401, in wrapped
return func(*args, **kwargs)
^^^^^^^^^^^^^^^^^^^^^
File "/home/francis/dev/weaver/weaver/wps_restapi/swagger_definitions.py", line 1697, in parse
parsed_expr = ecql.parse(filter_expr)
^^^^^^^^^^^^^^^^^^^^^^^
File "/home/francis/dev/conda/envs/weaver-py312/lib/python3.12/site-packages/pygeofilter/parsers/ecql/parser.py", line 211, in parse
return parser.parse(cql_text)
^^^^^^^^^^^^^^^^^^^^^^
File "/home/francis/dev/conda/envs/weaver-py312/lib/python3.12/site-packages/lark/lark.py", line 677, in parse
return self.parser.parse(text, start=start, on_error=on_error)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/home/francis/dev/conda/envs/weaver-py312/lib/python3.12/site-packages/lark/parser_frontends.py", line 131, in parse
return self.parser.parse(stream, chosen_start, **kw)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/home/francis/dev/conda/envs/weaver-py312/lib/python3.12/site-packages/lark/parsers/lalr_parser.py", line 42, in parse
return self.parser.parse(lexer, start)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/home/francis/dev/conda/envs/weaver-py312/lib/python3.12/site-packages/lark/parsers/lalr_parser.py", line 88, in parse
return self.parse_from_state(parser_state)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/home/francis/dev/conda/envs/weaver-py312/lib/python3.12/site-packages/lark/parsers/lalr_parser.py", line 100, in parse_from_state
for token in state.lexer.lex(state):
^^^^^^^^^^^^^^^^^^^^^^
File "/home/francis/dev/conda/envs/weaver-py312/lib/python3.12/site-packages/lark/lexer.py", line 691, in lex
yield lexer.next_token(lexer_state, parser_state)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/home/francis/dev/conda/envs/weaver-py312/lib/python3.12/site-packages/lark/lexer.py", line 618, in next_token
while line_ctr.char_pos < lex_state.text.end:
^^^^^^^^^^^^^^^^^^
AttributeError: 'HashDict' object has no attribute 'end'
--------------------------------------------------------------------------------------------------------------------------------------- Captured stdout call ----------------------------------------------------------------------------------------------------------------------------------------
STATE STACK DUMP
----------------
0) {<_binary_spatial_predicate_func : * INTERSECTS>, <wkt__geometry : * wkt__multipoint>, <predicate : * attribute __ANON_3>, <wkt__multipolygon : * MULTIPOLYGON LPAR LPAR wkt__coordinate_lists RPAR RPAR>, <wkt__geometry : * wkt__point>, <_binary_spatial_predicate_func : * CONTAINS>, <_binary_spatial_predicate_func : * TOUCHES>, <temporal_predicate : * expression DURING period>, <literal : * ewkt_geometry>, <_binary_spatial_predicate_func : * OVERLAPS>, <temporal_predicate : * expression AFTER DATETIME>, <wkt__multilinestring : * MULTILINESTRING LPAR wkt__coordinate_lists RPAR>, <predicate : * expression LESSTHAN expression>, <predicate : * expression ILIKE SINGLE_QUOTED>, <number : * FLOAT>, <sum : * sum PLUS product>, <attribute : * NAME>, <condition_1 : * LPAR condition RPAR>, <ewkt_geometry : * wkt__geometry>, <sum : * sum MINUS product>, <predicate : * expression IN LPAR expression __predicate_star_4 RPAR>, <atom : * MINUS atom>, <wkt__linestring : * LINESTRING LPAR wkt__coordinate_list RPAR>, <predicate : * spatial_predicate>, <predicate : * expression NOT IN LPAR expression RPAR>, <condition : * condition AND condition_1>, <wkt__geometrycollection : * GEOMETRYCOLLECTION LPAR wkt__geometry __wkt__geometrycollection_star_0 RPAR>, <predicate : * expression LIKE SINGLE_QUOTED>, <literal : * SINGLE_QUOTED>, <predicate : * expression IS NOT NULL>, <wkt__geometry : * wkt__geometrycollection>, <literal : * BOOLEAN>, <product : * product STAR atom>, <predicate : * attribute EXISTS>, <wkt__geometry : * wkt__linestring>, <temporal_predicate : * expression DURING OR AFTER period>, <spatial_predicate : * _binary_spatial_predicate_func LPAR expression COMMA expression RPAR>, <start : * condition>, <predicate : * EXCLUDE>, <atom : * literal>, <spatial_predicate : * BBOX LPAR expression COMMA full_number COMMA full_number COMMA full_number COMMA full_number RPAR>, <sum : * product>, <predicate : * expression IN LPAR expression RPAR>, <atom : * attribute>, <predicate : * expression EQUAL expression>, <condition_1 : * predicate>, <spatial_predicate : * RELATE LPAR expression COMMA expression COMMA SINGLE_QUOTED RPAR>, <_distance_spatial_predicate_func : * DWITHIN>, <predicate : * expression NOT BETWEEN expression AND expression>, <predicate : * expression IS NULL>, <product : * atom>, <condition : * condition_1>, <_binary_spatial_predicate_func : * WITHIN>, <wkt__point : * POINT LPAR wkt__coordinate RPAR>, <_binary_spatial_predicate_func : * DISJOINT>, <wkt__multipolygon : * MULTIPOLYGON LPAR LPAR wkt__coordinate_lists RPAR __wkt__multipolygon_star_2 RPAR>, <predicate : * expression NOT ILIKE SINGLE_QUOTED>, <_binary_spatial_predicate_func : * CROSSES>, <predicate : * expression __ANON_0 expression>, <$root_start : * start>, <atom : * NAME LPAR expression RPAR>, <predicate : * expression __ANON_2 expression>, <attribute : * DOUBLE_QUOTED>, <wkt__multipoint : * MULTIPOINT LPAR LPAR wkt__coordinate RPAR RPAR>, <spatial_predicate : * BBOX LPAR expression COMMA full_number COMMA full_number COMMA full_number COMMA full_number COMMA SINGLE_QUOTED RPAR>, <temporal_predicate : * expression BEFORE OR DURING period>, <wkt__geometry : * wkt__multilinestring>, <product : * product SLASH atom>, <condition_1 : * NOT predicate>, <atom : * NAME LPAR RPAR>, <wkt__geometry : * wkt__polygon>, <number : * INT>, <predicate : * expression __ANON_1 expression>, <atom : * NAME LPAR expression __predicate_star_4 RPAR>, <wkt__geometry : * wkt__multipolygon>, <literal : * envelope>, <predicate : * INCLUDE>, <predicate : * expression NOT LIKE SINGLE_QUOTED>, <wkt__polygon : * POLYGON LPAR wkt__coordinate_lists RPAR>, <temporal_predicate : * expression BEFORE DATETIME>, <condition : * condition OR condition_1>, <wkt__multipoint : * MULTIPOINT LPAR wkt__coordinate_list RPAR>, <wkt__geometrycollection : * GEOMETRYCOLLECTION LPAR wkt__geometry RPAR>, <predicate : * temporal_predicate>, <literal : * number>, <spatial_predicate : * _distance_spatial_predicate_func LPAR expression COMMA expression COMMA number COMMA distance_units RPAR>, <predicate : * expression BETWEEN expression AND expression>, <wkt__multipoint : * MULTIPOINT LPAR LPAR wkt__coordinate RPAR __wkt__multipoint_star_1 RPAR>, <attribute : * QUALIFIED_NAME>, <ewkt_geometry : * SRID EQUAL wkt__INT SEMICOLON wkt__geometry>, <_binary_spatial_predicate_func : * EQUALS>, <_distance_spatial_predicate_func : * BEYOND>, <predicate : * expression NOT IN LPAR expression __predicate_star_4 RPAR>, <expression : * sum>, <atom : * LPAR expression RPAR>, <predicate : * expression MORETHAN expression>, <envelope : * ENVELOPE LPAR number number number number RPAR>}