baron icon indicating copy to clipboard operation
baron copied to clipboard

baron raise exception when unicode is in code

Open James4Ever0 opened this issue 3 years ago • 0 comments

example:

the unicode below are just attributes inside a class.

orders=[
        BSP.all.order.最多点击,
        BSP.all.order.最多收藏,
        BSP.all.order.最新发布,
        BSP.all.order.最多弹幕,
        BSP.all.order.综合排序,
    ]

exception:

Traceback (most recent call last):
  File "/media/root/parrot/pyjom/tests/skipexception_code_and_continue_resurrection_time_travel_debugging/codemod_redbaron/redbaron_test.py", line 6, in <module>
    r=redbaron.RedBaron(t)
  File "/usr/local/lib/python3.9/dist-packages/redbaron/redbaron.py", line 36, in __init__
    self.node_list = base_nodes.NodeList.from_fst(baron.parse(source_code), parent=self, on_attribute="root")
  File "/usr/local/lib/python3.9/dist-packages/baron/baron.py", line 49, in parse
    tokens = tokenize(source_code, False)
  File "/usr/local/lib/python3.9/dist-packages/baron/baron.py", line 70, in tokenize
    splitted = split(pouet)
  File "/usr/local/lib/python3.9/dist-packages/baron/spliter.py", line 6, in split
    return list(split_generator(sequence))
  File "/usr/local/lib/python3.9/dist-packages/baron/spliter.py", line 76, in split_generator
    raise UntreatedError("Untreated elements: %s" % iterator.rest_of_the_sequence().__repr__()[:50])
baron.spliter.UntreatedError: Untreated elements: '最多点击,\n        BSP.all.order.最多收藏,\n        BSP.a

James4Ever0 avatar Nov 09 '22 08:11 James4Ever0