baron icon indicating copy to clipboard operation
baron copied to clipboard

Support python 3 non-ascii identifiers

Open tyrylu opened this issue 7 years ago • 1 comments

Hello. It would be helpful for one of my usecases of baron if it supported non ascii identifiers. Currently (in 0.6.6 but probably also in the latest master) the following code: >>> baron.parse("fňuk=42") Raises the following:

Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "C:\Python3\lib\site-packages\baron\baron.py", line 49, in parse
    tokens = tokenize(source_code, False)
  File "C:\Python3\lib\site-packages\baron\baron.py", line 70, in tokenize
    return mark_indentation(inner_group(space_group(_tokenize(group(split(pouet)), print_function))))
  File "C:\Python3\lib\site-packages\baron\spliter.py", line 6, in split
    return list(split_generator(sequence))
  File "C:\Python3\lib\site-packages\baron\spliter.py", line 76, in split_generator
    raise UntreatedError("Untreated elements: %s" % iterator.rest_of_the_sequence().__repr__()[:50])
baron.spliter.UntreatedError: Untreated elements: 'ňuk=42\n'

The reason looks quite clear, during the splitting the condition block with the identifier chars uses ascii only. It would be probably easy to fix this error without taking into account any python 2.x compatibility, but if it is desired, then it is more complicated. But anyway, could something be done with it?

tyrylu avatar Dec 07 '17 15:12 tyrylu

+1

asterbini avatar Jul 29 '20 17:07 asterbini