baron
baron copied to clipboard
Inconsistent failure to parse documentation/comment blocks ("""/''')
I get an Unexpected 'space' token: error when parsing comment blocks that have more than one word per line. For example, this parses fine even with spaces on each line:
'''
One
Two
Three
'''
However move one of the words to make a multi-word line:
'''
One
Two Three
'''
Results in this:
File "/usr/lib/python2.6/site-packages/baron/baron.py", line 51, in parse
tokens = tokenize(source_code, False)
File "/usr/lib/python2.6/site-packages/baron/baron.py", line 72, in tokenize
return mark_indentation(inner_group(space_group(_tokenize(group(split(pouet)), print_function))))
File "/usr/lib/python2.6/site-packages/baron/inner_formatting_grouper.py", line 112, in group
return list(group_generator(sequence))
File "/usr/lib/python2.6/site-packages/baron/inner_formatting_grouper.py", line 189, in group_generator
raise UnExpectedFormattingToken(debug_text)
baron.inner_formatting_grouper.UnExpectedFormattingToken: Unexpected 'space' token:
1
2 One
3 Two <--- here
Should have been grouped on either ('NAME', 'Two') (before) or ('NAME', 'Three') (after) token.
The part where it behaves inconsistency is in my actual code block that threw this error, the "here" marker was 3 words into the first line, so it's not necessarily going to get thrown on the first word boundary.
I cannot reproduce your error. I tried:
In [4]: baron.parse("def a():\n '''\none\ntwo three\n'''\n pass")
Out[4]:
[{'arguments': [],
'decorators': [],
'fifth_formatting': [],
'first_formatting': [{'type': 'space', 'value': ' '}],
'fourth_formatting': [],
'name': 'a',
'second_formatting': [],
'sixth_formatting': [],
'third_formatting': [],
'type': 'def',
'value': [{'formatting': [],
'indent': ' ',
'type': 'endl',
'value': '\n'},
{'first_formatting': [],
'second_formatting': [],
'type': 'string',
'value': "'''\none\ntwo three\n'''"},
{'formatting': [], 'indent': ' ', 'type': 'endl', 'value': '\n'},
{'type': 'pass'},
{'formatting': [], 'indent': '', 'type': 'endl', 'value': '\n'}]}]
That being said, it looks like #92 could fix this issue.