baron icon indicating copy to clipboard operation
baron copied to clipboard

Inconsistent failure to parse documentation/comment blocks ("""/''')

Open btgoodwin opened this issue 10 years ago • 1 comments

I get an Unexpected 'space' token: error when parsing comment blocks that have more than one word per line. For example, this parses fine even with spaces on each line:

'''
One 
Two
Three
'''

However move one of the words to make a multi-word line:

'''
One
Two Three
'''

Results in this:

File "/usr/lib/python2.6/site-packages/baron/baron.py", line 51, in parse
    tokens = tokenize(source_code, False)
File "/usr/lib/python2.6/site-packages/baron/baron.py", line 72, in tokenize
    return mark_indentation(inner_group(space_group(_tokenize(group(split(pouet)), print_function))))
File "/usr/lib/python2.6/site-packages/baron/inner_formatting_grouper.py", line 112, in group
    return list(group_generator(sequence))
File "/usr/lib/python2.6/site-packages/baron/inner_formatting_grouper.py", line 189, in group_generator
    raise UnExpectedFormattingToken(debug_text)
baron.inner_formatting_grouper.UnExpectedFormattingToken: Unexpected 'space' token:

   1 
   2 One
   3 Two <--- here

Should have been grouped on either ('NAME', 'Two') (before) or ('NAME', 'Three') (after) token.

The part where it behaves inconsistency is in my actual code block that threw this error, the "here" marker was 3 words into the first line, so it's not necessarily going to get thrown on the first word boundary.

btgoodwin avatar Jun 11 '15 13:06 btgoodwin

I cannot reproduce your error. I tried:

In [4]: baron.parse("def a():\n    '''\none\ntwo three\n'''\n    pass")
Out[4]: 
[{'arguments': [],
  'decorators': [],
  'fifth_formatting': [],
  'first_formatting': [{'type': 'space', 'value': ' '}],
  'fourth_formatting': [],
  'name': 'a',
  'second_formatting': [],
  'sixth_formatting': [],
  'third_formatting': [],
  'type': 'def',
  'value': [{'formatting': [],
    'indent': '    ',
    'type': 'endl',
    'value': '\n'},
   {'first_formatting': [],
    'second_formatting': [],
    'type': 'string',
    'value': "'''\none\ntwo three\n'''"},
   {'formatting': [], 'indent': '    ', 'type': 'endl', 'value': '\n'},
   {'type': 'pass'},
   {'formatting': [], 'indent': '', 'type': 'endl', 'value': '\n'}]}]

That being said, it looks like #92 could fix this issue.

ibizaman avatar Dec 22 '16 06:12 ibizaman