Unexpected token issues triggered by a comment

[nilshamerlinck]

First case:

class Test(object):
    def test(self):
        # IF True
        if True:
            print True
        # ELSE
        else:
            print False

Triggers:

baron.parser.ParsingError: Error, got an unexpected token NAME here:

   1 class Test(object):
   2     def test(self):
   3         # IF True
   4         if True:
   5             print True<---- here

Second case:

class Test(object):
    def test(self):
        # IF True
        if True:
            if True:
                test = True
            else:
                test = False
        # ELSE
        else:
            print False

Triggers:

baron.parser.ParsingError: Error, got an unexpected token ELSE here:

   3         # IF True
   4         if True:
   5             if True:
   6                 test = True
   7             else:
   8                 test = False
   9         # ELSE
  10         else<---- here

Both snippets are parsed without error if I remove the second comment:

        # ELSE

[Psycojoker]

Hello,

Thanks for reporting, this is actually a variation of this bug https://github.com/PyCQA/baron/issues/11 which is a weakness of the current parser which is based a bit too much on the original python grammar which doesn't takes comments into account.

[rowillia]

@Psycojoker Any suggestions for how I could get started on fixing this?

[Thijs-Riezebeek]

Running into this as well...

[Psycojoker]

@rowillia I've been thinking about it a bit and well... I'm not expecting this to be simple to fix (things are always more complexe than expected in red/baron).

My current intuition is that the rule to introduce the DEDENT token here https://github.com/PyCQA/baron/blob/a3cc187099c3a3a07a1869dc7f203535ade974df/baron/indentation_marker.py#L79-L86 should do that only when it encounter or will encounter a token that is not related to formatting. Or this whole process should only take into account indentation related lines that have meaningful codes on it.

But to be honest it's been quite a long time since I've looked at this part of the code (so my intuition might not be accurate) and I really have problems going back on that project after the big burnout I've made on it :/

[Psycojoker]

Ok, the fix was ridiculously simple, I regret taking so much time to dig into this https://github.com/PyCQA/baron/commit/bca4d4296b7a0442821075819e5b0a1ca7f145a3

@nilshamerlinck @rowillia @Thijs-Riezebeek can you test this and tell me if this solves your situation plz?

[Psycojoker]

No one :x ?

[nilshamerlinck]

@Psycojoker thanks for the fix, it works :)