Comments block parsing?

[astrand]

Moved from http://bugzilla.lysator.liu.se/show_bug.cgi?id=1605:

The following code

d = {"a": "A", "b": "B", # comment "c": "C"}

is not correctly handled: the last line containing the dictionary key "c" is ignored, and the generated obfuscated code is something like

oo000 = { "a" : "A" , "b" : "B" ,

if 9 - 9: Ii . o0o00Oo0O - iI11I1II1I1I

Marco Lombardi

------- Comment #1 From Marco Lombardi 2006-04-03 13:28:40 [reply] -------

I checked the problem in more detail, and it seems that this bug is due to the code lines

        elif t_type == tokenize.STRING:
            if self.first_on_line and self.brlevel == 0:
                # Skip over docstrings
                # FIXME: This simple approach fails with:
                # "foo"; print 3

In other words, the part of the dictionary after the comment is (wrongly) recognized as a docstring. A simple patch for this, which I did implement already, is to keep track for the number of parenthesis (), braces {}, and brackets [] encountered so far, and to prevent the recognition as a docstring of strings if we are inside a tuple, dictionary, or list.

------- Comment #2 From Peter �strand 2006-04-03 18:47:51 [reply] -------

Can you attach your patch to this bug?

------- Comment #3 From Marco Lombardi 2006-04-04 12:01:57 [reply] -------

Created an attachment (id=95) [details] Patched pyobfuscate

This patched version of pyobfuscate should solve a few bugs:

1. It does the parsing of list comprehension, including multiple for/if such as [a*b for a in as for b in bs]
2. It fixes a problem with global variables (which inside functions are not recognized as such, but taken as local variables)
3. Avoid the use of noop lines inside tuples, lists, or dictionaries
4. Changes the way a source is read to match exactly the Python library (in particular, use "U" to read sources in different newline formats, and add a "\n" at the end of the source; see compile.parseFile)

------- Comment #4 From Marco Lombardi 2006-04-04 12:03:05 [reply] -------

Forgot to mention... The patched version is based on the STABLE source, and not on the CVS.