Names not defined in list comprehension in the IPython shell

[asmeurer]

In the IPython shell, you can't seem to access names from within list comprehensions:

In [16]: a = 1

In [17]: [a for i in range(10)]
---------------------------------------------------------------------------
NameError                                 Traceback (most recent call last)
/Users/aaronmeurer/Documents/Continuum/conda/conda/resolve.py in <module>()
----> 1 [a for i in range(10)]

/Users/aaronmeurer/Documents/Continuum/conda/conda/resolve.py in <listcomp>(.0)
----> 1 [a for i in range(10)]

NameError: global name 'a' is not defined

It doesn't matter if the name was defined within the session or if it comes from the function you are debugging. This is debugging within a function scope, if that matters.

[takluyver]

I think this is the same as ipython/ipython#136 and ipython/ipython#62. When you're already inside a function scope, list comprehensions effectively make a closure, but you can't define closures in dynamically compiled code (such as anything run in a shell).

There might be a way round this by combining locals and globals in collections.ChainMap, and treating the combination the globals for dynamic execution. But since we've been encouraging things like Django's shell to use start_ipython() rather than embed(), reports of this problem have mostly gone away, so it's not high priority. A debugger is a valid place to use embed(), though.

[asmeurer]

How is it working in the regular Python shell?

[takluyver]

As in outside a debugger? Global variables work differently to variables in nested non-local scopes, so we're not actually defining closures there. Functions have a reference to the global namespace, but when they're closures, they only keep a reference to the specific variables they've closed over, not the entire namespace of their creation scope.

If you mean there's a way of embedding the plain Python shell that does work in this context, then I can only guess that either it uses the ChainMap hack I described, or it does some black magic in the interpreter internals to get round this.

[asmeurer]

Within PuDB, you can choose what shell you use. The Python shell works just fine with this. So does the builtin shell. I don't know how it works.

[asmeurer]

I guess it does do something like that. See https://github.com/inducer/pudb/blob/master/pudb/shell.py#L50. I'll have to see if this can be used with IPython.

[takluyver]

I think it should be possible to do that with IPython.

[astromancer]

A very irritating problem. Temp fix via:
globals().update(locals()) Though you have to do this every time you want to use a new local variable in list comp...

[clebio]

Is it acceptable to up-vote, :+1: ? This affects me as well. I can add my use-case if requested.

[asmeurer]

I don't know how to do this. The IPython code is significantly more complicated than the python or bpython code (note that you need to look at the v11 function).

[nikhilweee]

4 years later, is there a solution yet?

[billtubbs]

Still not solved I guess. Someone above asked if this occurs in the regular iPython REPL. It does not. Here is a related stack overflow question with an example:

in iPython:

$ ipython
Python 3.6.5 | packaged by conda-forge | (default, Apr  6 2018, 13:44:09) 
Type 'copyright', 'credits' or 'license' for more information
IPython 6.4.0 -- An enhanced Interactive Python. Type '?' for help.

In [1]: import pdb; pdb.set_trace()
--Call--
> /Users/billtubbs/anaconda/envs/py36/lib/python3.6/site-packages/IPython/core/displayhook.py(247)__call__()
-> def __call__(self, result=None):
(Pdb) x = 1; [x for i in range(3)]
*** NameError: name 'x' is not defined

In regular Python REPL:

$ python
Python 3.6.5 | packaged by conda-forge | (default, Apr  6 2018, 13:44:09) 
[GCC 4.2.1 Compatible Apple LLVM 6.1.0 (clang-602.0.53)] on darwin
Type "help", "copyright", "credits" or "license" for more information.
>>> import pdb; pdb.set_trace()
--Return--
> <stdin>(1)<module>()->None
(Pdb) x = 1; [x for i in range(3)]
[1, 1, 1]

[billtubbs]

I does anyone know if this is fixed in version 7.1? It is not in 7.0:

$ ipython
Python 3.6.6 | packaged by conda-forge | (default, Jul 26 2018, 09:55:02) 
Type 'copyright', 'credits' or 'license' for more information
IPython 7.0.1 -- An enhanced Interactive Python. Type '?' for help.

In [1]: import pdb; pdb.set_trace()                                                     
--Call--
> /Users/billtubbs/anaconda/envs/py36/lib/python3.6/site-packages/IPython/core/displayhook.py(247)__call__()
-> def __call__(self, result=None):
(Pdb) x=1; [x for i in range(3)]
*** NameError: name 'x' is not defined

[asmeurer]

It's a PuDB problem, not an IPython problem. PuDB needs to properly fake the function locals dictionary as a globals dictionary when using the IPython shell. It does this for other shells (using SetPropogatingDict, see https://documen.tician.de/pudb/shells.html). It isn't clear to me just from glancing at the IPython shell code how to do that, but it likely isn't difficult.

[fx-kirin]

For those who are using pdb, one of the way to avoid this, use interact command.

https://stackoverflow.com/questions/17290314/possible-bug-in-pdb-module-in-python-3-when-using-list-generators

[lega911]

The same for filter + lambda

x = 5
list(filter(lambda s: x, [{}]))

Command line: [Ctrl-X] gives exception:

>>> list(filter(lambda s: x, [{}]))
Traceback (most recent call last):
  File "<pudb command line>", line 1, in <module>
  File "<pudb command line>", line 1, in <lambda>
NameError: global name 'x' is not defined

Tested in python 2.7, 3.7, pudb 2018.1

[stdedos]

As-seen on the mailing list:

On Mon, Mar 11, 2019 at 9:24 PM Andreas Kloeckner [email protected] wrote: Stavros,

Stavros Ntentos [email protected] writes:

Hello there,

I am trying to debug a single program, and I am interactively executing this:

>>> test = 'ab\r\nc\r\n\r\nde\r\nf'
>>> re.split(r'(?:\r?\n){2}', test)
['ab\r\nc', 'de\r\nf']
>>> [re.sub(r'\r?\n', ' ', x.strip()) for x in re.split(r'(?:\r?\n){2}', test)]
Traceback (most recent call last):
  File "<pudb command line>", line 1, in <module>
  File "<pudb command line>", line 1, in <listcomp>
NameError: name 're' is not defined
>>>

Program already does import re and I did it myself. No luck. Working on "python console" works as expected

This sounds much like https://github.com/inducer/pudb/issues/103 and may in fact be an easier way to reproduce this issue. Could you please add a comment there describing your issue?

Thanks! Andreas

---

Program already does import re and I did it myself [also]. No luck.

My usecase involves the Ctrl+X internal shell.

Working on "python console" works as expected

As noted the "normal python shell" works out of the box.

---

I am invoking pudb3 with python3 script

[asmeurer]

As-seen on the mailing list:

For context, this is with the builtin shell (but it's the same fundamental issue).

It looks like the SetPropogatingDict is explicitly disabled in the builtin shell because of https://github.com/inducer/pudb/issues/166. Can we use the code module for the builtin shell? We use that for the classic shell and it works just fine.

Regardless, I believe that issue only occurs in Python 2, so we should be able to use SetPropogatingDict in Python 3 to get the correct behavior there.

[stdedos]

I updated my above comment with info as-they-come.

For the sake of keeping the timeline, however, I'll say I am using pudb3 with python3 script

[asmeurer]

Maybe it works with the classic shell even in Python 2 because the code module uses exec instead of eval: https://github.com/python/cpython/blob/701af605df336c9e32751e9031266a2da60656c1/Lib/code.py#L103

[asmeurer]

I'm a little confused by https://stackoverflow.com/questions/12185110/subclassed-python-dictionary-for-custom-namespace-in-exec-method. As far as I can tell, eval and exec work with custom dictionaries just fine

# test.py 
class CustomDict(dict):
    def __setitem__(self, key, val):
        print('setting', key, val)
        return dict.__setitem__(self, key, val)

    def __getitem__(self, key):
        print('getting', key)
        return dict.__getitem__(self, key)

d = CustomDict()
eval(compile('a = 1', '<none>', 'single'), d)
print d['a']

d = CustomDict()
exec 'a = 1' in d
print d['a']

$ python2 test.py
('setting', 'a', 1)
('getting', 'a')
1
('setting', 'a', 1)
('getting', 'a')
1

But then again #166 was a very subtle bug, so maybe things break in nonobvious ways.

[asmeurer]

OK, I can confirm the issue now. It only comes up when accessing global variables from a local scope. It seems to happen with both eval and exec, so I don't understand how the classic shell works.

# test2.py

class CustomDict(dict):
    def __getitem__(self, key):
        if key == 'a':
            return 1
        return dict.__getitem__(self, key)


print 'doing eval'

try:
    d = CustomDict()
    eval(compile('def t():\n    print a', '<none>', 'single'), d, d)
    eval(compile('t()', '<none>', 'single'), d, d)
except Exception as e:
    print '!!! Exception:', e

print 'doing exec'

try:
    d = CustomDict()
    exec 'def t():\n    print a\n\nt()' in d
except Exception as e:
    print '!!! Exception:', e

$ python2 test2.py
doing eval
!!! Exception: global name 'a' is not defined
doing exec
!!! Exception: global name 'a' is not defined

# test3.py

class CustomDict(dict):
    def __getitem__(self, key):
        if key == 'a':
            return 1
        return dict.__getitem__(self, key)


print('doing eval')

try:
    d = CustomDict()
    eval(compile('def t():\n    print(a)', '<none>', 'single'), d, d)
    eval(compile('t()', '<none>', 'single'), d, d)
except Exception as e:
    print('!!! Exception:', e)

print('doing exec')

try:
    d = CustomDict()
    exec('def t():\n    print(a)\n\nt()', d, d)
except Exception as e:
    print('!!! Exception:', e)

$ python3 test3.py
doing eval
1
doing exec
1

[asmeurer]

Now I'm confused how the classic shell in Python 2 even is working

# test2.py

class CustomDict(dict):
    def __getitem__(self, key):
        if key == 'a':
            return 1
        return dict.__getitem__(self, key)


print 'doing eval'

try:
    d = CustomDict()
    eval(compile('def t():\n    print a', '<none>', 'single'), d, d)
    eval(compile('t()', '<none>', 'single'), d, d)
except Exception as e:
    print '!!! Exception:', e

print 'doing exec'

try:
    d = CustomDict()
    exec 'def t():\n    print a' in d
    exec 't()' in d
except Exception as e:
    print '!!! Exception:', e

print 'code module'
from code import InteractiveConsole
d = CustomDict()
cons = InteractiveConsole(d)
cons.push('def t():')
cons.push('    print a')
cons.push('')
cons.push('t()')

$python2 test2.py
doing eval
!!! Exception: global name 'a' is not defined
doing exec
!!! Exception: global name 'a' is not defined
code module
Traceback (most recent call last):
  File "<console>", line 1, in <module>
  File "<console>", line 2, in t
NameError: global name 'a' is not defined

# test3.py

class CustomDict(dict):
    def __getitem__(self, key):
        if key == 'a':
            return 1
        return dict.__getitem__(self, key)


print('doing eval')

try:
    d = CustomDict()
    eval(compile('def t():\n    print(a)', '<none>', 'single'), d, d)
    eval(compile('t()', '<none>', 'single'), d, d)
except Exception as e:
    print('!!! Exception:', e)

print('doing exec')

try:
    d = CustomDict()
    exec('def t():\n    print(a)\n\nt()', d, d)
except Exception as e:
    print('!!! Exception:', e)

print('code module')
from code import InteractiveConsole
d = CustomDict()
cons = InteractiveConsole(d)
cons.push('def t():')
cons.push('    print(a)')
cons.push('')
cons.push('t()')

python3 test3.py
doing eval
1
doing exec
1
code module
1

[asmeurer]

If I use instead

from pudb.shell import SetPropagatingDict
l = {}
g = {'a': 1}
d = SetPropagatingDict([l, g], l)

then all three work in Python 2. But as we saw in #166 the breakage was subtle. I wouldn't be surprised if the classic shell breaks things in Python 2 as well. Unless we can figure out exactly why things broke, I would suggesting leaving the code as is for Python 2 and using SetPropogatingDict for Python 3. Fewer people are using Python 2 anymore anyway. Actually using the code module for the builtin shell would clean up the code anyway, but it's not needed to fix this bug.

[asmeurer]

Here's a PR that fixes the builtin shell in Python 3 https://github.com/inducer/pudb/pull/330. The #166 issue still exists in Python 2 as far as I know, so we have to leave it broken there.

[stdedos]

Please add those (and mine and whatever else) as unit tests.

The issue seems very delicate

[inducer]

Please add those (and mine and whatever else) as unit tests.

I agree that the issue seems delicate. Would you be willing to contribute some tests?

[asmeurer]

I'll have to take a look to see how hard it is to test. The issue only occurs when the debugger is in a local function scope. Also, list comprehensions work just fine in Python 2 because they don't create their own scope in Python 2. But a lambda as in https://github.com/inducer/pudb/issues/103#issuecomment-468284870 will always reproduce the problem, if it exists.

To clarify, the problem is that variables defined in the console (variables already defined in the file being debugged work just fine) cannot be used inside of something in the console that creates a new function scope, such as a comprehension or a lambda (or a def, but those don't even work in the built in console, which is a separate issue).

[asmeurer]

There don't seem to be any existing tests for the debugger to model off of. Do you know how to create a Debugger to manipulate programmatically? Do we have to use pexpect, or somehow mock everything that manipulates stdout?

Maybe it would be easier to test if the built in shell were factored out from the debugger. I do want to refactor it a bit at some point, to clean up some issues with it (like the def issue I mentioned above), but it probably won't be soon.

[stdedos]

The issue only occurs when the debugger is in a local function scope.

I understand that; however, for the NameError: name 're' is not defined doesn't exactly sound like it, since:

• Instead of test, I tried to use a variable in the debugged program. It had the same outcome.
• import re was called both from the debugged program and me (i.e. Ctrl+X pudb console)
• NameError: name 're' is not defined does not suggest a missing variable (it doesn't say test is not defined), unless:
  • Not imported packages are treated like variables
  • test missing somehow makes the command line to be parsed poorly, hence leading to the aforementioned error

>>> test = 'ab\r\nc\r\n\r\nde\r\nf'
>>> re.split(r'(?:\r?\n){2}', test)
['ab\r\nc', 'de\r\nf']
>>> [re.sub(r'\r?\n', ' ', x.strip()) for x in re.split(r'(?:\r?\n){2}', test)]
Traceback (most recent call last):
  File "<pudb command line>", line 1, in <module>
  File "<pudb command line>", line 1, in <listcomp>
NameError: name 're' is not defined
>>>

---

@inducer I don't object contributing tests. Adding, however, as constrains:

• Executable tests
• Using some kind of Travis/Coveralls hook (or Travis and Coveralls maybe)
• Not being familiar with the code base
• Amateur-to-intermediate Python user

My only idea would be to go down on the debugger/shell, obtain "the evaluator object", and then feed it data, but that may not be optimal or desirable. There aren't may "enforcable" boundaries in Python, so I can eventually reference any part of the program - but maybe that's not a nice way to test.

I will first try locally to see if the commit I pulled actually fixes my problem. Then, I will try to provide something that looks like testing it (at least my 2 lines), but I cannot promise that "the reviewer" will like it.

As @asmeurer has also noted, doing tests that also have/mock having a loaded program (to execute testing from that angle too), it might get even more complicated (i.e. going the pexpect way). I have no idea how can you can debug a Curses application anyway (I have barely written one).

[asmeurer]

Can you show a reduced version of the file you are debugging where the error occurs?