Ipython qtconsole continues line instead of executing on return

[ekipmanager]

When I start a kernel, everything works fine but I always get to this weird mode when I am playing with a dataframe in pandas and I use tab completion on some of its index functions (e.g. df.index.get_lev<tab> which makes it df.index.get_level_values)*. Then tab completion stops working and hitting enter will generate a new line. Even if I erase the line and hit enter it will go to a new line. I either have to restart the kernel or hit <shift+enter> and then hitting <ctrl+c> multiple times so it comes out of this mode. It happens on both mac and linux and with both anaconda 3.19.0 and python 2.7. I have this issue when I run ipython qtconsole from command line, I don't have problem with this in jupyter notebook or when I just enter ipython in terminal.

(*) This sequence might be unrelated but this is how I reproduce the problem. I don't get to this problem if I type df.index.get_level_values(0) but if anywhere after typing index I use <tab> this happens. Also it doesn't happen when I use on dataframe methods like df.get_<tab>, nor does it happen if I use <tab> for something that is a index property not a method like df.index.names. It also does not happen with small dataframes, what I am using is more than 500 MB when pickled.

[takluyver]

Is your version of qtconsole up to date? (latest is 4.1.1; jupyter qtconsole --version to check)

Is there any error message in the terminal where you started the Qt console when this happens? If you're not running it from a terminal, can you reproduce it when you do run from a terminal?

I've just tried reproducing it on qtconsole master with Linux, and wasn't able to trigger the bug.

[ekipmanager]

@takluyver yes both versions are 4.1.1. on anaconda3 and python2.7 environments that I have. I run it from the terminal but no message shows up there. These are the two consoles I tried on Linux (Fedora 23):

Jupyter QtConsole 4.1.1 Python 3.5.1 |Anaconda 2.4.1 (64-bit)| (default, Dec 7 2015, 11:16:01) Type "copyright", "credits" or "license" for more information. IPython 4.0.3 -- An enhanced Interactive Python.

and

Jupyter QtConsole 4.1.1 Python 2.7.10 (default, Sep 8 2015, 17:20:17) Type "copyright", "credits" or "license" for more information. IPython 4.0.3 -- An enhanced Interactive Python.

Both behave in similar ways. The dataframe I use to reproduce this issue is around 1.1GB in pickled format and it is multi-indexed with 4 index levels and 15288969 rows in case that matters.

[takluyver]

Can you reproduce it with a small dataframe, or only with that large one? I think pandas does some custom completion stuff, so it may be getting stuck dealing with the completion request.

[ekipmanager]

I dropped 90% of the table but I still had the same problem, maybe it is the structure. I made it even a 4x10 dataframe and had the same problem. I don't have any problem if I reset the index, and I also don't have a problem if I use a trivial dataframe like in pandas examples for multi-index dfs, so the problem exists with only this multi-indexed dataframe.

[ekipmanager]

So I made another dataframe by transforming this to a dict and then to a df, and I don't have the problem with the new one. Also, when I enter df.index in ipython it freezes. I'm not sure if this is pandas bug or not but this ~~doesn't happen in ipython alone, it only happend in qtconsole~~ Correction: df.index.get_level_values(0) does not cause a problem in ipython when it runs in terminal or in the notebook, it only causes problem in qtconsole. BUT running df.index freezes ipython (terminal, notebook, jupyter). It is not like the problem above (it does not cause enters to insert a new line) it only makes the kernel busy until you interrupt it. If you let it run ipython qtconsole will close itself and you see Segmentation fault (core dumped) in the terminal.

[takluyver]

That sounds like pandas' attempt to compute the representation of its index is going wrong, or the representation it's trying to compute is too large.

[ekipmanager]

Yes something goes wrong between qtconsole and pandas.MultiIndex object when it is too large. It does not happen in terminal ipython (it freezes if you want to print the whole index which is understandable but I mean the new line thing doesn't happen). My solution for now is just to hit shift+enter and then break it with control+C the next time I run the command using history it works fine.

[takluyver]

I would guess something in the Qt console is making it try to calculate the repr, and that's blocking the kernel from doing anything else. If repr(df.index) is slow, I think you should file an issue on pandas.

[tmr232]

I'm experiencing a similar issues using ipyida https://github.com/eset/ipyida/pull/5 ipyida embeds the RichJupyterWidget in IDA. When using IPython 3.2.3 pressing Enter works as expected. But when I updated the code to use latest Jupyter I suddenly had to press Shift+Enter.

[ElieGouzien]

Hi, I've experienced similar bugg, without any pandas. Here is a minimal exemple :

class Bugger(object):
    def __init__(self, size=10000):
        """Create bigg data."""
        self.big_data = [list(range(i)) for i in range(size)]

    def easy_method(self):
        """Method that should be really fast."""
        return self.big_data[-1][-1]

    def __repr__(self):
        output = ""
        for nested in self.big_data:
            for elem in nested:
                output += str(elem)
            output += '\n'
        return output

test = Bugger()

Then test.ea<tab> seems to trigger the computation of __repr__() that blocks the kernel and then line continuation and evaluation need to wait until __repr__() finish before working again correctly. For me sounds like a bugg that the completion launch the computation of __repr__() but don't know the internals of qtconsole.

[ccordoba12]

@takluyver?

[takluyver]

When I try that, completing test.ea<tab> does not trigger repr, and does complete instantly. However, when I open the parentheses for the method call test.easy_method(, the Qt console sends an inspect request, which does call repr (as I'd expect it to).

Possibly IPython should try to limit how long repr can take, but that's hard to do with synchronous code. AFAIK, you either need to do it in a separate thread, which would probably cause other problems, or use the blunt instrument of SIGALRM.

[ElieGouzien]

For the completion

I've put pdb.set_trace() at the beginning of the __repr__() method and called the completion through get_ipython().complete('test.ea') . The result is that we end there because it's requested as part of a string for error message (that is actually captured latter). The raise is in the standard inspect module itself called by jedi.

My python version is 3.4.2 ; ipython 6.0.0 ; qtconsole 4.3.0 and jedi 0.10.2. I guess one of jedi or python has been updated and the issue is actually fixed for your version.

For the inspection

Ok you expect the inspect request to call repr. But since the result is never shown to the user I think it would be more natural not to compute it better than altering it's computation.

I've found get_ipython().inspector that can inspect specific things. No idea if you use it. But it would be better to just ask for the actually displayed informations and nothing more.

[takluyver]

Aha, it was me with an old version of jedi. Updating to 0.10.2, I see it as well.

[takluyver]

I'm not sure what to do about that; there's a clash between two uses of repr - one to display data, which may reasonably take a while, and one to identify an object for an error message, which is always expected to finish quickly and reliably. I don't think it's unreasonable for inspection/completion machinery to use functions like inspect.getsourcefile(), but calling that leads quite directly to this issue. Maybe we/jedi should do the equivalent checks beforehand and avoid calling getsourcefile if it's going to produce that error message.

Pinging @Carreau (I don't know if you watch this repo anyway)

[Carreau]

No i'm not watching. Thanks for the ping. I already have some code to handle if jedi takes too long to compute completions, this one will likely be tough. I'll see if I can do something in IPython or upstream.

[ElieGouzien]

For the completion

@takluyver You are true there is a clear clash between those two jobs of repr (by the way I never understood why it's not str used for displaying in interpreter).

After importing jedi, jedi.Interpreter('test.ea', [locals()]).completions() is sufficient to trigger repr through error in inspect.getsourcefile(). Then I guess this is a "bugg" for jedi.

On qtconsole side I think there should be a way of interrupting repr when we try to execute the current line even if jedi didn't answered yet. That's actually the behavior of standard IPython. Any link with #175 ?

For the call tip

Running get_ipython().inspector.str_detail_level = 1 seems to fix the thing.

Saying that I have no idea of where the inspector is actually initiated. I don't see any side effect but it obviously doesn't mean there isn't.

[takluyver]

I agree that it would make sense from a user perspective for the Qt console to interrupt slow introspection for more important tasks such as execution, but it's technically tricky because:

1. The Qt console does not currently track "what is my kernel doing at the moment" - it sends off messages, handles replies when they arrive, but doesn't know what the kernel is doing at any specific point. It probably could, but it would be more work than it might sound, especially as multiple frontends can talk to one kernel.
2. Even if it did, acting on the kernel based on that would be subject to race conditions - e.g. the frontend might decide to interrupt the kernel while the reply it's waiting for is already in flight, and end up interrupting the next task that the kernel is starting.

It's not linked to #175 - that's about blocking the Qt event loop in the frontend, whereas this is a kernel issue.

Running get_ipython().inspector.str_detail_level = 1 seems to fix the thing.

Yup - that effectively turns off a feature of the inspection machinery to show object reprs:

 2│ a?
Type:        list
String form: [0, 1, 2, 3, 4, 5, 6, 7, 8, 9]
Length:      10
Docstring:  
list() -> new empty list
list(iterable) -> new list initialized from iterable's items

 3│ get_ipython().inspector.str_detail_level = 1

 4│ a?
Type:      list
Length:    10
Docstring:
list() -> new empty list
list(iterable) -> new list initialized from iterable's items

We could consider doing that by default in the kernel, especially if we can resolve the completion issue.

I have no idea of where the inspector is actually initiated

There's a method IPython.core.interactiveshell.InteractiveShell.init_inspector() that does it. The value is actually configurable as InteractiveShell.object_info_string_level.

[ElieGouzien]

Completion

I've filled an issue in jedi's repository : Jedi issue #919.

Call tip

For the call tip another solution would be rather than changing the default inspector behavior just having the info method called with detail_level=-1. Otherwise the infomethod is marked as deprecated, maybe only the specific showed info should be requested.

[ElieGouzien]

The completion part is now fixed (in jedi dev branch, not yet released). For the call type part I tried to put a detail_level=-1 at the right place but just didn't find the right place. Someone to make that or tell me where I should look ?

PS : what is the _call_tip method for ? line 757 in frontend_widget.py. It seems to have no effect when I modify it.

[Carreau]

I believe _call_tip will show you a pop-up with information about the object before the cursor.

detail_level=-1

I believe details level is a boolean (0/1) which dates from wayyyy back, and may not support value outside of these two. You likely want to use pinfo or pinfo2 directly, but that should not matter from the qtconsole side as things like that should go through the protocol, so should be fixed either on IPython, or ipykernel side.

[ElieGouzien]

Now I understand better how things work (by the way my question about _call_tip was because I failed to properly isolate my dev version).

@Carreau You are completely true detail_level=-1 is a bad idea.

Now I see 2 ways of solving the issue :

• Setting something equivalent to get_ipython().inspector.str_detail_level = 1. But It would be nice to have a qtconsole call_tip specific config and not affecting the ? magic.
• Adding in the Jupyter messaging protocol the possibility to explicitly precise what exactly is requested by an inspect_request. This is obviously beyond this issue but would be nice.