Tab completion ignores __all__

[rhattersley]

The Python docs describe "public names" as being defined by __all__ where it exists. It would be helpful if this rule was respected by the Python kernel to avoid module authors that want to provide helpful tab-completion having to obfuscate imports, etc. with leading underscores. NB. This use case is explicitly recognised in the Python docs:

... to avoid accidentally exporting items that are not part of the API (such as library modules which were imported and used within the module).

WIth the current kernel behaviour, code such as:

from __future__ import (division, print_function)
from six.moves import (filter, map)

import contextlib

__all__ = ['my_api']

def my_api():
    ...

needs to become:

from __future__ import (division as _, print_function as _)
from six.moves import (filter as _filter, map as _map)

import contextlib as _contextlib

__all__ = ['my_api']

def my_api():
    ...

[takluyver]

It's configurable, with the option IPCompleter.limit_to__all__ (config options docs).

The reason we do this is that we often want to use IPython to poke about and debug modules, so we like to see what's really there, not just what's whitelisted as public API. We should, however, do a better job of prioritising completions, so that things in __all__ come before things that aren't in it.

[rhattersley]

Thanks for the documentation link. :+1:

we like to see what's really there, not just what's whitelisted as public API

I guess the question here is, who is "we"? I suspect most users are just getting on with using existing modules and not poking about debugging modules. That's what I've seen in workshops, user drop-ins, etc. Also, the ability to change the configuration is probably something a "developer" will be more aware of & happier doing. So perhaps a change of the default is in order?

[rhattersley]

(Perhaps a different key combination could be used to allow an "enhanced" tab completion that shows more attributes, e.g. Ctrl+TAB)

[takluyver]

I'm not sure we can detect Ctrl+tab as a separate key combo in the terminal, and it would require changes to the message protocol for the Jupyter frontends - which is possible, but makes it quite a significant change.

I think resolving the priority is a sensible compromise, so that we show the names in __all__ before those that aren't.

[rhattersley]

So changing the default isn't an option?

[takluyver]

It's definitely an option, just not my preferred one. ;-) Let's see if anyone else wants to weigh in.

[ludwigschwardt]

I tend to agree with @rhattersley.

For me tab completion is not just for poking about and debugging; it is my main point of entry for any module on the command line.

I've tried to restrict my own packages to return only their main API during tab completion but it leads to ugly and brittle code like this:

# Hide submodules in module namespace, to avoid confusion with corresponding class names
# If the module is reloaded, this will fail - ignore the resulting NameError
try:
    _target, _antenna, _timestamp, _flux, _catalogue, _ephem_extra, \
      _conversion, _projection, _pointing, _refraction, _delay = \
      target, antenna, timestamp, flux, catalogue, ephem_extra, \
      conversion, projection, pointing, refraction, delay
    del target, antenna, timestamp, flux, catalogue, ephem_extra, \
        conversion, projection, pointing, refraction, delay
except NameError:
    pass

(I have a class Target in the submodule target etc. which might confuse users of the package if both appear in the tab completion list)

[shoyer]

I noticed that IPCompleter.limit_to__all__ is listed as deprecated in the docs now: http://ipython.readthedocs.io/en/stable/config/options/terminal.html

Is there another supported way to do this?

(For what it's worth, I have also found this to be a minor pain point. Often users explore APIs using IPython, and this adds significant noise.)

[aiguofer]

I'm running into this issue as well. I'm creating some modules for our team to use, and we rely on IPython/Jupyter pretty heavily. I'm writing my modules in such a way that they're easily explored interactively, and having to 'hide' imported functions and variables with _ is a bit ugly (and annoying).

[rowanc1]

One of our developers/scientists just ran into this issue as well, and has been hiding, among other things, import numpy as _np so as not to expose this to our scientific user base who are not interested in the debugging so much as exploring, interacting and using code.

With the current defaults it seems that designing for interactivity and usability at some level is promoting non standard coding practices, as others have noticed. Echoing @aiguofer's sentiment here.

More discussion here: https://github.com/simpeg/simpeg/pull/611#discussion_r120962083

[shoyer]

TensorFlow goes to some heroics to delete undocumented variables from modules to work around this same issue.

[Carreau]

sigh . I'll see what I can do to re-enable limit-to-all. The issue is that regardless of the default about 50% of our users will be unhappy

The completer is currently getting (some) refactor – showing types, function signature... etc. Would a "This function/attribute is private" ui indicator, and sorting these further down in the completion list be an acceptable middle ground ?

[hardkrash]

Here is a thought, could the auto completer recognize a triple tab?

\t is complete if single completion \t\t is show all list \t\t\t is show everything (either with or without _ is a question to consider)

[DiegoAlbertoTorres]

If ipython finds packages through dir(), Python 3.7 will allow overriding dir for a module (https://www.python.org/dev/peps/pep-0562/), thus giving back power to developers to control what is visible through tab-completion.

[shoyer]

If ipython finds packages through dir(), Python 3.7 will allow overriding dir for a module (https://www.python.org/dev/peps/pep-0562/), thus giving back power to developers to control what is visible through tab-completion.

This looks like a great solution -- does IPython use dir()?

[Carreau]

We have some custom logic and also fallback on Jedi.... we really need to get function to push tab-completiosn.

[pilattebe]

So there is the option IPCompleter.limit_to__all__, why isn't it the other way around ? There should be an option to allow autocomplete for privates but it shouldn't be the default. It the mean time, some people are doing horrible things.

[lheagy]

:wave: Just checking in here, are there any updates on what the preferred way to expose an API that facilitates code-exploration through tab-completion?

P.S. I am happy to help with a pr if someone is willing to give me a few pointers :)

[Stannislav]

So there is the option IPCompleter.limit_to__all__, why isn't it the other way around ? There should be an option to allow autocomplete for privates but it shouldn't be the default. It the mean time, some people are doing horrible things.

Totally agree, limit_to__all__ = True should in my opinion be the default option. Developers who want to expose non-public variables can either change that option or just call dir manually.

[BrenBarn]

I agree with the other comments that limiting to __all__ should be the default behavior. It's great for IPython to be able to list hidden names to allow under-the-hood poking and tweaking, but that doesn't make sense as a default behavior. Most users are just using libraries and the default should support using libraries as they were intended to be used, which means hiding things not in __all__. Showing such hidden names should be an advanced option.

[idantene]

Over five years later and this is still an issue (aka - "bump"). EDIT: This seems to be rooted with jedi more than Ipython. Using %config IPCompleter.use_jedi = False actually clears this up.

[joelostblom]

If ipython finds packages through dir(), Python 3.7 will allow overriding dir for a module (https://www.python.org/dev/peps/pep-0562/), thus giving back power to developers to control what is visible through tab-completion.

We have some custom logic and also fallback on Jedi.... we really need to get function to push tab-completiosn.

@Carreau It seems to me that overriding __dir__ works for changing the tab completion while still allowing access to the names that are removed from tab completion if they are typed out manually. Could this behavior be relied upon or is there something in the custom logic you mentioned that might make this unstable (if that logic is still in use since you made the comment two years ago)?

# Inside __init__.py
import sys


__all__ = [x for x in dir(sys.modules[__name__]) if x not in ['deprecated_module1', 'deprecated_module2']]

def __dir__():
    return sorted(__all__)