Running pylint on namespace modules results in import-error

[gpakosz]

Here's a simple repro

(tmp) bash-3.2$ tree
.
├── foo
│   ├── __init__.py
│   └── bar.py
└── tools
    └── foo
        └── foo.py

3 directories, 3 files

(tmp) bash-3.2$ cat foo/__init__.py
"""foo lib"""

(tmp) bash-3.2$ cat foo/bar.py
"""foo.bar module"""

def baz():
    print("baz")

(tmp) bash-3.2$ cat tools/foo/foo.py
"""foo tool"""
import foo.bar

foo.bar.baz()

When I launch the code with python -m, the code runs just fine.

(tmp) bash-3.2$ python -m tools.foo.foo
baz

However, when running pylint on a single file, I'm hitting E0611. My real goal is to launch pylint on modified file from a Git pre-commit hook. I tried using --init-hook and manipulate sys.path without success.

(tmp) bash-3.2$ pylint ./tools/foo/foo.py
************* Module foo
tools/foo/foo.py:1:0: C0102: Black listed name "foo" (blacklisted-name)
tools/foo/foo.py:2:0: E0611: No name 'bar' in module 'foo' (no-name-in-module)
tools/foo/foo.py:2:0: E0401: Unable to import 'foo.bar' (import-error)
tools/foo/foo.py:4:0: E1101: Module 'foo' has no 'bar' member (no-member)

-----------------------------------------------------------------------
Your code has been rated at -70.00/10 (previous run: -46.67/10, -23.33)

Thanks for the help

[PCManticore]

@gpakosz Thanks for reporting an issue. I believe this is an issue related to namespace packages. In Python 3, you can have directories without __init__ that you can import, just like in your example. We technically should support that in pylint, but we've been having various edge cases around the implementation that leads to the error you see. If you'd try with __init__ files in your tools directory, you'll notice that this works as intended, but ideally it should work without __init__ files on Python 3.

[gpakosz]

Thanks for the reply @PCManticore,

FYI in this specific repro, I had to create both tools/__init__.py and tools/foo/__init__.py to make pylint happy.

[thormick]

I seem to be running into this issue too, with pylint 2.5.2. A test-case based on the structure of the project I've been assigned to:

$ tree
.
├── __init__.py
├── main.py
└── one
    ├── foo.py
    └── __init__.py

1 directory, 4 files
$ cat main.py 
#!/usr/bin/env python3
"""
main.py
"""

from one.foo import MyClass

foo = MyClass()

print(foo.do_foo())
$ cat one/foo.py 
#!/usr/bin/env python3
"""
Test lib
"""


class MyClass:

    def do_foo(self):
        return self.__class__

main.py runs fine:

$ ./main.py 
<class 'one.foo.MyClass'>

Even if pylint complains:

$ pylint main.py 
************* Module main
main.py:6:0: E0401: Unable to import 'one.foo' (import-error)

---------------------------------------------------------------------
Your code has been rated at -6.67/10 (previous run: 10.00/10, -16.67)

Adding the current directory to PYTHONPATH or sys.path, or removing the outer __init__.py fixes it:

$ pylint --init-hook="sys.path.append('.')" main.py 

---------------------------------------------------------------------
Your code has been rated at 10.00/10 (previous run: -6.67/10, +16.67)

$ export PYTHONPATH=.
$ pylint main.py 

--------------------------------------------------------------------
Your code has been rated at 10.00/10 (previous run: 10.00/10, +0.00)

$ export PYTHONPATH=
$ pylint main.py 
************* Module main
main.py:6:0: E0401: Unable to import 'one.foo' (import-error)

---------------------------------------------------------------------
Your code has been rated at -6.67/10 (previous run: 10.00/10, -16.67)

$ mv __init__.py foo__init__.py 
$ pylint main.py 

---------------------------------------------------------------------
Your code has been rated at 10.00/10 (previous run: -6.67/10, +16.67)

$ mv foo__init__.py __init__.py 

The presence or absence of __init__.py in one doesn't seem to affect this.

2.4.4 doesn't exhibit the same behavior.

$ pylint --version
pylint 2.5.2
astroid 2.4.1
Python 3.8.2 (default, Apr 27 2020, 15:53:34) 
[GCC 9.3.0]
$ pip install pylint==2.4.4
... snip ...
Successfully installed astroid-2.3.3 pylint-2.4.4
$ pylint main.py 
************* Module main
main.py:8:0: C0102: Black listed name "foo" (blacklisted-name)

-------------------------------------------------------------------
Your code has been rated at 6.67/10 (previous run: 10.00/10, -3.33)

[scottschreckengaust]

Please bring back support for native namespace packaging. Is there a work-around other than keeping pylint less than version 2.5.x?

[domibarton]

@gpakosz Thanks for reporting an issue. I believe this is an issue related to namespace packages. In Python 3, you can have directories without __init__ that you can import, just like in your example. We technically should support that in pylint, but we've been having various edge cases around the implementation that leads to the error you see. If you'd try with __init__ files in your tools directory, you'll notice that this works as intended, but ideally it should work without __init__ files on Python 3.

Yes, you can add a new __init__.py to the namespace package and pylint might work again. But this can lead to even worse side effects. For example take the following use case:

• There is a 3rd party package called bar
• You create a new native namespace package called foo.bar (foo is the namespace)
• In bar you use import something from bar which imports something from the 3rd party bar package

This works as long as you don't add an __init__.py file to the foo namespace package. As soon as you add the __init__.py, the pylint will be happy but you'll end up with an ImportError, because now something is imported from (foo.)bar.

The __init__.py can't be empty (!!!), it must be a proper namespace package init file as stated in this comment.

[jacobtylerwalls]

A test for unittest_lint.py. Doesn't seem to be fixed on PyCQA/astroid@d45b33c72b33d719224efeb88d7b8e4f493717c7.

def test_lint_namespace_package(initialized_linter: PyLinter) -> None:
    linter = initialized_linter
    with tempdir():
        create_files(["namespace/__init__.py", "namespace/submodule.py"])
        create_files(["other/namespace/namespace.py"])
        with open(Path("namespace/submodule.py"), "w", encoding="utf-8") as f:
            f.write("""\"\"\"This is namespace.submodule\"\"\"
def noop():
    pass
""")
        with open(Path("other/namespace/example.py"), "w", encoding="utf-8") as f:
            f.write("""\"\"\"This module imports namespace.submodule\"\"\"
import namespace.submodule
namespace.submodule.noop()
""")
        linter.check(["other/namespace/example.py"])
    assert not linter.stats.by_msg

[jacobtylerwalls]

See #3984 for an even simpler test case. I'm optimistic but not certain that a fix would handle both cases.