setuptools icon indicating copy to clipboard operation
setuptools copied to clipboard
verified publisher icon pypa

Users expect `[options.packages.find] exclude` to be recursive, but it is not (adding `.*` to the pattern is needed)

Open KOLANICH opened this issue 4 years ago • 6 comments

setuptools version

57.0.1.dev3+ga5131f0b.post.20210528a

Python version

cpython 3.9

OS

Ubuntu 21.04

Additional environment information

No response

Description

[options.packages.find]
exclude =
    arpeggio.tests

doesn't work. In fact it doesn't work in a lot of other projects. But in other projects there usually no more than 2 levels in the excluded dir, so the workaround tests and tests.* works.

Expected behavior

Should exclude the whole hierarchy if a dir is mentioned.

How to Reproduce

https://github.com/textX/Arpeggio/tree/f24ceb0e79f6c9216dc035de123a388d113a332e

Output

....
adding 'arpeggio/tests/__init__.py'
adding 'arpeggio/tests/test_decorator_combine.py'
adding 'arpeggio/tests/test_default_semantic_action.py'
adding 'arpeggio/tests/test_eolterm.py'
adding 'arpeggio/tests/test_error_reporting.py'
adding 'arpeggio/tests/test_examples.py'
....

Code of Conduct

  • [X] I agree to follow the PSF Code of Conduct

KOLANICH avatar May 28 '21 14:05 KOLANICH

I'm having a very similar issue: When running pytproject-build, the resulting sdist has the tests folder included even though it is explicitly excluded using the exclude=tests,test* config. Interestingly, the .whl file does not include the tests.

As I understand, pyproject-build first creates an sdist, and then creates the wheel from that sdist. So the result surprises me.

Example contents of the sdists (notice that it includes the test-files):

vscode ➜ /workspaces/puresnmp (develop ✗) $ tar tzvf dist/puresnmp-1.10.2.post1.tar.gz 
drwxr-xr-x vscode/vscode     0 2021-07-11 12:16 puresnmp-1.10.2.post1/
-rw-r--r-- vscode/vscode  1069 2021-07-11 11:28 puresnmp-1.10.2.post1/LICENSE.txt
-rw-r--r-- vscode/vscode    90 2021-07-11 11:28 puresnmp-1.10.2.post1/MANIFEST.in
-rw-r--r-- vscode/vscode  3240 2021-07-11 12:16 puresnmp-1.10.2.post1/PKG-INFO
-rw-r--r-- vscode/vscode  2280 2021-07-11 11:29 puresnmp-1.10.2.post1/README.rst
drwxr-xr-x vscode/vscode     0 2021-07-11 12:16 puresnmp-1.10.2.post1/puresnmp/
-rw-r--r-- vscode/vscode   708 2021-07-11 11:39 puresnmp-1.10.2.post1/puresnmp/__init__.py
drwxr-xr-x vscode/vscode     0 2021-07-11 12:16 puresnmp-1.10.2.post1/puresnmp/aio/
[...]
drwxr-xr-x vscode/vscode     0 2021-07-11 12:16 puresnmp-1.10.2.post1/tests/
-rw-r--r-- vscode/vscode  7886 2021-07-11 11:29 puresnmp-1.10.2.post1/tests/__init__.py
-rw-r--r-- vscode/vscode   442 2021-07-11 11:29 puresnmp-1.10.2.post1/tests/asyncmock.py
-rw-r--r-- vscode/vscode   115 2021-07-11 11:29 puresnmp-1.10.2.post1/tests/conftest.py
drwxr-xr-x vscode/vscode     0 2021-07-11 12:16 puresnmp-1.10.2.post1/tests/external_api/
-rw-r--r-- vscode/vscode     0 2021-07-11 11:29 puresnmp-1.10.2.post1/tests/external_api/__init__.py
-rw-r--r-- vscode/vscode 16904 2021-07-11 11:29 puresnmp-1.10.2.post1/tests/external_api/test_v1.py
[...]

Contents of the wheel (no test files):

vscode ➜ /workspaces/puresnmp (develop ✗) $ unzip -l dist/puresnmp-1.10.2.post1-py3-none-any.whl 
Archive:  dist/puresnmp-1.10.2.post1-py3-none-any.whl
  Length      Date    Time    Name
---------  ---------- -----   ----
      708  2021-07-11 11:39   puresnmp/__init__.py
      733  2021-07-11 11:29   puresnmp/const.py
     5438  2021-07-11 11:29   puresnmp/exc.py
    10450  2021-07-11 11:29   puresnmp/pdu.py
        0  2021-07-11 11:28   puresnmp/py.typed
     2353  2021-07-11 11:29   puresnmp/snmp.py
     5166  2021-07-11 11:29   puresnmp/transport.py
     5195  2021-07-11 11:29   puresnmp/types.py
      764  2021-07-11 11:29   puresnmp/typevars.py
     7046  2021-07-11 11:29   puresnmp/util.py
        0  2021-07-11 11:28   puresnmp/aio/__init__.py
     4461  2021-07-11 11:29   puresnmp/aio/transport.py
        0  2021-07-11 11:28   puresnmp/aio/api/__init__.py
     9106  2021-07-11 11:29   puresnmp/aio/api/pythonic.py
    22491  2021-07-11 11:29   puresnmp/aio/api/raw.py
        0  2021-07-11 11:28   puresnmp/api/__init__.py
    11618  2021-07-11 11:30   puresnmp/api/pythonic.py
    23267  2021-07-11 11:30   puresnmp/api/raw.py
     1069  2021-07-11 12:16   puresnmp-1.10.2.post1.dist-info/LICENSE.txt
     3570  2021-07-11 12:16   puresnmp-1.10.2.post1.dist-info/METADATA
       92  2021-07-11 12:16   puresnmp-1.10.2.post1.dist-info/WHEEL
        9  2021-07-11 12:16   puresnmp-1.10.2.post1.dist-info/top_level.txt
     1813  2021-07-11 12:16   puresnmp-1.10.2.post1.dist-info/RECORD
---------                     -------
   115349                     23 files

setup.cfg

vscode ➜ /workspaces/puresnmp (develop ✗) $ cat setup.cfg 
[metadata]
name = puresnmp
project_urls =
    Homepage = https://github.com/exhuma/puresnmp
; [...]

[options]
include_package_data = True
packages = find:
install_requires =
    t61codec >= 1.0, <2.0
    x690 == 0.2
    dataclasses; python_version < "3.7"

[options.packages.find]
exclude =
    tests
    tests.*

exhuma avatar Jul 11 '21 12:07 exhuma

Hi @KOLANICH, thank you very much for bringing this topic for discussion.

The current implementation specifically suggests that this behaviour is intentional, although I don't know what is the rationale behind it. So I will re-label this from bug to enhancement.

(I can conjecture that if the exclusion is automatically recursive there is no way of "re-including" a nested package, so in some sense the current implementation is more general/flexible than the proposal - not sure how relevant this rationale is).

I don't know what would be the impacts of changing this behaviour (it would definitely not be backwards compatible), so I will leave the assessment of the proposal for the other maintainers.

abravalheri avatar Feb 24 '22 15:02 abravalheri

@abravalheri nowadays and I am still facing this same issue.

project structure:

╰─ ls -R --ignore="venv" --ignore="__pycache__" --ignore="build" --ignore="dist" --ignore="*egg-info" 
.:
pyproject.toml  README.md  src  tests  VERSION

./src:
mproject

./src/mproject:
goal.py  __init__.py

./tests:
__init__.py  test_keep.py

pyproject.toml content:

[build-system]
requires = ["setuptools >= 67.2.0"]
build-backend = "setuptools.build_meta"

[project]
name = "mproject"
authors = [{name = "Pablo Emidio S.S", email = "[email protected]"}]
description = "A test project structure"
readme = "README.md"
requires-python = ">=3.9"
keywords = ["One", "Two"]
license = {text = "MIT"}
dependencies = [
	'importlib-metadata; python_version<"3.10"'
]
dynamic = ["version"]

[tool.setuptools]
include-package-data = false

[tool.setuptools.packages.find]
where = ["src"]
include = ["mproject"]
exclude = ["tests", "tests.*"]

[tool.setuptools.dynamic]
version = {file = "VERSION"}

whel content:

╰─ unzip -l dist/mproject-0.0.1.dev0-py3-none-any.whl
Archive:  dist/mproject-0.0.1.dev0-py3-none-any.whl
  Length      Date    Time    Name
---------  ---------- -----   ----
      175  02-14-2023 00:59   mproject/__init__.py
       25  02-14-2023 01:00   mproject/goal.py
      320  02-14-2023 01:39   mproject-0.0.1.dev0.dist-info/METADATA
       92  02-14-2023 01:39   mproject-0.0.1.dev0.dist-info/WHEEL
        9  02-14-2023 01:39   mproject-0.0.1.dev0.dist-info/top_level.txt
      467  02-14-2023 01:39   mproject-0.0.1.dev0.dist-info/RECORD
---------                     -------
     1088                     6 files

tar content:

╰─ tar -tvf dist/mproject-0.0.1.dev0.tar.gz
drwxr-xr-x pabloemidio/pabloemidio     0 2023-02-13 22:39 mproject-0.0.1.dev0/
-rw-r--r-- pabloemidio/pabloemidio   260 2023-02-13 22:39 mproject-0.0.1.dev0/PKG-INFO
-rw-r--r-- pabloemidio/pabloemidio    17 2023-02-13 21:47 mproject-0.0.1.dev0/README.md
-rw-r--r-- pabloemidio/pabloemidio    10 2023-02-13 22:02 mproject-0.0.1.dev0/VERSION
-rw-r--r-- pabloemidio/pabloemidio   626 2023-02-13 22:28 mproject-0.0.1.dev0/pyproject.toml
-rw-r--r-- pabloemidio/pabloemidio    38 2023-02-13 22:39 mproject-0.0.1.dev0/setup.cfg
drwxr-xr-x pabloemidio/pabloemidio     0 2023-02-13 22:39 mproject-0.0.1.dev0/src/
drwxr-xr-x pabloemidio/pabloemidio     0 2023-02-13 22:39 mproject-0.0.1.dev0/src/mproject/
-rw-r--r-- pabloemidio/pabloemidio   175 2023-02-13 21:59 mproject-0.0.1.dev0/src/mproject/__init__.py
-rw-r--r-- pabloemidio/pabloemidio    25 2023-02-13 22:00 mproject-0.0.1.dev0/src/mproject/goal.py
drwxr-xr-x pabloemidio/pabloemidio     0 2023-02-13 22:39 mproject-0.0.1.dev0/src/mproject.egg-info/
-rw-r--r-- pabloemidio/pabloemidio   260 2023-02-13 22:39 mproject-0.0.1.dev0/src/mproject.egg-info/PKG-INFO
-rw-r--r-- pabloemidio/pabloemidio   276 2023-02-13 22:39 mproject-0.0.1.dev0/src/mproject.egg-info/SOURCES.txt
-rw-r--r-- pabloemidio/pabloemidio     1 2023-02-13 22:39 mproject-0.0.1.dev0/src/mproject.egg-info/dependency_links.txt
-rw-r--r-- pabloemidio/pabloemidio    47 2023-02-13 22:39 mproject-0.0.1.dev0/src/mproject.egg-info/requires.txt
-rw-r--r-- pabloemidio/pabloemidio     9 2023-02-13 22:39 mproject-0.0.1.dev0/src/mproject.egg-info/top_level.txt
drwxr-xr-x pabloemidio/pabloemidio     0 2023-02-13 22:39 mproject-0.0.1.dev0/tests/
-rw-r--r-- pabloemidio/pabloemidio    35 2023-02-13 22:02 mproject-0.0.1.dev0/tests/test_keep.py

package has being published in https://test.pypi.org/project/mproject/ to maybe help debbuging proposes.

PabloEmidio avatar Feb 14 '23 02:02 PabloEmidio

I wasted a day and night fighting with this issue. Having this in pyproject.toml:

[tool.setuptools.packages.find]
include = ['bug_demo*']
exclude = ['bug_demo.subpack', 'bug_demo.subpack.*', 'bug_demo.subpack*']
namespaces = false

bug_demo.subpack (and everything under it) is still included in distribution and installation from folder using pip install .

The core of the evil is *.egg-info - it is not (properly?) updated on fresh build or installation. So if there is *.egg-info from the previous build/installation without exclude-option, it makes you hair off. Killing *.egg-info folder before build/install solves the problem.

valq7711 avatar Apr 01 '23 11:04 valq7711

I'm having a very similar issue: When running pytproject-build, the resulting sdist has the tests folder included even though it is explicitly excluded using the exclude=tests,test* config. Interestingly, the .whl file does not include the tests.

This behavior is by design. The package inference happens during a build phase, but sdist happens earlier and is more abstract than the build phase. See this comment for a detailed explanation.

nowadays and I am still facing this same issue.

Same as the prior comment. The wheel looks fine and is excluding the packages from the wheel. You should expect source files (even tests) to be included in the sdist. To affect the sdist, you'll want to customize the MANIFEST.in.

The core of the evil is *.egg-info - it is not (properly?) updated on fresh build or installation. So if there is *.egg-info from the previous build/installation without exclude-option, it makes you hair off. Killing *.egg-info folder before build/install solves the problem.

This behavior is inconsistent with the reported issue. The reported issue is that users shouldn't have to specify both bug_demo.subpack and bug_demo.subpack.* to exclude it. The issue you encountered was with a stale egg-info, which seems to be a separate issue. https://github.com/pypa/setuptools/issues/1347 seems to be related, though not precisely. I searched around and didn't find anything better, though.

I'm marking all of these comments as off-topic, as they're distracting from the primary report.

jaraco avatar Jan 07 '24 17:01 jaraco

I encountered this issue recently when working with distutils. I was trying to figure out why the distutils package was not being installed. It turns out it was because I'd included a find.exclude=['dist*'] directive in an attempt to exclude the dist directory, but that also excluded the distutils directory.

As it turns out, that directive wasn't necessary, so I removed it, but while troubleshooting, I noticed I was also excluding docs* and tests*, which are necessary. Unfortunately, these exclusions aren't having the intended effect (exclude the docs/ and tests/ directories and everything beneath them as packages. Instead, what's necessary is to exclude=['docs', 'docs.*', 'tests', 'tests.*'], which is rather clumsy. The user should be able to exclude a directory in one declaration and not two.

Like abravalheri, I'm trying to imagine in what scenario one would want to exclude the top-level package, but include the child package. That seems unlikely. My instinct is just to drop all nested packages and see if it causes any disruption and if so, roll it back and deprecate the behavior to identify affected use-cases.

jaraco avatar Jan 07 '24 17:01 jaraco