NEW: Better package splitting for multi-output recipes with negative glob in outputs/files

[carterbox]

Description

Ressurects #4197. Closes #4196.

The purpose of this PR is to make it easier to split packages into multiple outputs using glob expressions. This is accomplished in two ways:

1. A negative pattern match is provided. This makes it easier to use fewer glob expressions if you need to include an entire directory tree except for a single file (type). For example, you might want to glob lib/**/libfoo*, but not lib/**/*.a
2. Only the files installed to the PREFIX in the top-level build are considered. This removes the need to craft your glob expressions to avoid the artifacts installed to the PREFIX by host dependencies. For example, you can now glob include/**/* without pulling in the headers of other packages. It is already the behavior of a single-output recipe to ignore files added to the prefix by host dependencies, so I'm not sure why this feature didn't make it into multi-output recipes.

These new behaviors are only used if the include/exclude keywords are used under the files key. The previous behavior is retained.

outputs:
  - name: foo
    files:
      - bin/*  # old behavior; includes artifacts from other packages
  - name: bar
    files:
      include:
        - bin/*  # new behavior; only matches artifacts from this recipe
      exclude:  # optional
        - bin/*.exe # new behavior
  - name: zee
    script: install.py  # old behavior

Checklist - did you ...

• [x] Add a file to the news directory (using the template) for the next release's release notes?
• [x] Add / update necessary tests?
• [x] Add / update outdated documentation?

[conda-bot]

We require contributors to sign our Contributor License Agreement and we don't have one on file for @carterbox.

In order for us to review and merge your code, please e-sign the Contributor License Agreement PDF. We then need to manually verify your signature, merge the PR (https://github.com/conda/infrastructure/pull/891), and ping the bot to refresh the PR.

[codspeed-hq[bot]]

CodSpeed Performance Report

Merging #5216 will not alter performance

_{Comparing carterbox:files-exclude (c92f1a1) with main (cdca0b4)}

Summary

✅ 3 untouched benchmarks

[carterbox]

I'm not sure that the failure for linux 3.8 23.5.0 serial is related to this PR.

----------------------------- Captured stderr call -----------------------------
Traceback (most recent call last):
  File "/usr/share/miniconda/envs/test/lib/python3.8/site-packages/conda/exception_handler.py", line 16, in __call__
  File "/usr/share/miniconda/envs/test/lib/python3.8/site-packages/conda/cli/main.py", line 66, in main_subshell
  File "/usr/share/miniconda/envs/test/lib/python3.8/site-packages/conda/cli/conda_argparse.py", line 31, in <module>
  File "/usr/share/miniconda/envs/test/lib/python3.8/site-packages/conda/base/context.py", line 24, in <module>
ModuleNotFoundError: No module named 'conda._vendor.appdirs'

[jaimergp]

Traceback (most recent call last):
  File "/private/var/folders/24/8k48jl6d249_n_qfxwsl6xvm0000gn/T/pytest-of-runner/pytest-0/popen-gw0/test_subpackage_recipes_copyin0/split_packages_file_list_1713435205299/test_tmp/run_test.py", line 51, in <module>
    assert os.path.isfile(filename)
AssertionError
WARNING: Tests failed for my_script_subpackage_files-1.0-0.tar.bz2 - moving package to /private/var/folders/24/8k48jl6d249_n_qfxwsl6xvm0000gn/T/pytest-of-runner/pytest-0/popen-gw0/test_subpackage_recipes_copyin0/broken

:(

[carterbox]

What is the difference between the parallel and serial tests? Only the macos parallel tests are failing. It looks like it only changes parameters related to pytest and codecov, not conda itself?

[jaimergp]

Serial jobs run tests that can't be run concurrently. Parallel jobs take rhe rest and run them with two processes wt the same time. The tests need to be marked as serial with a decorator to be considered as such, so the default is parallel.

[carterbox]

I think the errors are coming from the fact that the subpackage tests use the same artifacts for both outputs, so the subpackages are colliding when they are installed to the same environment?

[carterbox]

Needs approval to run the refactored tests.

[jaimergp]

tests/test_subpackages.py::test_subpackage_recipes[copying_files] is failing on macOS and Windows @carterbox

[wolfv]

@carterbox I've started to implement this idea in rattler-build, too: https://github.com/prefix-dev/rattler-build/pull/819

Would love your feedback once it's in a testable state. This is also in preparation of a "top-level" cache build (that can be split up into multiple packages).

[carterbox]

Thanks, @isuruf for fixing my platform detection logic in the tests.

@jaimergp, Looks like the failed test on Linux is in a test unrelated to this PR. Maybe someone can restart the tests to see if it is ephemeral or not?

[jaimergp]

Thanks @carterbox and everyone involved! Three months and 64 comments later, this one is in!