--constraint support

[orsinium]

What's the problem this feature will solve?

There are 2 projects:

• lib is an internal library. It has setup.py with dependencies of the library.
• srv is an internal service that uses lib. It has a lock file requirements.txt.

The goal is to generate lib/requirements.txt which has only packages from lib/setup.py but the same versions as specified in srv/requirements.txt. Motivation:

• The same version is needed to make sure that if the tests have passed for lib, it will work as expected in the environment of srv.
• Running tests for lib in the environment of srv is complicated on CI: the env of srv is too big, and we want to avoid making it for pipelines in lib.

Describe the solution you'd like

The idea is the same as in pip's Constraints Files (-c/--constraint option). So, it makes sense to introduce the same key for pip-tools.

Alternative Solutions

pip-tools already uses the output file as the constraint. So, a workaround I found is to specify the constraint file as the output file and then restore it. The PoC:

import sys
import shutil
import subprocess
from argparse import ArgumentParser


def main():
    parser = ArgumentParser()
    parser.add_argument('-c', '--constraint', required=True)
    parser.add_argument('--output-file', default='requirements.txt')
    args, rest = parser.parse_known_args()
    shutil.copy(args.constraint, '/tmp/c.txt')
    cmd = [sys.executable, '-m', 'piptools', 'compile', '--output-file', args.constraint]
    try:
        code = subprocess.call(cmd + rest)
        shutil.copy(args.constraint, args.output_file)
    finally:
        shutil.copy('/tmp/c.txt', args.constraint)
    sys.exit(code)

if __name__ == '__main__':
    main()

Additional context

[orsinium]

I think the implementation can be pretty simple. We can just do the same for constraint file as we do for the output file for the purpose of detecting the current constraints.

        ireqs = parse_requirements(
            output_file.name,
            finder=tmp_repository.finder,
            session=tmp_repository.session,
            options=tmp_repository.options,
        )
        if constraint_file:
            ireqs = itertools.chain(ireqs, parse_requirements(
                constraint_file,
                finder=tmp_repository.finder,
                session=tmp_repository.session,
                options=tmp_repository.options,
            ))

[AndydeCleyre]

Alternative approach:

Create lib/requirements.in with something like:

.
-c ../srv/requirements.txt

Relative paths may be trouble until a certain PR gets approved, but you can use absolute paths to test the approach.

Anyway, compile that to lib/requirements.txt

[orsinium]

That's an interesting idea, thank you. I finally tried it today, it's close enough. However, the . itself is also added as a dependency which is not desired:

# WARNING: pip install will require the following package to be hashed.
# Consider using a hashable URL like https://github.com/jazzband/pip-tools/archive/SOMECOMMIT.zip
file:///home/gram/Documents/lib
    # via -r requirements.in

[orsinium]

IDK why I didn't think about it, but apparently you can pass . as another argument: pip-tools requirements.in pyproject.toml. And requirements.in is just -c ../srv/constraint.txt. Now it works 👍🏿

It still would be great to have a separate flag for it, though. Sounds like a quite common case to me, let's see if there is demand for it.

[lesnik512]

Similar feature request here with comment about drawbacks and proposed alternative

[AndydeCleyre]

I'm a -1 on this for now, since the method above (comment) and the comment linked by @lesnik512 seem to cover the need well.

[orsinium]

The described method allows to pass the constraints file as a content of another file but not as a CLI argument. The first thing I tried before opening the issue is to pass it as --pip-args but it doesn't work. This is an ugly workaround I have now in my Taskfile:

echo "-c {{.CONSTR_PATH}}" > requirements.in
pip-tools requirements.in pyproject.toml
rm requirements.in

Even if we decide that this workaround is ok and there is no need for code changes, it still should be at least documented, I'd never figure it out on my own.

[oselcuk-iqm]

Thank you for providing the workaround, I was banging my head against a wall for a while before finding this thread. Agree with @orsinium that it would be great to at least get this documented, though an explicit --constraints flag for pip-compile would be a better user experience in my opinion.

As an additional data point, my use case involved creating separate requirements files for different extras defined in a pyproject.toml file. Having a constraints.in file that just adds -c constraints.txt works perfectly. Here's a short description of our use case, since it might be a bit more common than two separate libraries needing to share constraints:

We define our dependencies in addition to test and doc extras in pyproject.toml. What we expected to be possible was to run pip-compile --extra test,doc --output-file=constraints.txt pyproject.toml, then create different lockfiles with pip-compile -extra test --output-file=test.txt --pip-args=-c=requirements.txt pyproject.toml (repeat for doc and base, latter has no extras). However this leads to constraints.txt file being completely ignored, producing potentially incompatible test.txt and doc.txt files, when there is a valid solution.

[sshishov]

Guys, how to make dependabot working with the proposed workaround? We are getting these errors from dependabot:

Fetching info for <package>
  update for 'package: X.Y.Z' is impossible

We are using unconstrained in file, and we have one constraints.in file which includes inside using -r flag all unconstrained files. Then when constraints.txt file is built, we are using it again for files like this:

# content of `base.in` file

-c constraints.txt
-r unconstrained/base.in

Somehow it is not woking with dependabot...

[atugushev]

I'm a -1 on this for now, since the method above (comment) and the comment linked by @lesnik512 seem to cover the need well.

@AndydeCleyre considering https://github.com/jazzband/pip-tools/issues/1891 wouldn't it makes sense to support pip-compile --extra dev -c requirements.txt -o dev-requirements.txt?

[AndydeCleyre]

Yes probably, especially since AFAIK you can't specify constraints in a pyproject.toml the same way.

[atugushev]

This PR #1936 adds -c option to the pip-compile. Any tests and reviews would be much appreciated.