Exclude package from pinning during pip-compile

[jamesbraza]

What's the problem this feature will solve?

In the requirements.txt compiled by pip-compile, I would like to leave a particular dependency unpinned.

Describe the solution you'd like

A new command-line flag (or config file option) to not pin certain packages. In other words, an allow-list of packages to not pin.

Our internal packages infer their own dependency dynamically inside setup.py, and if pip-compile pins it in the generated requirements.txt, this messes things up.

Alternative Solutions

None tried.

Additional context

[AndydeCleyre]

I think this is already solved in master for inclusion in the next release, as --unsafe-package.

$ python3 -m venv venv
$ . ./venv/bin/activate
$ pip install -U 'pip-tools @ git+https://github.com/jazzband/pip-tools@master'
$ printf '%s\n' requests httpx >requirements.in
$ pip-compile --unsafe-package httpx --annotation-style=line --no-header

anyio==3.6.1              # via httpcore
certifi==2022.6.15        # via httpcore, httpx, requests
charset-normalizer==2.1.0  # via requests
h11==0.12.0               # via httpcore
httpcore==0.15.0          # via httpx
idna==3.3                 # via anyio, requests, rfc3986
requests==2.28.1          # via -r requirements.in
rfc3986[idna2008]==1.5.0  # via httpx
sniffio==1.2.0            # via anyio, httpcore, httpx
urllib3==1.26.11          # via requests

# The following packages are considered to be unsafe in a requirements file:
# httpx

$ pip-compile --unsafe-package requests --annotation-style=line --no-header

anyio==3.6.1              # via httpcore
certifi==2022.6.15        # via httpcore, httpx, requests
charset-normalizer==2.1.0  # via requests
h11==0.12.0               # via httpcore
httpcore==0.15.0          # via httpx
httpx==0.23.0             # via -r requirements.in
idna==3.3                 # via anyio, requests, rfc3986
rfc3986[idna2008]==1.5.0  # via httpx
sniffio==1.2.0            # via anyio, httpcore, httpx
urllib3==1.26.11          # via requests

# The following packages are considered to be unsafe in a requirements file:
# requests

$ pip-compile --unsafe-package requests --unsafe-package httpx --annotation-style=line --no-header

anyio==3.6.1              # via httpcore
certifi==2022.6.15        # via httpcore, httpx, requests
charset-normalizer==2.1.0  # via requests
h11==0.12.0               # via httpcore
httpcore==0.15.0          # via httpx
idna==3.3                 # via anyio, requests, rfc3986
rfc3986[idna2008]==1.5.0  # via httpx
sniffio==1.2.0            # via anyio, httpcore, httpx
urllib3==1.26.11          # via requests

# The following packages are considered to be unsafe in a requirements file:
# httpx
# requests

$ pip-compile --unsafe-package sniffio --annotation-style=line --no-header

anyio==3.6.1              # via httpcore
certifi==2022.6.15        # via httpcore, httpx, requests
charset-normalizer==2.1.0  # via requests
h11==0.12.0               # via httpcore
httpcore==0.15.0          # via httpx
httpx==0.23.0             # via -r requirements.in
idna==3.3                 # via anyio, requests, rfc3986
requests==2.28.1          # via -r requirements.in
rfc3986[idna2008]==1.5.0  # via httpx
urllib3==1.26.11          # via requests

# The following packages are considered to be unsafe in a requirements file:
# sniffio

[jamesbraza]

@AndydeCleyre thank you for the fast response! This almost works, except I need the --unsafe-package=foo to be uncommented at the bottom.

So for example from bottom example above, sniffio needs to be uncommented:

...
urllib3==1.26.11          # via requests

# The following packages are considered to be unsafe in a requirements file:
sniffio

How can one accomplish that in the next release? Also, any ideas when the next release will happen?

[AndydeCleyre]

As for when the next release will happen, my guess is as soon as #1649 is all wrapped up.

My recommendation for your case is to arrange files something like:

• pinned-requirements.in compiled to pinned-requirements.txt, specifying --unsafe-package PKG
• unpinned-requirements.txt containing all pkgs passed as "unsafe"
• requirements.txt:

  -r pinned-requirements.txt
  -r unpinned-requirements.txt
  

[jamesbraza]

I do hear your solution, and it makes sense. It requires having 1 requirements.in and 3 requirements.txt. Thanks for the proposed workaround! An aside is I wouldn't need to use --unsafe-package in the pinned-requirements.in because the "unsafe" package would actually live in unpinned-requirements.txt.

However, I think the workaround is heavy handed, I don't want four files when there only needs to be two files.

Ideally there's 1 requirements.in and 1 requirements.txt + invoking pip-compile --allow-unpinned=foo (or something like this). That takes us back to the feature request in the original post. Let me know what you think!

[AndydeCleyre]

Passing them as unsafe during compilation ensures you won't get conflicts if one of your pinned deps starts depending on an unpinned dep, but I guess you don't need to bother in your case.

I don't know about the kind of change you're asking for, as pip-compile so far always creates exactly pinned lockfiles, and including unpinned reqs may not fit the tool.

I'll step back and see if any contributors or users have something to say about it.

EDIT: I guess you can easily drop unpinned-requirements.txt in favor of including those directly in requirements.txt, reducing the file count to 3.

[jamesbraza]

Passing them as unsafe during compilation ensures you won't get conflicts if one of your pinned deps starts depending on an unpinned dep, but I guess you don't need to bother in your case.

Gotchu on this, thanks for clarifying. And thanks again @AndydeCleyre for the suggestions, I actually tried to implement them just now.

One thing came up: when I move the unpinned foo package to another unpinned-requirements.txt file, its indirect dependencies no longer get reflected by pip-compile. Ideally its indirect dependencies get compared + optimized along with all other indirect dependencies coming from pinned-requirements.in.

So in other words, I realize the workaround posed won't work for my use case. Ideally foo gets slurped up along with all other packages for analysis by pip, and then foo is left unpinned in the output requirements.txt.

[jamesbraza]

EDIT: I guess you can easily drop unpinned-requirements.txt in favor of including those directly in requirements.txt, reducing the file count to 3.

Just seeing this edit. I see what you're saying, but again the issue is the same as my prior message, the indirect dependencies inside the foo package won't get reflected in the pinning.

[AndydeCleyre]

You should include those packages in pinned-requirements.in and exclude them via --unsafe-package when compiling.

[jamesbraza]

Ah I am tracking again. Thanks for your patience @AndydeCleyre ! 😄

So the workaround again works. Tho I still think it would be nice to have the ability to leave things unpinned in a pip-compiled output.

[AndydeCleyre]

Just adding a convenience Zsh function for that behavior here:

pc-unpinned () {  # <unpinned-pkg-csv> <output-file> [<pip-compile-arg>...]
  emulate -L zsh

  local help_text='pc-unpinned <unpinned-pkg-csv> <output-file> [<pip-compile-arg>...]'

  if [[ ! $1 ]] || [[ ! $2 ]] || [[ $1 == -* ]] {
    print -u2 $help_text
    return 1
  }

  local unpinned_pkgs=(${(s:,:)1})
  shift

  local destination=$1
  shift

  local unsafe_args=() pkg
  for pkg ( $unpinned_pkgs )  unsafe_args+=(--unsafe-package $pkg)

  pip-compile $unsafe_args -o $destination $@

  print -l $unpinned_pkgs >>$destination
}