Allow pip to send arbitrary headers to various hosts

[amancevice]

What's the problem this feature will solve?

Currently, there is no simple solution for pip to authenticate with a custom PyPI index using token-based authentication in HTTP headers.

The proposed feature should allow users to supply custom headers as options on various pip commands.

Based on discussion in PR #8030, there are a couple of concerns that warrant consideration:

1. The command-line options should behave like cURL's implementation, eg

curl -H 'Authorizer: Bearer ...' https://my.host.com/`

2. To prevent sending sensitive information to the wrong host(s), the user should be able to specify how to route the headers on a per-host basis.

Describe the solution you'd like

If I am a user maintaining a Python application that is built from a mix of public and private pips, I might want to install the dependencies in one command:

pip install --extra-index-host https://pypi.mine.com/simple/ \
  [proposed header option(s)] \
  public-pip-1 public-pip-2 ... public-pip-n \
  private-pip-1 private-pip-2 ... private-pip-n

or I may opt for a two-step install, installing the public pips first then installing the private ones on top (ignoring dependencies—assuming any public ones were installed in the previous step):

pip install public-pip-1 public-pip-2 ... public-pip-n
pip install --no-deps --index-host https://pypi.mine.com/simple/ \
  [proposed header option(s)] \
  private-pip-1 private-pip-2 ... private-pip-n

There is no obvious way to implement the desired header option(s), so I will propose a few in the next section...

Alternative Solutions

The JSON solution:

Send headers nested in a JSON document where top-level keys are the target hostnames.

pip install --extra-index-url https://pypi.mine.com/simple/ \
  --host-headers '{"pypi.mine.com": {"Authorization": "Bearer ..."}}' \
  ...

Pros:

• Easy to understand
• Fairly straightforward to implement

Cons:

• Verbose
• Not very cURL-like (no -H option)

---

The cURL-like solution:

Use a cURL-like DSL to set headers + hosts in one line.

The key difference is headers are optionally prefixed by their intended host, eg: instead of -H 'Header: Value', it's -H 'hostname: Header: value (or something similar)

If the hostname prefix is omitted, the header would be sent to all hosts (possibly with a warning)

pip install --extra-index-url https://pypi.mine.com/simple/ \
  -H 'pypi.mine.com: Authorization: Bearer ...' \
  ...

Pros:

• More like cURL's familiar syntax

Cons:

• Introduces new DSL
• A little harder to implement (have to parse the string)

---

A blended solution:

Allow users to specify headers using cURL-like syntax (-H 'Header: Value') AND allow users to supply a more verbose JSON document with per-host headers.

In this solution, the -H option is interpreted just like cURL and would be attached to all outgoing requests (possibly with a warning).

pip install --extra-index-url https://pypi.mine.com/simple/ \
  -H 'X-Spam: Spam Spam' \
  --host-headers '{"pypi.mine.com": {"Authorization": "Bearer ..."}}' \
  ...

Pros:

• Fairly best-of-both-worlds approach

Cons:

• More complex to implement
• Still pretty verbose

---

Some other solution! No wrong answers!

Additional context

Original discussion at #4475 and original PR at #8030

[amancevice]

I have a bare-minimum implementation open at #8078 that accepts -H / --header option(s) for pip commands configured to use a single index URL.

If the option is used with extra index URLs, the implementation issues a warning that the option is being ignored:

pip install boto3 --extra-index-url http://localhost:8000/ -H 'Authorization: SPAM'
WARNING: Refusing to set -H / --header option(s) because multiple index URLs are configured.

I think this implementation adds a useful feature without opening users up to any unintended risks and it could always be expanded upon in the future.

[pradyunsg]

I'm still not sure about how this interacts with multiple indexes. We have feature requests for adding support for different configuration for different URLs and I'm not very keen on locking ourselves out of that: https://github.com/pypa/pip/issues/8232.

[amancevice]

this section of code explains how I'm currently dealing with multiple indices: https://github.com/amancevice/pip/blob/header/src/pip/_internal/cli/req_command.py#L91-L97

In a nutshell, at the point when the -H / --header options are parsed there is a check to see how many index URLs are configured. If that number is > 1 then none of the headers are parsed and a warning is emitted (although @uranusjr just requested that be upgraded to a full-on error).

So this feature would really only be useful for users wanting to run pip commands against a single index, but I think—at least for now—that's a big win for my me at my workplace because we want to set up our own private PyPI that uses JWTs to manage auth.

[ghost]

Hi,

We are also trying to set up a private PyPI index that uses JWTs to manage auth so this feature would be great to have! Looking forward to following along with the progress on this – thanks!

[uranusjr]

This feature (along with providing alternative authenticating methods in general) has been proposed as a Python Software Foundation fundable project. The proposal also outlines why pip maintiners are not actively working on the feature. I would recommend interested parties to take a look, and consider donating resources, including developer time, to developing a solution. Please refer to the linked document on how you can contact the Packaging Working Group to move the project forward.

[rrlamichhane]

What's the status of this issue? It'd be nice to have

[HealsCodes]

Sad that this was tossed into "lets get founding" when there have been a number of perfectly usable PRs and PoC for this. Being able to pass a simple HTTP header along for a request feels like it doesn't warrant the amount of debate it got over the past 3 years.

[uranusjr]

All tasks listed in the fundable project page are available for anyone. The reason it eventually ended up on the list is no volunteer contributor was able to complete the implementation. Feel free to work on it.

[HealsCodes]

All tasks listed in the fundable project page are available for anyone. The reason it eventually ended up on the list is no volunteer contributor was able to complete the implementation. Feel free to work on it.

but.. there's multiple PRs in here that fulfil the requirement of the issue - they just didn't meet the extended requirements of the people discussing the topic. The original "let me add headers" - this one for example implemented the requested features just fine and then was closed for the sake of more debate. So I would argue against "no contributor was able to complete the implementation [of this issue as the OP asked for]"

[pradyunsg]

I'm sorry but can you confirm that you've (a) read the funding page and (b) read the discussions that took place on the PRs?

[integrii]

I hit this today as a blocker while trying to add an Authorization: heder to a package install.

[pradyunsg]

So I would argue against "no contributor was able to complete the implementation [of this issue as the OP asked for]"

Right, and the first concern raised about the proposed design was the thing that still hasn't been resolved. I appreciate that sending an Authorization header is something that people want; pip can't do that today -- if you need this, here's a comment from earlier in this issue linking to page about what needs to happen for this: https://github.com/pypa/pip/issues/8042#issuecomment-659313162. On that page, it also says:

Please contact the Packaging WG by emailing [email protected] to ask us to estimate how much one of these improvements would cost; we'll get back to you within a few business days.

[nwp90]

Actually per-host auth is not enough; per-source is required in order to cater for the likely use of bearer tokens to access git repositories, which are likely to be configured per-repository (this is our current use case).

[mattmshell]

Please correct me if I'm wrong, but it appears that this issue is dead in the water because it steps on the toes of the Packaging Working Group who recognise it as fundable work. Does that mean community contributions are equally futile... so if I want to address this particular issue in my spare time it, submit a patch and navigate the mailing lists PSF members are going to run interference and reject my change?

[pfmoore]

Not at all. It is recorded as a potential fundable piece of work because it's complex, and it seems likely that the only way of getting sufficient effort made on the design and implementation complexities is to pay someone to do it. However, if someone wants to do the work on a volunteer basis, that would be fine. As long as you're willing to commit the time and effort to address any comments and come up with a solution that works for us, we're happy to accept PRs. There are no PSF members waiting to reject contributions out of hand.

So far, though, no-one has addressed the first comment made in this thread - how would a design handle multiple indexes which may have different header requirements? Ignoring the option when multiple indexes are configured doesn't address the problem - if an index needs a custom header, it will still need it if it's one of two indexes being considered by pip.

If you are serious about contributing this feature to pip, I would start with a design, before you write any code. This feature is user-facing, so agreeing what the user interface will look like should be the first step.