Credential helpers

[chrispyduck]

Is your feature request related to a problem? Please describe. I would like to configure a private PyPi repository which requires short-lived credentials (AWS CodeArtifact), and I don't want my users to have to manage their credentials manually by setting environment variables.

Describe the solution you'd like I'd like some sort of credential helper framework, similar to docker's credential helpers, which pants could use to fetch credentials automatically as needed. With this, I can write and configure a credential helper for AWS CodeArtifact that would handle authentication automatically for my users.

Describe alternatives you've considered

• Writing a pants plugin dedicated to this purpose.
• Wrapping pants in another CLI that handles authentication and sets the required environment variable, but this isn't seamless to the users.

Additional context This is coming from a Slack thread with @tgolsson, who has a draft plugin that would provide similar functionality.

[tgolsson]

I will try to clean up the secrets framework for upstreaming and see how we can plumb it through, unless other maintainers think there are nicer ways of handling this.

This is some very old discussion when I built it originally... https://pantsbuild.slack.com/archives/C01CQHVDMMW/p1668980543623289

[cburroughs]

Some breakcrumbs from prior work at #21852 and #21853

[tgolsson]

Thanks @cburroughs! I'd completely missed that. I unfortunately have no idea how keyring works; I don't think I've ever used it personally... Do you know if it integrates with something like credential helpers? i.e. if you can use it as a frontend for short-lived credentials of any sort?

[cburroughs]

https://pip.pypa.io/en/stable/topics/authentication/#keyring-support

I think you can call through to all sorts of backends, including injecting a binary that happens to be called keyring but is your own code. But the bootstraping issues here are thorny.

Are "credential helpers" in this context the Docker variety?

[chrispyduck]

Are "credential helpers" in this context the Docker variety?

The idea is the same, but I'm specifically interested in applying the concept to PyPi repos.

[tdyas]

FYI, some users on Slack have been able to use the Pex --keyring-provider=subprocess option to use private PyPI repos with short-lived credentials. See this conversation on the Pantsbuild Slack for the details.

[ndellosa95]

FYI, some users on Slack have been able to use the Pex --keyring-provider=subprocess option to use private PyPI repos with short-lived credentials. See this conversation on the Pantsbuild Slack for the details.

It would definitely be desirable IMO to have a keyring pex that Pants can build and manage itself. Or even better, have the necessary credential helper installed in the same venv as pex pip. @tdyas I see you went pretty deep on implementing some stuff on the pex side of things. Any insights on what would need to be done to make --keyring-provider=import possible in pex pip?

[tdyas]

Any insights on what would need to be done to make --keyring-provider=import possible in pex pip?

It is a tough problem since you would need to include keyring alongside Pex's own code. Maybe generate a new pex PEX with keyring included? Then tell Pants to use that new pex PEX by putting it somewhere and pointing Pants at it with the options in the pex-cli subsystem. Beyond that idea, I don't really know offhand.

[tdyas]

A further thought: With --keying-provider=import, it is the pip invoked by pex which needs to have the keyring distribution accessible to it. It might not be enough to include keyring as part of pex. This is part of the "bootstrapping" problem discussed in https://github.com/pex-tool/pex/pull/2592.

[ndellosa95]

A further thought: With --keying-provider=import, it is the pip invoked by pex which needs to have the keyring distribution accessible to it. It might not be enough to include keyring as part of pex. This is part of the "bootstrapping" problem discussed in pex-tool/pex#2592.

Yes I tried your initial suggestion at some point and found that even when using a PEX.pex with keyring installed it didn't see it. My understanding is that the pip within pex is isolated from everything else in the pex, even when using a non-vendored version of pip. Based on like half an hour of research, I get the impression getting --keyring-provider=import to work would be fairly difficult.

I may have some time this week to rig up a subprocess implementation for Pants.

[salotz]

Hitting this as well. Would be nice to have keyring work alongside GCP (Google Artifact Registry) application credentials and not have to create a service account and key to get a username and password.