Keyring errors during non-publishing operations

[wahuneke]

I gather that keyring integration was added as a convenience feature so that you can publish packages using keys stored in your keyring. This ticket describes that feature:

https://github.com/python-poetry/poetry/issues/210

But it appears that poetry tries to access the keyring even for install operations. In one of my first times using poetry - I, as a security paranoid person, decided to abort installation of a py package I wanted because the installation process seemed to be wanting to access my keyring.

I think people should be careful about granting programs access to things they should not need to access. This behavior is an example of a problem that makes that hard to achieve.

For me, I worked around the problem by pip uninstalling the 'keyring' package from that virt env.

In the future, I think it would be beneficial to poetry's adoption and to its users, if users are not prompted for keyring unless keys should actually be needed (for a 'publish' operation).

Thanks! I'm happy to be trying out poetry.

[sdispater]

Do you have private repositories present in your pyproject.toml file?

I think you are affected by the issue that is fixed by #1892. The fix will be available in the next bug fix release.

[colatkinson]

I was affected by something similar on the latest release (1.0.5), but I'm not sure if it's entirely the same.

The keyring will be hit for private repos if there is an auth.toml with only the username set, even if credentials are provided via environment variables.

I believe the relevant code is here. In this case, auth will be truthy, causing the keyring path to be taken.

In our case, the "fix" was to remove any leftover auth.toml files in the environment. Manually uninstalling keyring had the same effect.

I was wondering your thoughts on adding a config variable that would prevent any keyring hits altogether (by e.g. unconditionally marking the keyring as unavailable). This would have largely the same effect as @wahuneke's workaround, but might be a bit cleaner for CI pipelines and such.

[asandeep]

+1 to this. I am facing similar issue while integrating poetry with Jenkins. Even though private repo credentials were provided via environment variables, poetry still tries to look into keyring for credentials.

[dariobig]

+1 running into this issue while working over ssh

[bdowling]

fwiw, I came across this issue and agree with the concerns mentioned above.

But fwiw, I came here due to a issue with my distro that shows a dependency mismatch on keyring. keyring==18 is required (circa 2019), where as my distro has keyring-21.2 available, not sure without looking on the compatibility issues, but thought I'd mention if in case that can be relaxed....

$ poetry init
Traceback (most recent call last):
  File "/usr/lib/python3.8/site-packages/pkg_resources/__init__.py", line 583, in _build_master
    ws.require(__requires__)
  File "/usr/lib/python3.8/site-packages/pkg_resources/__init__.py", line 900, in require
    needed = self.resolve(parse_requirements(requirements))
  File "/usr/lib/python3.8/site-packages/pkg_resources/__init__.py", line 791, in resolve
    raise VersionConflict(dist, req).with_context(dependent_req)
pkg_resources.ContextualVersionConflict: (keyring 21.2.1 (/usr/lib/python3.8/site-packages), Requirement.parse('keyring==18.*,>=18.0.0'), {'poetry'})

[kakarukeys]

I'm seeing this issue in poetry 1.1.6

My setup: MacOS Mojave 10.14.6 localhost private pypiserver with authentication required for upload only

[[tool.poetry.source]]
name = "foo"
url = "http://localhost/simple/"
secondary = true

$ poetry config --list
cache-dir = "..."
experimental.new-installer = true
installer.parallel = true
repositories.foo.url = "http://localhost"
virtualenvs.create = true
virtualenvs.in-project = null
virtualenvs.path = "{cache-dir}/virtualenvs" 

$ cat auth.toml
[http-basic]
[http-basic.foo]
username = "kakarukeys"

The commands that poetry asked for keyring access: poetry add lxml ??? why? it should get the package from the official PyPI poetry run python --version this makes no sense either.

if you need the installation scripts for the local private pypi let me know

[alesdakshanin]

+1 this

[abn]

Poetry searches for a package in all package sources by default.

Additionally you might want to add the publishing repository with a different name, if the source uses the same name authenticator will load the credentials.

[erooke]

Poetry 1.2 now hits the keyring for most operations (add, update, install, self update, etc) rendering it unusable for me. I can "fix" the problem by exporting PYTHON_KEYRING_BACKEND=keyring.backends.null.Keyring but it would be much nicer if poetry just didn't hit the keyring at all if it wasn't needed.

[txtsd]

I tried poetry for the first time today.

This command will guide you through creating your pyproject.toml config.

Package name [herpderp]:  
Version [0.1.0]:  
Description []:  
Author [txtsd <[email protected]>, n to skip]:  
License []:  GPL-3.0-only
Compatible Python versions [^3.10]:  

Would you like to define your main dependencies interactively? (yes/no) [yes] yes
You can specify a package in the following forms:
  - A single name (requests): this will search for matches on PyPI
  - A name and a constraint (requests@^2.23.0)
  - A git url (git+https://github.com/python-poetry/poetry.git)
  - A git url with a revision (git+https://github.com/python-poetry/poetry.git#develop)
  - A file path (../my-package/my-package.whl)
  - A directory (../my-package/)
  - A url (https://example.com/packages/my-package-0.1.0.tar.gz)

Package to add or search for (leave blank to skip): requests
Found 20 packages matching requests
Showing the first 10 matches

Enter package # to add, or the complete package name if it is not listed []:
 [ 0] requests
 [ 1] requests5
 [ 2] requests3
 [ 3] requests2
 [ 4] pycopy-requests
 [ 5] requests-middleware
 [ 6] grasspy-requests
 [ 7] requests-kerberos
 [ 8] fed-requests
 [ 9] requests-httpsproxy
 [10] 
 > 1
Enter the version constraint to require (or leave blank to use the latest version): 

Failed to open keyring: org.freedesktop.DBus.Error.ServiceUnknown: The name :1.146 was not provided by any .service files.

What a horrible experience. If there isn't a keyring, assume it doesn't exist and continue. Why hard fail at this stage? It's not even like I don't have a keyring. I use KeePassXC's freedesktop.org Secrets integration. It works fine system-wide.

[neersighted]

@txtsd Keyring support is a new feature, and this is a complex interaction across the matrix of possible platforms, configurations, versions, and backends that a user may invoke Poetry in. Many of these sharp edges simply haven't been filed off because of how large that matrix is and how dependent on external factors keychain interactions are.

MRs improving the authentication experience or docs are welcome -- we're going to have to gradually file down these rough edges as keychain support matures.

[willforde]

Had the same issue. Had to change the keyring package config to point to the keyring.backends.SecretService.Keyring backend. For me the keyring package was trying to use kwallet, but I'm using Gnome not KDE.

OS = Arch Linux DE = Gnome

[txtsd]

Okay, I created this file ~/.config/python_keyring/keyringrc.cfg and populated it with

[backend]
default-keyring=keyring.backends.SecretService.Keyring

Then I unset PYTHON_KEYRING_BACKEND and ran poetry init:

λ poetry init

This command will guide you through creating your pyproject.toml config.

Package name [test]:  
Version [0.1.0]:  
Description []:  
Author [txtsd <[email protected]>, n to skip]:  
License []:  
Compatible Python versions [^3.10]:  

Would you like to define your main dependencies interactively? (yes/no) [yes] 
You can specify a package in the following forms:
  - A single name (requests): this will search for matches on PyPI
  - A name and a constraint (requests@^2.23.0)
  - A git url (git+https://github.com/python-poetry/poetry.git)
  - A git url with a revision (git+https://github.com/python-poetry/poetry.git#develop)
  - A file path (../my-package/my-package.whl)
  - A directory (../my-package/)
  - A url (https://example.com/packages/my-package-0.1.0.tar.gz)

Package to add or search for (leave blank to skip): lxml
Found 20 packages matching lxml
Showing the first 10 matches

Enter package # to add, or the complete package name if it is not listed []:
 [ 0] lxml
 [ 1] lxml-stubs
 [ 2] zsi-lxml
 [ 3] lxml-wrapper
 [ 4] readability-lxml
 [ 5] lxml2dict
 [ 6] suds-lxml
 [ 7] json-lxml
 [ 8] lxml-asserts
 [ 9] gocept.lxml
 [10] 
 > 0
Enter the version constraint to require (or leave blank to use the latest version): 

Empty module name

Different error now.

[neersighted]

There's some busted keyring behavior with your particular backends that seem to

1. invoke with poor error handling during init -- odd (though it's possible that the search code in poetry init was not made robust against keyring errors as an oversight)
2. bypass most of our error handling and spit out unhelpful messages

Some more insight into your system (what backend should be available, OS version, package versions), etc would be helpful for those that attempt to address yours/similar issues @txtsd.

In the meantime, you can revert to 1.1-style writing to disk using the keyring.backends.null.Keyring backend, of course.

[txtsd]

@neersighted I use Arch Linux

λ pip freeze | grep poetry
poetry==1.2.0
poetry-core==1.1.0
poetry-plugin-export==1.0.6

I set the config to use keyring.backends.null.Keyring for now. Thanks!

[croqaz]

Happened to me today for the first time. I used Poetry until last month without any issues, but today I saw same error. export PYTHON_KEYRING_BACKEND=keyring.backends.null.Keyring seems to fix the issue.

[lazka]

I'm seeing the same error on Windows with libsecret installed:

  • Installing urllib3 (1.26.12): Failed
  Error
  g-dbus-error-quark: The name org.freedesktop.secrets is unknown (2)

[blakjak44]

I wanted to add my experience with this issue. We are running our CI on Github self-hosted runners (Ubuntu 20.04) and instead of getting an error, Poetry would just hang when running poetry install. After some digging I found that it was stalling on the collection.unlock() call in the keyring library, presumably because the gnome-keyring-daemon was not launching without a user login.

The PYTHON_KEYRING_BACKEND=keyring.backends.null.Keyring fix worked for me but it cost me a lot of time troubleshooting this.

My suggestion is to make the keychain backend an opt-in feature.

[rdbisme]

I have the same problem on WSL2:

 1  /usr/lib/python3.10/site-packages/keyring/core.py:72 in get_credential
        return get_keyring().get_credential(service_name, username)

  Error

  g-spawn-exit-error-quark: Error spawning command line “dbus-launch --autolaunch=93716890c157416eacbb2d8801a1e4f6 --binary-syntax --close-stderr”: Child process exited with code 1 (1)

disabling the keyring fixes the problem. I think this feature should be opt-in as suggested by @blakjak44

[netopsengineer]

I get the following on Ubuntu Mint 20.04.5 LTS and Python 3.10.7, using Poetry 1.2.1, when I remotely SSH to the machine.

It seems that if I do an RDP session to the GUI of this machine and unlock my keyring which is empty for what its worth, then I don't get this error. The fix mentioned about disabling it with PYTHON_KEYRING_BACKEND=keyring.backends.null.Keyring also worked as well.

clab_arista$ poetry install
Installing dependencies from lock file

Package operations: 8 installs, 0 updates, 0 removals

  • Installing markupsafe (2.1.1): Failed

  KeyringLocked

  Failed to unlock the collection!

  at ~/.local/share/pypoetry/venv/lib/python3.10/site-packages/keyring/backends/SecretService.py:67 in get_preferred_collection
       63│             raise InitError("Failed to create the collection: %s." % e)
       64│         if collection.is_locked():
       65│             collection.unlock()
       66│             if collection.is_locked():  # User dismissed the prompt
    →  67│                 raise KeyringLocked("Failed to unlock the collection!")
       68│         return collection
       69│
       70│     def unlock(self, item):
       71│         if hasattr(item, 'unlock'):

[trygveaa]

Poetry searches for a package in all package sources by default.

@abn: But it happens even with no configured package sources, i.e. when only going against PyPI

@txtsd Keyring support is a new feature, and this is a complex interaction across the matrix of possible platforms, configurations, versions, and backends that a user may invoke Poetry in. Many of these sharp edges simply haven't been filed off because of how large that matrix is and how dependent on external factors keychain interactions are.

MRs improving the authentication experience or docs are welcome -- we're going to have to gradually file down these rough edges as keychain support matures.

@neersighted: If that's the case, it shouldn't be enabled by default IMO. For me add just hanged for a while with no feedback on why it was hanging until the keyring password prompt appeared. I didn't want it to access the keyring, so I canceled that, and then the add was aborted. This is a pretty bad experience out of the box. poetry add --help doesn't mention anything about the keyring either, so I had to search and find this issue to figure out how to disable it. I found https://python-poetry.org/docs/repositories/ which mentions keyrings before this issue, but that doesn't mention disabling it either.

[felix-ht]

I just tried upgrading to poetry 1.2.x, did poetry install and got thousands of Failed to create the collection: Prompt dismissed.. errors. All other commands are broken as well even self update is broken without the mitigation.

Going back to 1.1.x until this is fixed. I don't want to tell all my developers that hey have to mess with PYTHON_KEYRING_BACKEND just to be able to use the most basic features.

And this while i was really looking to move to 1.2

[romanzdk]

keyring --disable solved the issue for me

[felix-ht]

keyring --disable solved the issue for me

It just changed to a new error with that.

[zyv]

This is happening on Poetry 1.2.2 for me, the following fixes the issue as mentioned in #5250:

export PYTHON_KEYRING_BACKEND=keyring.backends.fail.Keyring

[remram44]

I'm running into this too. I see that there is already a fallback if the returned credential is None, so maybe this is just a matter of catching KeyringLocked and returning the default too?

https://github.com/python-poetry/poetry/blob/9df21a72138663e9d5a9cf80beb2f62751eed361/src/poetry/utils/password_manager.py#L50-L57

A better option would probably be for Authenticator to try the request without credentials, only trying to obtain credentials if they are required (for example HTTP 401). This is the way most clients work, including web browsers. Unlocking the keyring for every request is definitely the wrong behavior.

https://github.com/python-poetry/poetry/blob/9df21a72138663e9d5a9cf80beb2f62751eed361/src/poetry/utils/authenticator.py#L206-L211

edit: I'm happy to contribute a fix, but this requires a maintainer to pick between the options above. Or should I open 2 PRs?

[swills1]

I had the same problem on Fedora 37 beta. Poetry had been working until I updated to latest. I was trying to do poetry add. I was having a problem where my lock file wasn't updating with the toml requirements. So I tried poetry add and was getting the error mentioned here.

@croqaz suggestion fixed it for me.

export PYTHON_KEYRING_BACKEND=keyring.backends.null.Keyring

[MiMiMeowMeow]

I'm getting this issue when ssh session into a Windows 11 machine. poetry 1.2.2

poetry install -vvv Installing dependencies from lock file

Finding the necessary packages for the current system

Package operations: 26 installs, 1 update, 0 removals

• Installing shiboken6 (6.4.0.1) [keyring.backend] Loading KWallet [keyring.backend] Loading SecretService [keyring.backend] Loading Windows [keyring.backend] Loading chainer [keyring.backend] Loading libsecret [keyring.backend] Loading macOS

Stack trace:

2 C:\Program Files\Python310\lib\site-packages\win32ctypes\pywin32\pywintypes.py:35 in pywin32error 33│ def pywin32error(): 34│ try: → 35│ yield 36│ except WindowsError as exception: 37│ raise error(exception.winerror, exception.function, exception.strerror)

1 C:\Program Files\Python310\lib\site-packages\win32ctypes\pywin32\win32cred.py:70 in CredRead 68│ else: 69│ pcreds = _authentication.PCREDENTIAL() → 70│ _authentication._CredRead( 71│ TargetName, Type, flag, _common.byreference(pcreds)) 72│ try:

OSError

[WinError 1312] A specified logon session does not exist. It may already have been terminated.

at C:\Program Files\Python310\lib\site-packages\win32ctypes\core\ctypes_util.py:53 in check_zero 49│ 50│ def check_zero_factory(function_name=None): 51│ def check_zero(result, function, arguments, *args): 52│ if result == 0: → 53│ raise make_error(function, function_name) 54│ return result 55│ return check_zero 56│ 57│

The following error occurred when trying to handle this error:

Stack trace:

21 C:\Program Files\Python310\lib\site-packages\poetry\installation\executor.py:261 in _execute_operation 259│ 260│ try: → 261│ result = self._do_execute_operation(operation) 262│ except EnvCommandError as e: 263│ if e.e.returncode == -2:

20 C:\Program Files\Python310\lib\site-packages\poetry\installation\executor.py:334 in _do_execute_operation 332│ return 0 333│ → 334│ result: int = getattr(self, f"execute{method}")(operation) 335│ 336│ if result != 0:

19 C:\Program Files\Python310\lib\site-packages\poetry\installation\executor.py:454 in _execute_install 452│ 453│ def _execute_install(self, operation: Install | Update) -> int: → 454│ status_code = self._install(operation) 455│ 456│ self._save_url_reference(operation)

18 C:\Program Files\Python310\lib\site-packages\poetry\installation\executor.py:488 in _install 486│ archive = self._download_link(operation, Link(package.source_url)) 487│ else: → 488│ archive = self._download(operation) 489│ 490│ operation_message = self.get_operation_message(operation)

17 C:\Program Files\Python310\lib\site-packages\poetry\installation\executor.py:633 in _download 631│ 632│ def _download(self, operation: Install | Update) -> Path: → 633│ link = self._chooser.choose_for(operation.package) 634│ 635│ if link.yanked:

16 C:\Program Files\Python310\lib\site-packages\poetry\installation\chooser.py:77 in choose_for 75│ """ 76│ links = [] → 77│ for link in self._get_links(package): 78│ if link.is_wheel: 79│ if not self._no_binary_policy.allows(package.name):

15 C:\Program Files\Python310\lib\site-packages\poetry\installation\chooser.py:119 in _get_links 117│ else: 118│ repository = self._pool.repository("pypi") → 119│ links = repository.find_links_for_package(package) 120│ 121│ hashes = [f["hash"] for f in package.files]

14 C:\Program Files\Python310\lib\site-packages\poetry\repositories\pypi_repository.py:158 in find_links_for_package 156│ 157│ def find_links_for_package(self, package: Package) -> list[Link]: → 158│ json_data = self._get(f"pypi/{package.name}/{package.version}/json") 159│ if json_data is None: 160│ return []

13 C:\Program Files\Python310\lib\site-packages\poetry\repositories\pypi_repository.py:246 in _get 244│ ) -> dict[str, Any] | None: 245│ try: → 246│ json_response = self.session.get( 247│ self._base_url + endpoint, 248│ raise_for_status=False,

12 C:\Program Files\Python310\lib\site-packages\poetry\utils\authenticator.py:247 in get 245│ 246│ def get(self, url: str, **kwargs: Any) -> requests.Response: → 247│ return self.request("get", url, **kwargs) 248│ 249│ def post(self, url: str, **kwargs: Any) -> requests.Response:

11 C:\Program Files\Python310\lib\site-packages\poetry\utils\authenticator.py:188 in request 186│ headers = kwargs.get("headers") 187│ request = requests.Request(method, url, headers=headers) → 188│ credential = self.get_credentials_for_url(url) 189│ 190│ if credential.username is not None or credential.password is not None:

10 C:\Program Files\Python310\lib\site-packages\poetry\utils\authenticator.py:311 in get_credentials_for_url 309│ # no credentials were provided in the url, try finding the 310│ # best repository configuration → 311│ self._credentials[url] = self._get_credentials_for_url(url) 312│ else: 313│ # Split from the right because that's how urllib.parse.urlsplit()

9 C:\Program Files\Python310\lib\site-packages\poetry\utils\authenticator.py:272 in _get_credentials_for_url 270│ 271│ credential = ( → 272│ self._get_credentials_for_repository(repository=repository) 273│ if repository is not None 274│ else HTTPAuthCredential()

8 C:\Program Files\Python310\lib\site-packages\poetry\utils\authenticator.py:260 in _get_credentials_for_repository 258│ 259│ if key not in self._credentials: → 260│ self._credentials[key] = repository.get_http_credentials( 261│ password_manager=self._password_manager, username=username 262│ )

7 C:\Program Files\Python310\lib\site-packages\poetry\utils\authenticator.py:90 in get_http_credentials 88│ if credential.password is None: 89│ # fallback to url and netloc based keyring entries → 90│ credential = password_manager.keyring.get_credential( 91│ self.url, self.netloc, username=credential.username 92│ )

6 C:\Program Files\Python310\lib\site-packages\poetry\utils\password_manager.py:51 in get_credential 49│ 50│ for name in names: → 51│ credential = keyring.get_credential(name, username) 52│ if credential: 53│ return HTTPAuthCredential(

5 C:\Program Files\Python310\lib\site-packages\keyring\core.py:72 in get_credential 70│ ) -> typing.Optional[credentials.Credential]: 71│ """Get a Credential for the specified service.""" → 72│ return get_keyring().get_credential(service_name, username) 73│ 74│

4 C:\Program Files\Python310\lib\site-packages\keyring\backends\Windows.py:167 in get_credential 165│ # get any first password under the service name 166│ if not res: → 167│ res = self._get_password(service) 168│ if not res: 169│ return None

3 C:\Program Files\Python310\lib\site-packages\keyring\backends\Windows.py:108 in _get_password 106│ def _get_password(self, target): 107│ try: → 108│ res = win32cred.CredRead( 109│ Type=win32cred.CRED_TYPE_GENERIC, TargetName=target 110│ )

2 C:\Program Files\Python310\lib\site-packages\win32ctypes\pywin32\win32cred.py:63 in CredRead 61│ 62│ flag = 0 → 63│ with _pywin32error(): 64│ if _backend == 'cffi': 65│ ppcreds = _authentication.PPCREDENTIAL()

1 C:\Program Files\Python310\lib\contextlib.py:153 in exit 151│ value = typ() 152│ try: → 153│ self.gen.throw(typ, value, traceback) 154│ except StopIteration as exc: 155│ # Suppress StopIteration unless it's the same exception that

error

(1312, 'CredRead', 'A specified logon session does not exist. It may already have been terminated.')

at C:\Program Files\Python310\lib\site-packages\win32ctypes\pywin32\pywintypes.py:37 in pywin32error 33│ def pywin32error(): 34│ try: 35│ yield 36│ except WindowsError as exception: → 37│ raise error(exception.winerror, exception.function, exception.strerror) 38│

[neersighted]

Please try and refrain from "me too" style comments given the number of participants in this issue. A :+1: is a simple way to show support/interest, and doesn't generate unnecessary notifications.

[Daniel451]

@neersighted is this expected behavior or is there any downside to making keyring optional alltogether? I am asking because I am also running poetry in (1) CI pipelines and also on (2) headless servers. Both do not have a keyring, thus Poetry 1.2 basically is unusable there without the PYTHON_KEYRING_BACKEND=keyring.backends.null.Keyring workaround.

CI pipelines usually do not need a keyring. For headless servers we SSH agent forwarding, which is super neat for Git and any ssh based stuff on the server, thus also no local keyring needed.

I would guess that there are lots of CI pipelines, headless servers, nodes in datacenters, ... where simply no keyring will be present and it is often also not an option to set one up.