_pygit2.GitError: error loading known_hosts

[jamesbernardi]

Since upgrading to 1.12.0 this has been a recurring error with Salt Stack 3005

_pygit2.GitError: error loading known_hosts

Rolling back to v 1.11.1 fixes this issue

[wwimberly]

Ran into the same issue in 1.12.0 for salt 3006.0. Downgrading to 1.11.1 made things happy.

[jdavid]

Can you try with 1.12.2 ? There we bundle libssh2 1.11.0 which apparently fixes some compatibility issues with OpenSSL 3.0

[wwimberly]

I'm seeing the same behavior with pygit 1.12.2.

2023-06-28 14:58:53,833 [salt.utils.gitfs :1894][ERROR   ][12776] Error occurred fetching gitfs remote 'git@git-host:our-org/our-repo.git': error loading known_hosts: 
Traceback (most recent call last):
  File "/opt/saltstack/salt/lib/python3.10/site-packages/salt/utils/gitfs.py", line 1870, in _fetch
    fetch_results = origin.fetch(**fetch_kwargs)
  File "/opt/saltstack/salt/extras-3.10/pygit2/remote.py", line 140, in fetch
    payload.check_error(err)
  File "/opt/saltstack/salt/extras-3.10/pygit2/callbacks.py", line 98, in check_error
    check_error(error_code)
  File "/opt/saltstack/salt/extras-3.10/pygit2/errors.py", line 65, in check_error
    raise GitError(message)
_pygit2.GitError: error loading known_hosts: 
2023-06-28 14:58:55,370 [salt.utils.gitfs :1894][ERROR   ][12772] Error occurred fetching git_pillar remote 'master git@git-host:our-org/our-repo.git': error loading known_hosts: 
Traceback (most recent call last):
  File "/opt/saltstack/salt/lib/python3.10/site-packages/salt/utils/gitfs.py", line 1870, in _fetch
    fetch_results = origin.fetch(**fetch_kwargs)
  File "/opt/saltstack/salt/extras-3.10/pygit2/remote.py", line 140, in fetch
    payload.check_error(err)
  File "/opt/saltstack/salt/extras-3.10/pygit2/callbacks.py", line 98, in check_error
    check_error(error_code)
  File "/opt/saltstack/salt/extras-3.10/pygit2/errors.py", line 65, in check_error
    raise GitError(message)
_pygit2.GitError: error loading known_hosts:

root@syndic01:/var/log/salt# salt --versions
Salt Version:
          Salt: 3006.1
 
Python Version:
        Python: 3.10.11 (main, May  5 2023, 02:31:54) [GCC 11.2.0]
 
Dependency Versions:
          cffi: 1.14.6
      cherrypy: unknown
      dateutil: 2.8.1
     docker-py: Not Installed
         gitdb: Not Installed
     gitpython: Not Installed
        Jinja2: 3.1.2
       libgit2: 1.6.4
  looseversion: 1.0.2
      M2Crypto: Not Installed
          Mako: Not Installed
       msgpack: 1.0.2
  msgpack-pure: Not Installed
  mysql-python: Not Installed
     packaging: 22.0
     pycparser: 2.21
      pycrypto: Not Installed
  pycryptodome: 3.9.8
        pygit2: 1.12.2
  python-gnupg: 0.4.8
        PyYAML: 5.4.1
         PyZMQ: 23.2.0
        relenv: 0.12.3
         smmap: Not Installed
       timelib: 0.2.4
       Tornado: 4.5.3
           ZMQ: 4.3.4
 
System Versions:
          dist: ubuntu 22.04.2 jammy
        locale: utf-8
       machine: x86_64
       release: 5.15.0-71-generic
        system: Linux
       version: Ubuntu 22.04.2 jammy

[OrangeDog]

~~This is a problem with Salt's custom python environment. Using pygit2 1.12.0 in any other environment, including with Salt, works fine.~~ https://github.com/saltstack/salt/issues/64345

Since 1.13 this now happens without Salt's custom python.

[howbazaar]

Is there a change that causes HOME to be determined at package load time? That is what I've observed.

[dwoz]

Wat this not fixed in 3006.3?

[oogali]

I ran into this issue with a fresh install of Salt 3006.10 and it appears that the underlying issue is indeed that libgit2 cannot function without the HOME environment variable defined.

I traced it back to a an error encountered in git_sysdir_find_homedir , which is called by git_sysdir_expand_homedir_file which is called by load_known_hosts.

When this function does fail, it doesn't set an error message (which I presume would be done with ssh_error). This leads to the seemingly incomplete error message of Error occurred fetching gitfs remote 'git@git-host:our-org/our-repo.git': error loading known_hosts:.

While one can configure the user home directory in libgit2 (GIT_OPT_SET_HOMEDIR), that option isn't available for setting via pygit2 (src/options.c).

The TL;DR workaround is to set a value for the HOME environment variable.

/**
 * Find the home directory. On Windows, this will look at the `HOME`,
 * `HOMEPATH`, and `USERPROFILE` environment variables (in that order)
 * and return the first path that is set and exists. On other systems,
 * this will simply return the contents of the `HOME` environment variable.
 *
 * @param path buffer to write the full path into
 * @return 0 if found, GIT_ENOTFOUND if not found, or -1 on other OS error
 */
extern int git_sysdir_find_homedir(git_str *path);

/**
 * Expand the name of a file in the user's home directory. This
 * function makes no attempt to check for the existence of the file,
 * and is useful if you want the full path regardless of existence.
 *
 * @param path buffer to write the full path into
 * @param filename name of file in the home directory
 * @return 0 on success or -1 on error
 */
extern int git_sysdir_expand_homedir_file(git_str *path, const char *filename);

Reproduced with:

• Salt 3006.10 (Sulfur)
• libgit2 1.9.0
• pygit2 1.18.0

[oogali]

reproduce.py

#!/opt/saltstack/salt/bin/python3.10

import pygit2
import sys


def main(argv=None):
    if argv is None:
        argv = sys.argv

    keypair = pygit2.Keypair('git', '/home/oogali/.ssh/id_testing.pub', '/home/oogali/.ssh/id_testing', '')
    callbacks = pygit2.RemoteCallbacks(credentials=keypair)

    repo_path = '[email protected]:libgit2/pygit2.git'
    pygit2.clone_repository(url=repo_path, path='/tmp/gitrepo', callbacks=callbacks)
    print('Cloned repository!')

    return 0

if __name__ == '__main__':
    sys.exit(main())

Without home directory defined

root@salt-master:~# unset HOME
root@salt-master:/root# ./reproduce.py
Traceback (most recent call last):
  File "/root/./reproduce.py", line 21, in <module>
    sys.exit(main())
  File "/root/./reproduce.py", line 15, in main
    pygit2.clone_repository(url=repo_path, path='/tmp/gitrepo', callbacks=callbacks)
  File "/opt/saltstack/salt/extras-3.10/pygit2/__init__.py", line 236, in clone_repository
    payload.check_error(err)
  File "/opt/saltstack/salt/extras-3.10/pygit2/callbacks.py", line 99, in check_error
    check_error(error_code)
  File "/opt/saltstack/salt/extras-3.10/pygit2/errors.py", line 66, in check_error
    raise GitError(message)
_pygit2.GitError: error loading known_hosts:

With home directory defined

root@salt-master:/root# export HOME=/root
root@salt-master:~# ./reproduce.py
Cloned repository!

[jdavid]

Thanks @oogali for the detailed analysis. From this my understanding is that:

• pygit2 should expose GIT_OPT_SET_HOMEDIR (and GIT_OPT_GET_HOMEDIR). Do you mind submitting a PR? In pygit2.enums.Option these constants can be found, but commented out.
• libgit2 maybe should raise a more informative error? ping @ethomson