[BUG] gitfs_remotes no longer work when upgrading from 3006.2 to 3006.3

[Foorack]

trafficstars

Description When running 3006.2, remote pillars via gitfs works. When upgrading to 3006.3 with the same config, a critical error is displayed.

[ERROR   ] gitfs is configured but could not be loaded, are pygit2 and libgit2 installed?
[CRITICAL] No suitable gitfs provider module is installed.

Setup Using Docker image saltstack/salt:3006.3

master config contains:

fileserver_backend:
  - roots
  - git

gitfs_provider: pygit2
gitfs_base: main
gitfs_remotes:
  - https://github.com/mypublicrepo/salt-config.git

gitfs_global_lock: False

Please be as specific as possible and give set-up details.

• [ ] on-prem machine
• [ ] VM (Virtualbox, KVM, etc. please specify)
• [ ] VM running on a cloud service, please be explicit and add details
• [X] container (Kubernetes, Docker, containerd, etc. please specify)
• [ ] or a combination, please be explicit
• [ ] jails if it is FreeBSD
• [ ] classic packaging
• [ ] onedir packaging
• [ ] used bootstrap to install

Steps to Reproduce the behavior docker-compose.yml

version: '3.7'

services:
  master:
    image: saltstack/salt:3006.3
    hostname: master
    command: salt-master -l info
    ports:
      - 4505:4505
      - 4506:4506
    environment:
      - PYTHONUNBUFFERED=true
    volumes:
      - ./etc/master:/etc/salt/master
      - ./salt:/srv/salt
      - ./pillar:/srv/pillar

[welcome[bot]]

Hi there! Welcome to the Salt Community! Thank you for making your first contribution. We have a lengthy process for issues and PRs. Someone from the Core Team will follow up as soon as possible. In the meantime, here’s some information that may help as you continue your Salt journey. Please be sure to review our Code of Conduct. Also, check out some of our community resources including:

• Community Wiki
• Salt’s Contributor Guide
• Join our Community Slack
• IRC on LiberaChat
• Salt Project YouTube channel
• Salt Project Twitch channel

There are lots of ways to get involved in our community. Every month, there are around a dozen opportunities to meet with other contributors and the Salt Core team and collaborate in real time. The best way to keep track is by subscribing to the Salt Community Events Calendar. If you have additional questions, email us at [email protected]. We’re glad you’ve joined our community and look forward to doing awesome things with you!

[OrangeDog]

are pygit2 and libgit2 installed?

Are they?

[Foorack]

are pygit2 and libgit2 installed?

Are they?

@OrangeDog Yes.

1. They were preinstalled and available in 3006.2 and no change has been made when trying to upgrade to 3006.3
2. It is fully reproducable. If I change back to 3006.2 image it works, if I change to 3006.3 it breaks again.
3. Even if I explicitly tell it to install the libs, it still fails:

   command: sh -c "pip3 install pygit2 && apk add libgit2 && salt-master -l info"

[Foorack]

This is the only change in the Docker image between 3006.2 and 3006.3

https://gitlab.com/saltstack/open/saltdocker/-/commit/6367726561cefe7604af4a1ecce7f022ec191e5e

[Foorack]

@felippeb

[Foorack]

Running pip3 show pygit2 in each container

3006.2 Docker Image:

salt-master-1  | Name: pygit2
salt-master-1  | Version: 1.6.1
salt-master-1  | Summary: Python bindings for libgit2.
salt-master-1  | Home-page: http://github.com/libgit2/pygit2
salt-master-1  | Author: 
salt-master-1  | Author-email: 
salt-master-1  | License: GPLv2 with linking exception
salt-master-1  | Location: /usr/local/lib/python3.10/site-packages
salt-master-1  | Requires: cffi
salt-master-1  | Required-by: 

3006.3 Docker Image:

salt-master-1  | Name: pygit2
salt-master-1  | Version: 1.6.1
salt-master-1  | Summary: Python bindings for libgit2.
salt-master-1  | Home-page: http://github.com/libgit2/pygit2
salt-master-1  | Author: 
salt-master-1  | Author-email: 
salt-master-1  | License: GPLv2 with linking exception
salt-master-1  | Location: /usr/local/lib/python3.10/site-packages
salt-master-1  | Requires: cffi
salt-master-1  | Required-by: 

Yet it works in 3006.2, but not in 3006.3.

[OrangeDog]

/ # python -c 'import pygit2'
Traceback (most recent call last):
  File "<string>", line 1, in <module>
  File "/usr/local/lib/python3.10/site-packages/pygit2/__init__.py", line 33, in <module>
    from .blame import Blame, BlameHunk
  File "/usr/local/lib/python3.10/site-packages/pygit2/blame.py", line 27, in <module>
    from .ffi import ffi, C
  File "/usr/local/lib/python3.10/site-packages/pygit2/ffi.py", line 27, in <module>
    from ._libgit2 import ffi, lib as C
ModuleNotFoundError: No module named 'pygit2._libgit2'

Looks like it didn't install properly. No indication of why in the build log: https://gitlab.com/saltstack/open/saltdocker/-/jobs/5040710657

[Foorack]

@OrangeDog salt --versions-report on both 3006.2 vs 3006.3.

For unknown reasons it is not detecting it is installed, even though it is. Or it is incorrectly installed in the image.

[OrangeDog]

The 3006.4 image has the same problem.

[Foorack]

Yes, we noticed this by upgrading from 3006.2 -> 3006.4. I then narrowed it down to breaking in 3006.3.

[OrangeDog]

It's possible the 3006.2 image would also fail if built anew.

[Foorack]

@OrangeDog We found it!

We verified internally and you are partially correct. Building 3006.2 on latest master fails, however building 3006.2 with the commit that was used to build it in the past succeeds. We've therefore identified that it is the change introduced in 6367726561cefe7604af4a1ecce7f022ec191e5e adding --no-build-isolation that is breaking pygit2.

You can try build the container twice - once with and once without - to independently verify this.

Why it breaks we don't know yet, but would it be possible to re-publish the 3006.3 and 3006.4 without this argument?

[OrangeDog]

Well that is strange, because --no-build-isolation is a compatibility option to make new versions of pip behave like old ones.

[Foorack]

Have not investigated why adding this flag breaks it, only confirmed that reverting that change fixes the problem. We are currently planning on building internal images so we can upgrade from .2 to .4, but would like to upstream this change as well.

I am considering making an issue on the Docker image Gitlab (why is it on GitLab?) (why so little activity?) (why is it not onedir?).

@felippeb Do you know why --no-build-isolation flag was added in the first place?

[tlex]

Still affected by it on 3006.6.

My "solution" was to downgrade to 3006.2 as well. I think I'll look into creating my own image.

[Foorack]

We are potentially looking into creating own image as well. This bug is preventing us from upgrading past 3006.2.

@OrangeDog @felippeb Is there any chance we could get this flag removed from the official image, as there isn't a documented reason why it is there?

[Foorack]

saltdocker Repository Now Archived

https://saltproject.io/blog/saltdocker-archival/

---

As the Salt Docker image has now been archived, I personally will use the following configuration instead:

services:
  master:
    image: docker.io/ubuntu:22.04
    command: /bin/bash -c "apt-get update && apt-get install curl -y && curl -L https://bootstrap.saltstack.com -o install_salt.sh && sh install_salt.sh -M -X onedir 3007.0 && salt-master -l info"
    hostname: master
    volumes:
      - ./docker/saltmaster/etc/master:/etc/salt/master:ro
      - ./docker/saltmaster/pillar:/srv/pillar:ro
      - ./docker/saltmaster/salt:/srv/salt:ro
    environment:
      - PYTHONUNBUFFERED=true

The command-line can be customized as needed.

As this issue was raised against a docker image which no longer is maintained, I will close it.