is there a way to use git fileserver backend to sync the salt-master repo from github behind a proxy?

[ccrivelli]

trafficstars

I'm currently testing the gitfs provider 'pygit2' with https authentication. My salt-master is behind a squid proxy.

• I properly configured the env vars 'http_proxy' and 'https_proxy' (as root user and salt-master is running as root)
• a manual 'git clone' command works fine

But the salt-master sync is not able to contact my github repo:

#  salt-run fileserver.update
[ERROR   ] Error occured fetching gitfs remote 'https://github.com/myrepo.git': Failed to connect to github.com: Connection timed out
True

And checking the squid proxy logs, It sounds like salt-master is not going through the proxy.

Is there a way to set up this proxy config in the 'gitfs_remote' config?

Eventually I know I could do it using SSH, but I was wonder If it can be done using https.

Thanks

[jfindlay]

@ccrivelli, thanks for the report. Similar to #23617.

[stale[bot]]

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.

If this issue is closed prematurely, please leave a comment and we will gladly reopen the issue.

[ptitdoc]

Hello,

I did not manage to use gitfs + pygit2 + http/https web proxy either.

[ptitdoc]

It has been apparently fixed upstream : https://github.com/libgit2/pygit2/pull/1063/files

With this pygit2 patch, forcing proxy detection work. It may require adaptation in salt to make proxy automatic (add argument proxy=True in fetch call).

[dbethke]

Can this issue be re-opened? I'm currently hitting this issue too and ptitdoc mentioned above it was fixed upstream in pygit2, but perhaps something in salt needs to be adjusted to use it.

[dbethke]

My environment requires an http proxy to reach github where the repo lives and my gitfs_remotes is configured with the https url. My salt master logs show timeouts trying to reach the server yet my squid proxy logs don't show the requests were even sent to it.

I've tried setting environment var HTTP_PROXY and HTTPS_PROXY as well as tried the git config --global http.proxy and git config --global https.proxy settings. I am able to manually git clone the repo as root user--which salt is running as, but I can't get salt-master to do it.

Additional discussion/troubleshooting on your slack channel https://saltstackcommunity.slack.com/archives/C7K04SEJC/p1625155626188500

# salt-master --versions
Salt Version:
          Salt: 3003.1

Dependency Versions:
          cffi: 1.14.5
      cherrypy: 18.6.0
      dateutil: Not Installed
     docker-py: Not Installed
         gitdb: 4.0.7
     gitpython: 3.1.18
        Jinja2: 2.11.1
       libgit2: 1.1.0
      M2Crypto: 0.35.2
          Mako: Not Installed
       msgpack: 0.6.2
  msgpack-pure: Not Installed
  mysql-python: Not Installed
     pycparser: 2.20
      pycrypto: Not Installed
  pycryptodome: Not Installed
        pygit2: 1.6.1
        Python: 3.6.8 (default, Nov 16 2020, 16:55:22)
  python-gnupg: Not Installed
        PyYAML: 3.13
         PyZMQ: 17.0.0
         smmap: 4.0.0
       timelib: Not Installed
       Tornado: 4.5.3
           ZMQ: 4.1.4

System Versions:
          dist: centos 7 Core
        locale: UTF-8
       machine: x86_64
       release: 3.10.0-1160.24.1.el7.x86_64
        system: Linux
       version: CentOS Linux 7 Core

[OrangeDog]

It may require adaptation in salt to make proxy automatic (add argument proxy=True in fetch call).

It looks like it, yes. Why they didn't make it automatic by default I don't know.

Just to note, you need to set the environment for the salt-master/salt-minion service, not the call to salt or salt-call nor the root user's shell. How you do that depends on the service implementation.

For systemd create a file e.g. /etc/systemd/system/salt-master.service.d/proxy.conf:

[Service]
Environment=http_proxy=http://<myproxyserver>:<port>
Environment=https_proxy=https://<myproxyserver>:<port>

[ptitdoc]

I confirm I had to change the following line (in addition to setting http_proxy and https_proxy environment variables): https://github.com/saltstack/salt/blob/306aa6dd2948e8d149fa65f51e9959a2e464a930/salt/utils/gitfs.py#L1857

To:

fetch_kwargs = {"proxy": True}

[DaAwesomeP]

Any chance of {"proxy": True} being set as the default since it allows for auto detection via env vars? And/or for a gitfs_proxy option to be added?

[YuhiAida]

Hi, after setting the

For systemd create a file e.g. /etc/systemd/system/salt-master.service.d/proxy.conf:

[Service]
Environment=http_proxy=http://<myproxyserver>:<port>
Environment=https_proxy=https://<myproxyserver>:<port>

and

fetch_kwargs = {"proxy": True}

I get :

[ERROR   ] Exception caught while fetching gitfs remote 'https://github.com/saltstack-formulas/salt-formula.git': fetch() got an unexpected keyword argument 'proxy'
Traceback (most recent call last):
  File "/usr/lib/python3.6/site-packages/salt/utils/gitfs.py", line 2536, in fetch_remotes
    if repo.fetch():
  File "/usr/lib/python3.6/site-packages/salt/utils/gitfs.py", line 819, in fetch
    return self._fetch()
  File "/usr/lib/python3.6/site-packages/salt/utils/gitfs.py", line 1870, in _fetch
    fetch_results = origin.fetch(**fetch_kwargs)
TypeError: fetch() got an unexpected keyword argument 'proxy'

Am I the only one?

[OrangeDog]

@YuhiAida you also need pygit2 >= 1.9.0