salt [BUG] win_pkg fails to find any packages if cachedir is relocated to "hidden" folder

[BUG] win_pkg fails to find any packages if cachedir is relocated to "hidden" folder

Open yangskyboxlabs opened this issue 6 months ago • 1 comments

Description In a masterless minion, if cachedir is configured to be part of a "hidden" folder (begins with .), win_pkg will fail to find any packages.

Setup (Please provide relevant configs and/or SLS files (be sure to remove sensitive info. There is no general set-up of Salt.)

Please be as specific as possible and give set-up details.

[x] on-prem machine
[ ] VM (Virtualbox, KVM, etc. please specify)
[ ] VM running on a cloud service, please be explicit and add details
[ ] container (Kubernetes, Docker, containerd, etc. please specify)
[ ] or a combination, please be explicit
[ ] jails if it is FreeBSD
[ ] classic packaging
[ ] onedir packaging
[ ] used bootstrap to install

Steps to Reproduce the behavior (My winrepo-ng is a fork that contains far fewer packages, but is confirmed to be working in other contexts)

# minion conf

file_client: local

root_dir: .
cachedir: D:/infra/salt-config-skyboxlabs-pitch/.cache

winrepo_remotes: []
winrepo_remotes_ng: []

PS D:\infra\salt-config-skyboxlabs-pitch> salt-call -c .local -l debug pkg.refresh_db
[DEBUG   ] Reading configuration from D:\infra\salt-config-skyboxlabs-pitch\.local\minion
[DEBUG   ] Including configuration from 'D:\infra\salt-config-skyboxlabs-pitch\.local\minion.d\grain_test.conf'
[DEBUG   ] Reading configuration from D:\infra\salt-config-skyboxlabs-pitch\.local\minion.d\grain_test.conf
[DEBUG   ] Including configuration from 'D:\infra\salt-config-skyboxlabs-pitch\.local\minion.d\roots.conf'
[DEBUG   ] Reading configuration from D:\infra\salt-config-skyboxlabs-pitch\.local\minion.d\roots.conf
[WARNING ] Insecure logging configuration detected! Sensitive data may be logged.
[DEBUG   ] Configuration file path: D:\infra\salt-config-skyboxlabs-pitch\.local\minion
[DEBUG   ] Grains refresh requested. Refreshing grains.
[DEBUG   ] Reading configuration from D:\infra\salt-config-skyboxlabs-pitch\.local\minion
[DEBUG   ] Including configuration from 'D:\infra\salt-config-skyboxlabs-pitch\.local\minion.d\grain_test.conf'
[DEBUG   ] Reading configuration from D:\infra\salt-config-skyboxlabs-pitch\.local\minion.d\grain_test.conf
[DEBUG   ] Including configuration from 'D:\infra\salt-config-skyboxlabs-pitch\.local\minion.d\roots.conf'
[DEBUG   ] Reading configuration from D:\infra\salt-config-skyboxlabs-pitch\.local\minion.d\roots.conf
[DEBUG   ] The functions from module 'core' are being loaded by dir() on the loaded module
[DEBUG   ] The functions from module 'disks' are being loaded by dir() on the loaded module
[DEBUG   ] The functions from module 'extra' are being loaded by dir() on the loaded module
[DEBUG   ] The functions from module 'lvm' are being loaded by dir() on the loaded module
[DEBUG   ] The functions from module 'mdadm' are being loaded by dir() on the loaded module
[DEBUG   ] The functions from module 'minion_process' are being loaded by dir() on the loaded module
[DEBUG   ] The functions from module 'opts' are being loaded by dir() on the loaded module
[DEBUG   ] The functions from module 'package' are being loaded by dir() on the loaded module
[DEBUG   ] The functions from module 'pending_reboot' are being loaded by dir() on the loaded module
[DEBUG   ] Override  __utils__: <module 'salt.loaded.int.grains.zfs' from 'C:\\Program Files\\Salt Project\\Salt\\Lib\\site-packages\\salt\\grains\\zfs.py'>
[DEBUG   ] Determining pillar cache
[DEBUG   ] The functions from module 'roots' are being loaded by dir() on the loaded module
[DEBUG   ] LazyLoaded roots.envs
[DEBUG   ] Popen(['git', 'version'], cwd=D:\infra\salt-config-skyboxlabs-pitch, stdin=None, shell=False, universal_newlines=False)
[DEBUG   ] Popen(['git', 'version'], cwd=D:\infra\salt-config-skyboxlabs-pitch, stdin=None, shell=False, universal_newlines=False)
[DEBUG   ] The functions from module 's3fs' are being loaded by dir() on the loaded module
[DEBUG   ] Could not LazyLoad roots.init: 'roots.init' is not available.
[DEBUG   ] Updating roots fileserver cache
[DEBUG   ] The functions from module 'jinja' are being loaded by dir() on the loaded module
[DEBUG   ] LazyLoaded jinja.render
[DEBUG   ] The functions from module 'yaml' are being loaded by dir() on the loaded module
[DEBUG   ] LazyLoaded yaml.render
[DEBUG   ] compile template: .local/pillar\top.sls
[DEBUG   ] Jinja search path: ['.local/pillar', 'pillar']
[PROFILE ] Time (in seconds) to render '.local/pillar\top.sls' using 'jinja' renderer: 0.0010001659393310547
[PROFILE ] Time (in seconds) to render '.local/pillar\top.sls' using 'yaml' renderer: 0.0
[DEBUG   ] The functions from module 'confirm_top' are being loaded by dir() on the loaded module
[DEBUG   ] LazyLoaded confirm_top.confirm_top
[DEBUG   ] The functions from module 'compound_match' are being loaded by dir() on the loaded module
[DEBUG   ] LazyLoaded compound_match.match
[DEBUG   ] compound_match: __local__ ? *
[DEBUG   ] The functions from module 'glob_match' are being loaded by dir() on the loaded module
[DEBUG   ] LazyLoaded glob_match.match
[DEBUG   ] compound_match __local__ ? "*" => "True"
[DEBUG   ] compile template: pillar\fastbuild/worker.sls
[DEBUG   ] Jinja search path: ['.local/pillar', 'pillar']
[PROFILE ] Time (in seconds) to render 'pillar\fastbuild/worker.sls' using 'jinja' renderer: 0.0010001659393310547
[PROFILE ] Time (in seconds) to render 'pillar\fastbuild/worker.sls' using 'yaml' renderer: 0.0
[DEBUG   ] compile template: pillar\teamcity/agent.sls
[DEBUG   ] Jinja search path: ['.local/pillar', 'pillar']
[PROFILE ] Time (in seconds) to render 'pillar\teamcity/agent.sls' using 'jinja' renderer: 0.0019996166229248047
[PROFILE ] Time (in seconds) to render 'pillar\teamcity/agent.sls' using 'yaml' renderer: 0.0010001659393310547
[DEBUG   ] The functions from module 'jinja' are being loaded by dir() on the loaded module
[DEBUG   ] LazyLoaded jinja.render
[DEBUG   ] The functions from module 'yaml' are being loaded by dir() on the loaded module
[DEBUG   ] LazyLoaded yaml.render
[DEBUG   ] Failed to import module kernelpkg_linux_yum:
Traceback (most recent call last):
  File "C:\Program Files\Salt Project\Salt\Lib\site-packages\salt\loader\lazy.py", line 806, in _load_module
    spec.loader.exec_module(mod)
  File "<frozen importlib._bootstrap_external>", line 883, in exec_module
  File "<frozen importlib._bootstrap>", line 241, in _call_with_frames_removed
  File "C:\Program Files\Salt Project\Salt\Lib\site-packages\salt\modules\kernelpkg_linux_yum.py", line 8, in <module>
    import salt.modules.yumpkg
ModuleNotFoundError: No module named 'salt.modules.yumpkg'
[DEBUG   ] The functions from module 'pkg_resource' are being loaded by dir() on the loaded module
[DEBUG   ] The functions from module 'pkg' are being loaded by dir() on the loaded module
[DEBUG   ] LazyLoaded pkg.refresh_db
[DEBUG   ] The functions from module 'direct_call' are being loaded by dir() on the loaded module
[DEBUG   ] LazyLoaded direct_call.execute
[DEBUG   ] Refreshing pkg metadata db for saltenv 'base' (age of existing metadata is -1 day, 23:59:59)
[INFO    ] Removing all *.sls files under 'D:/infra/salt-config-skyboxlabs-pitch/.cache\files\base\win\repo-ng'
[DEBUG   ] The functions from module 'roots' are being loaded by dir() on the loaded module
[DEBUG   ] LazyLoaded roots.envs
[DEBUG   ] Removed base file list cache for saltenv 'roots'
[INFO    ] Fetching *.sls files from salt://win/repo-ng/
[DEBUG   ] The functions from module 'cp' are being loaded by dir() on the loaded module
[DEBUG   ] LazyLoaded cp.cache_dir
[DEBUG   ] The functions from module 'roots' are being loaded by dir() on the loaded module
[DEBUG   ] LazyLoaded roots.envs
[DEBUG   ] The functions from module 's3fs' are being loaded by dir() on the loaded module
[DEBUG   ] Could not LazyLoad roots.init: 'roots.init' is not available.
[DEBUG   ] Updating roots fileserver cache
[INFO    ] Caching directory 'win/repo-ng/' for environment 'base'
[DEBUG   ] salt environment 'base' maps to __env__ file_roots directory
[DEBUG   ] salt environment 'base' maps to __env__ file_roots directory
[DEBUG   ] In saltenv 'base', looking at rel_path 'win/repo-ng/salt-winrepo-skyboxlabs/java/amazon-corretto.sls' to resolve 'salt://win/repo-ng/salt-winrepo-skyboxlabs/java/amazon-corretto.sls'
[DEBUG   ] In saltenv 'base', ** considering ** path 'D:\infra\salt-config-skyboxlabs-pitch\.cache\files\base\win\repo-ng\salt-winrepo-skyboxlabs\java\amazon-corretto.sls' to resolve 'salt://win/repo-ng/salt-winrepo-skyboxlabs/java/amazon-corretto.sls'
[DEBUG   ] Fetching file from saltenv 'base', ** attempting ** 'salt://win/repo-ng/salt-winrepo-skyboxlabs/java/amazon-corretto.sls'
[DEBUG   ] No dest file found
[DEBUG   ] salt environment 'base' maps to __env__ file_roots directory
[DEBUG   ] salt environment 'base' maps to __env__ file_roots directory
[DEBUG   ] salt environment 'base' maps to __env__ file_roots directory
[DEBUG   ] salt environment 'base' maps to __env__ file_roots directory
[INFO    ] Fetching file from saltenv 'base', ** done ** 'win/repo-ng/salt-winrepo-skyboxlabs/java/amazon-corretto.sls'
[DEBUG   ] salt environment 'base' maps to __env__ file_roots directory
[DEBUG   ] In saltenv 'base', looking at rel_path 'win/repo-ng/salt-winrepo-skyboxlabs/prometheus/windows_exporter.sls' to resolve 'salt://win/repo-ng/salt-winrepo-skyboxlabs/prometheus/windows_exporter.sls'
[DEBUG   ] In saltenv 'base', ** considering ** path 'D:\infra\salt-config-skyboxlabs-pitch\.cache\files\base\win\repo-ng\salt-winrepo-skyboxlabs\prometheus\windows_exporter.sls' to resolve 'salt://win/repo-ng/salt-winrepo-skyboxlabs/prometheus/windows_exporter.sls'
[DEBUG   ] Fetching file from saltenv 'base', ** attempting ** 'salt://win/repo-ng/salt-winrepo-skyboxlabs/prometheus/windows_exporter.sls'
[DEBUG   ] No dest file found
[DEBUG   ] salt environment 'base' maps to __env__ file_roots directory
[DEBUG   ] salt environment 'base' maps to __env__ file_roots directory
[DEBUG   ] salt environment 'base' maps to __env__ file_roots directory
[DEBUG   ] salt environment 'base' maps to __env__ file_roots directory
[INFO    ] Fetching file from saltenv 'base', ** done ** 'win/repo-ng/salt-winrepo-skyboxlabs/prometheus/windows_exporter.sls'
[DEBUG   ] salt environment 'base' maps to __env__ file_roots directory
[DEBUG   ] In saltenv 'base', looking at rel_path 'win/repo-ng/salt-winrepo-skyboxlabs/salt-minion.sls' to resolve 'salt://win/repo-ng/salt-winrepo-skyboxlabs/salt-minion.sls'
[DEBUG   ] In saltenv 'base', ** considering ** path 'D:\infra\salt-config-skyboxlabs-pitch\.cache\files\base\win\repo-ng\salt-winrepo-skyboxlabs\salt-minion.sls' to resolve 'salt://win/repo-ng/salt-winrepo-skyboxlabs/salt-minion.sls'
[DEBUG   ] Fetching file from saltenv 'base', ** attempting ** 'salt://win/repo-ng/salt-winrepo-skyboxlabs/salt-minion.sls'
[DEBUG   ] No dest file found
[DEBUG   ] salt environment 'base' maps to __env__ file_roots directory
[DEBUG   ] salt environment 'base' maps to __env__ file_roots directory
[DEBUG   ] salt environment 'base' maps to __env__ file_roots directory
[DEBUG   ] salt environment 'base' maps to __env__ file_roots directory
[INFO    ] Fetching file from saltenv 'base', ** done ** 'win/repo-ng/salt-winrepo-skyboxlabs/salt-minion.sls'
[DEBUG   ] salt environment 'base' maps to __env__ file_roots directory
[DEBUG   ] In saltenv 'base', looking at rel_path 'win/repo-ng/salt-winrepo-skyboxlabs/visualstudio/visualstudio-installer.sls' to resolve 'salt://win/repo-ng/salt-winrepo-skyboxlabs/visualstudio/visualstudio-installer.sls'
[DEBUG   ] In saltenv 'base', ** considering ** path 'D:\infra\salt-config-skyboxlabs-pitch\.cache\files\base\win\repo-ng\salt-winrepo-skyboxlabs\visualstudio\visualstudio-installer.sls' to resolve 'salt://win/repo-ng/salt-winrepo-skyboxlabs/visualstudio/visualstudio-installer.sls'
[DEBUG   ] Fetching file from saltenv 'base', ** attempting ** 'salt://win/repo-ng/salt-winrepo-skyboxlabs/visualstudio/visualstudio-installer.sls'
[DEBUG   ] No dest file found
[DEBUG   ] salt environment 'base' maps to __env__ file_roots directory
[DEBUG   ] salt environment 'base' maps to __env__ file_roots directory
[DEBUG   ] salt environment 'base' maps to __env__ file_roots directory
[DEBUG   ] salt environment 'base' maps to __env__ file_roots directory
[INFO    ] Fetching file from saltenv 'base', ** done ** 'win/repo-ng/salt-winrepo-skyboxlabs/visualstudio/visualstudio-installer.sls'
[DEBUG   ] Skipping files in directory: D:/infra/salt-config-skyboxlabs-pitch/.cache\files\base\win\repo-ng
[DEBUG   ] Skipping files in directory: D:/infra/salt-config-skyboxlabs-pitch/.cache\files\base\win\repo-ng\salt-winrepo-skyboxlabs
[DEBUG   ] Skipping files in directory: D:/infra/salt-config-skyboxlabs-pitch/.cache\files\base\win\repo-ng\salt-winrepo-skyboxlabs\java
[DEBUG   ] Skipping files in directory: D:/infra/salt-config-skyboxlabs-pitch/.cache\files\base\win\repo-ng\salt-winrepo-skyboxlabs\prometheus
[DEBUG   ] Skipping files in directory: D:/infra/salt-config-skyboxlabs-pitch/.cache\files\base\win\repo-ng\salt-winrepo-skyboxlabs\visualstudio
[DEBUG   ] The functions from module 'nested' are being loaded by dir() on the loaded module
[DEBUG   ] LazyLoaded nested.output
local:
    ----------
    failed:
        0
    success:
        0
    total:
        0

Expected behavior win_pkg should succeed in finding packages.

Versions Report

salt --versions-report

Salt Version:
          Salt: 3007.1

Python Version:
        Python: 3.10.14 (heads/main:c1ec015, Apr  3 2024, 21:36:37) [MSC v.1938 64 bit (AMD64)]

Dependency Versions:
          cffi: 1.16.0
      cherrypy: 18.8.0
      dateutil: 2.8.2
     docker-py: Not Installed
         gitdb: 4.0.10
     gitpython: 3.1.43
        Jinja2: 3.1.4
       libgit2: Not Installed
  looseversion: 1.3.0
      M2Crypto: Not Installed
          Mako: Not Installed
       msgpack: 1.0.7
  msgpack-pure: Not Installed
  mysql-python: Not Installed
     packaging: 23.1
     pycparser: 2.21
      pycrypto: Not Installed
  pycryptodome: 3.19.1
        pygit2: Not Installed
  python-gnupg: 0.5.2
        PyYAML: 6.0.1
         PyZMQ: 25.1.2
        relenv: 0.16.0
         smmap: 5.0.1
       timelib: 0.3.0
       Tornado: 6.3.3
           ZMQ: 4.3.4

Salt Package Information:
  Package Type: onedir

System Versions:
          dist:
        locale: utf-8
       machine: AMD64
       release: 10
        system: Windows
       version: 10 10.0.19045 SP0 Multiprocessor Free

Additional context The same setup with a different cachedir works as expected:

PS D:\infra\salt-config-skyboxlabs-pitch> salt-call -c .local -l info pkg.refresh_db
[INFO    ] Removing all *.sls files under 'D:/infra/salt-config-skyboxlabs-pitch/_cache\files\base\win\repo-ng'
[INFO    ] Fetching *.sls files from salt://win/repo-ng/
[INFO    ] Caching directory 'win/repo-ng/' for environment 'base'
[INFO    ] Fetching file from saltenv 'base', ** done ** 'win/repo-ng/salt-winrepo-skyboxlabs/java/amazon-corretto.sls'
[INFO    ] Fetching file from saltenv 'base', ** done ** 'win/repo-ng/salt-winrepo-skyboxlabs/prometheus/windows_exporter.sls'
[INFO    ] Fetching file from saltenv 'base', ** done ** 'win/repo-ng/salt-winrepo-skyboxlabs/salt-minion.sls'
[INFO    ] Fetching file from saltenv 'base', ** done ** 'win/repo-ng/salt-winrepo-skyboxlabs/visualstudio/visualstudio-installer.sls'
local:
    ----------
    failed:
        0
    success:
        4
    total:
        4

This is due to an overly aggressive filter in win_pkg.genrepo().

        # Skip hidden directories (.git)
        if re.search(r"[\\/]\..*", root):
            log.debug("Skipping files in directory: %s", root)
            continue

An alternative implementation is to filter such directories as part of walk():

    for root, dirs, files in salt.utils.path.os_walk(
        repo_details.local_dest, followlinks=False
    ):

        # Skip hidden directories (.git/, et al)
        # (this is concise, but loses logging information)
        dirs[:] = [d for d in dirs if not d.startswith(".")]

Aug 15 '24 21:08 yangskyboxlabs

salt salt copied to clipboard

[BUG] win_pkg fails to find any packages if cachedir is relocated to "hidden" folder

salt
salt copied to clipboard