salt [BUG] job cache is saved locally even if job_cache is set to False in master config

[BUG] job cache is saved locally even if job_cache is set to False in master config

Open tomekrzymyszkiewicz opened this issue 9 months ago • 1 comments

Description The salt master config file field job_cache: False does not apply in the work of the system. The jobs are being saved to the job cache at salt master even if job_cache: False is set up in salt master and salt syndic.

Setup There are 2 salt masters iabc-saltmaster-1 and oabc-saltmaster-1 , salt syndic abc-saltsyndic and minions (for example minion-123).

Salt masters config (iabc-saltmaster-1 and oabc-saltmaster-1 have same configs) The salt masters /etc/salt/master - part with returner and job cache:


job_cache: False
ext_job_cache: pgjsonb
returner.pgjsonb.host: <postgres db FQDN>
returner.pgjsonb.sslmode: 'require'
returner.pgjsonb.user: <db username>
returner.pgjsonb.pass:  <db password>
returner.pgjsonb.db: <db name>
returner.pgjsonb.port: <db port>
extension_modules: /opt/salt_extension_modules
eauth_tokens: jwt
keep_jobs: 720
keep_jobs_seconds: 2592000
cachedir: /dev/shm/var/cache/salt/master

schedule:
  clean_old_jobs:
    run_on_start: true
    enabled: true
    function: pgjsonb.clean_old_jobs
    splay: 1800
    hours: 6

presence_events: True

The salt masters /etc/salt/minion

master:
 - <oabc-saltmaster-1 FQDN>
 - <iabc-saltmaster-1 FQDN>
metadata_server_grains: True

Salt syndic config (abc-saltsyndic) The salt syndic /etc/salt/master - part with returner and job cache:

job_cache: False
ext_job_cache: pgjsonb
returner.pgjsonb.host: <postgres db FQDN>
returner.pgjsonb.sslmode: 'require'
returner.pgjsonb.user: <db username>
returner.pgjsonb.pass: <db password>
returner.pgjsonb.db: <db name>
returner.pgjsonb.port: <db port>
extension_modules: /opt/salt_extension_modules
eauth_tokens: jwt
keep_jobs: 720
keep_jobs_seconds: 2592000
cachedir: /dev/shm/var/cache/salt/master

The salt syndic /etc/salt/minion

master:
  - <iabc-saltmaster-1 FQDN>
  - <oabc-saltmaster-1 FQDN>

The machines are VMs running on cloud service with onedir packaging

Steps to Reproduce the behavior

Find any minion assigned to syndic

root@abc-saltsyndic:~# salt-key -L
[DEBUG   ] Configuration file path: /etc/salt/master
[DEBUG   ] MasterEvent PUB socket URI: /var/run/salt/master/master_event_pub.ipc
[DEBUG   ] MasterEvent PULL socket URI: /var/run/salt/master/master_event_pull.ipc
[DEBUG   ] The functions from module 'key' are being loaded by dir() on the loaded module
[DEBUG   ] LazyLoaded key.output
Accepted Keys:
minion-123
Denied Keys:
Rejected Keys:

Run job in the minion

root@abc-saltsyndic:~# salt 'minion-123' test.ping --async
[DEBUG   ] Configuration file path: /etc/salt/master
[DEBUG   ] Reading configuration from /etc/salt/master
[DEBUG   ] Using cached minion ID from /etc/salt/minion_id: abc-saltsyndic
[DEBUG   ] Missing configuration file: /root/.saltrc
[DEBUG   ] MasterEvent PUB socket URI: /var/run/salt/master/master_event_pub.ipc
[DEBUG   ] MasterEvent PULL socket URI: /var/run/salt/master/master_event_pull.ipc
[DEBUG   ] Closing AsyncReqChannel instance
Executed command with job ID: 20240521124445278489

Check if job cache exists

root@abc-saltsyndic:~# find /dev/shm/var/cache/salt/master/jobs -type f -mmin -1
/dev/shm/var/cache/salt/master/jobs/37/3237afb5de707c329da87265ad69686867d1476d691c47110cac50c9354209/.minions.p
/dev/shm/var/cache/salt/master/jobs/37/3237afb5de707c329da87265ad69686867d1476d691c47110cac50c9354209/.load.p
/dev/shm/var/cache/salt/master/jobs/37/3237afb5de707c329da87265ad69686867d1476d691c47110cac50c9354209/jid

root@abc-saltsyndic:~# cat /dev/shm/var/cache/salt/master/jobs/37/3237afb5de707c329da87265ad69686867d1476d691c47110cac50c9354209/jid
20240521124445278489

As a resuktl there is jid file with job ID of the job and in the directory are files .minion.p and .load.p.

Expected behavior According to the docs (Managing the Job Cache) there should be only jid file for checking for and preventing JID collisions. The .minion.p and `.load.p. should not be created.

Versions Report

on `iabc-saltmaster-1`

Salt Version:
          Salt: 3006.5
 
Python Version:
        Python: 3.10.13 (main, Nov 15 2023, 04:34:27) [GCC 11.2.0]
 
Dependency Versions:
          cffi: 1.16.0
      cherrypy: unknown
      dateutil: 2.8.1
     docker-py: 1.10.6
         gitdb: 4.0.11
     gitpython: 3.1.40
        Jinja2: 3.1.2
       libgit2: 1.3.0
  looseversion: 1.0.2
      M2Crypto: Not Installed
          Mako: Not Installed
       msgpack: 1.0.2
  msgpack-pure: Not Installed
  mysql-python: Not Installed
     packaging: 22.0
     pycparser: 2.21
      pycrypto: 2.6.1
  pycryptodome: 3.9.8
        pygit2: 1.7.0
  python-gnupg: 0.4.8
        PyYAML: 6.0.1
         PyZMQ: 23.2.0
        relenv: 0.14.2
         smmap: 5.0.1
       timelib: 0.2.4
       Tornado: 4.5.3
           ZMQ: 4.3.4
 
System Versions:
          dist: ubuntu 20.04.5 focal
        locale: utf-8
       machine: x86_64
       release: 5.4.0-169-generic
        system: Linux
       version: Ubuntu 20.04.5 focal

on `oabc-saltmaster-1`

Salt Version:
          Salt: 3006.5
 
Python Version:
        Python: 3.10.13 (main, Nov 15 2023, 04:34:27) [GCC 11.2.0]
 
Dependency Versions:
          cffi: 1.14.6
      cherrypy: unknown
      dateutil: 2.8.1
     docker-py: 1.10.6
         gitdb: 4.0.11
     gitpython: 3.1.40
        Jinja2: 3.1.2
       libgit2: 1.3.0
  looseversion: 1.0.2
      M2Crypto: Not Installed
          Mako: Not Installed
       msgpack: 1.0.2
  msgpack-pure: Not Installed
  mysql-python: Not Installed
     packaging: 22.0
     pycparser: 2.21
      pycrypto: 2.6.1
  pycryptodome: 3.9.8
        pygit2: 1.7.0
  python-gnupg: 0.4.8
        PyYAML: 6.0.1
         PyZMQ: 23.2.0
        relenv: 0.14.2
         smmap: 5.0.1
       timelib: 0.2.4
       Tornado: 4.5.3
           ZMQ: 4.3.4
 
System Versions:
          dist: ubuntu 20.04.6 focal
        locale: utf-8
       machine: x86_64
       release: 5.4.0-150-generic
        system: Linux
       version: Ubuntu 20.04.6 focal

on `abc-saltsyndic`

Salt Version:
          Salt: 3006.5
 
Python Version:
        Python: 3.10.13 (main, Nov 15 2023, 04:34:27) [GCC 11.2.0]
 
Dependency Versions:
          cffi: 1.14.6
      cherrypy: unknown
      dateutil: 2.8.1
     docker-py: Not Installed
         gitdb: 4.0.11
     gitpython: 3.1.40
        Jinja2: 3.1.2
       libgit2: 1.3.0
  looseversion: 1.0.2
      M2Crypto: Not Installed
          Mako: Not Installed
       msgpack: 1.0.2
  msgpack-pure: Not Installed
  mysql-python: Not Installed
     packaging: 22.0
     pycparser: 2.21
      pycrypto: Not Installed
  pycryptodome: 3.9.8
        pygit2: 1.7.0
  python-gnupg: 0.4.8
        PyYAML: 6.0.1
         PyZMQ: 23.2.0
        relenv: 0.14.2
         smmap: 5.0.1
       timelib: 0.2.4
       Tornado: 4.5.3
           ZMQ: 4.3.4
 
System Versions:
          dist: ubuntu 20.04.5 focal
        locale: utf-8
       machine: x86_64
       release: 5.4.0-169-generic
        system: Linux
       version: Ubuntu 20.04.5 focal

Additional context Because of need to obfuscate sensitive data all values in triangular brackets <> have been described and the value is not shared.

May 21 '24 13:05 tomekrzymyszkiewicz

salt salt copied to clipboard

[BUG] job cache is saved locally even if job_cache is set to False in master config

salt
salt copied to clipboard