semaphore 'could not find job' on async_status with delegate

I've a playbook where I execute an async task (by using async and poll: 0) and later on wait for it with async_status. Both tasks are delegated to localhost. When executing the playbook on command line everything works as expected. But when executing it via ansible-semaphore I get a 'could not find job' error message from the async_status task. Debug output shows that the first task safes the result file at /tmp/semaphore/.ansible_async/119402064606.1077895 while the async_wait looks at /home/semaphore/.ansible_async/119402064606.1077895. Obviously this is a mismatch. But I was not able to figure out what causes it.

Sample playbook:

- hosts: all
  gather_facts: no
  vars:
    delegate_to_host: localhost
  tasks:
    - command: sleep 5
      async: 7
      poll: 0
      register: async_task
      delegate_to: "{{delegate_to_host}}"
    - async_status:
        jid: "{{async_task.ansible_job_id}}"
      register: job_result
      until: job_result.finished
      retries: 7
      delay: 1
      delegate_to: "{{delegate_to_host}}"

Execution on command line with ansible-playbook -i foo, async-delegate.yml -v works (even on the ansible-semaphore machine under the same user (namely 'semaphore') as ansible-semaphore runs). The results_file is stored under /home/semaphore/.ansible_async/.

Output

semaphore@ansible:/tmp/semaphore/repository_1_1$ ansible-playbook -i ../inventory_2147483559 async-delegate-test.yml -vvvv
ansible-playbook [core 2.12.4]
  config file = /etc/ansible/ansible.cfg
  configured module search path = ['/tmp/semaphore/.ansible/plugins/modules', '/usr/share/ansible/plugins/modules']
  ansible python module location = /usr/lib/python3/dist-packages/ansible
  ansible collection location = /tmp/semaphore/.ansible/collections:/usr/share/ansible/collections
  executable location = /usr/bin/ansible-playbook
  python version = 3.8.10 (default, Mar 15 2022, 12:22:08) [GCC 9.4.0]
  jinja version = 2.10.1
  libyaml = True
Using /etc/ansible/ansible.cfg as config file
setting up inventory plugins
host_list declined parsing /tmp/semaphore/inventory_2147483559 as it did not pass its verify_file() method
script declined parsing /tmp/semaphore/inventory_2147483559 as it did not pass its verify_file() method
auto declined parsing /tmp/semaphore/inventory_2147483559 as it did not pass its verify_file() method
Parsed /tmp/semaphore/inventory_2147483559 inventory source with ini plugin
Loading callback plugin default of type stdout, v2.0 from /usr/lib/python3/dist-packages/ansible/plugins/callback/default.py
Skipping callback 'default', as we already have a stdout callback.
Skipping callback 'minimal', as we already have a stdout callback.
Skipping callback 'oneline', as we already have a stdout callback.

PLAYBOOK: async-delegate-test.yml **********************************************
Positional arguments: async-delegate-test.yml
verbosity: 4
connection: smart
timeout: 10
become_method: sudo
tags: ('all',)
inventory: ('/tmp/semaphore/inventory_2147483559',)
forks: 5
1 plays in async-delegate-test.yml

PLAY [all] *********************************************************************
META: ran handlers

TASK [command] *****************************************************************
task path: /tmp/semaphore/repository_1_1/async-delegate-test.yml:6
<localhost> ESTABLISH LOCAL CONNECTION FOR USER: semaphore
<localhost> EXEC /bin/sh -c 'echo ~semaphore && sleep 0'
<localhost> EXEC /bin/sh -c '( umask 77 && mkdir -p "` echo /home/semaphore/.ansible/tmp `"&& mkdir "` echo /home/semaphore/.ansible/tmp/ansible-tmp-1653305100.1865366-1681690-184103346183749 `" && echo ansible-tmp-1653305100.1865366-1681690-184103346183749="` echo /home/semaphore/.ansible/tmp/ansible-tmp-1653305100.1865366-1681690-184103346183749 `" ) && sleep 0'
Using module file /usr/lib/python3/dist-packages/ansible/modules/command.py
<localhost> PUT /tmp/ansible-test/ansible-local-1681685kzrx868d/tmp1hz93fh1 TO /home/semaphore/.ansible/tmp/ansible-tmp-1653305100.1865366-1681690-184103346183749/AnsiballZ_command.py
<localhost> PUT /tmp/ansible-test/ansible-local-1681685kzrx868d/tmp45rpplmn TO /home/semaphore/.ansible/tmp/ansible-tmp-1653305100.1865366-1681690-184103346183749/async_wrapper.py
<localhost> EXEC /bin/sh -c 'chmod u+x /home/semaphore/.ansible/tmp/ansible-tmp-1653305100.1865366-1681690-184103346183749/ /home/semaphore/.ansible/tmp/ansible-tmp-1653305100.1865366-1681690-184103346183749/AnsiballZ_command.py /home/semaphore/.ansible/tmp/ansible-tmp-1653305100.1865366-1681690-184103346183749/async_wrapper.py && sleep 0'
<localhost> EXEC /bin/sh -c 'ANSIBLE_ASYNC_DIR='"'"'~/.ansible_async'"'"' /usr/bin/python3 /home/semaphore/.ansible/tmp/ansible-tmp-1653305100.1865366-1681690-184103346183749/async_wrapper.py 169775103514 7 /home/semaphore/.ansible/tmp/ansible-tmp-1653305100.1865366-1681690-184103346183749/AnsiballZ_command.py _ && sleep 0'
changed: [bar -> localhost] => {
    "ansible_job_id": "169775103514.1681709",
    "changed": true,
    "finished": 0,
    "results_file": "/home/semaphore/.ansible_async/169775103514.1681709",
    "started": 1
}

TASK [async_status] ************************************************************
task path: /tmp/semaphore/repository_1_1/async-delegate-test.yml:11
<localhost> ESTABLISH LOCAL CONNECTION FOR USER: semaphore
<localhost> EXEC /bin/sh -c 'echo ~semaphore && sleep 0'
<localhost> EXEC /bin/sh -c 'echo ~semaphore && sleep 0'
<localhost> EXEC /bin/sh -c '( umask 77 && mkdir -p "` echo /home/semaphore/.ansible/tmp `"&& mkdir "` echo /home/semaphore/.ansible/tmp/ansible-tmp-1653305100.6461701-1681718-111069481504310 `" && echo ansible-tmp-1653305100.6461701-1681718-111069481504310="` echo /home/semaphore/.ansible/tmp/ansible-tmp-1653305100.6461701-1681718-111069481504310 `" ) && sleep 0'
Using module file /usr/lib/python3/dist-packages/ansible/modules/async_status.py
<localhost> PUT /tmp/ansible-test/ansible-local-1681685kzrx868d/tmp7rfaape0 TO /home/semaphore/.ansible/tmp/ansible-tmp-1653305100.6461701-1681718-111069481504310/AnsiballZ_async_status.py
<localhost> EXEC /bin/sh -c 'chmod u+x /home/semaphore/.ansible/tmp/ansible-tmp-1653305100.6461701-1681718-111069481504310/ /home/semaphore/.ansible/tmp/ansible-tmp-1653305100.6461701-1681718-111069481504310/AnsiballZ_async_status.py && sleep 0'
<localhost> EXEC /bin/sh -c '/usr/bin/python3 /home/semaphore/.ansible/tmp/ansible-tmp-1653305100.6461701-1681718-111069481504310/AnsiballZ_async_status.py && sleep 0'
<localhost> EXEC /bin/sh -c 'rm -f -r /home/semaphore/.ansible/tmp/ansible-tmp-1653305100.6461701-1681718-111069481504310/ > /dev/null 2>&1 && sleep 0'
FAILED - RETRYING: [bar -> localhost]: async_status (7 retries left).Result was: {
    "ansible_job_id": "169775103514.1681709",
    "attempts": 1,
    "changed": false,
    "finished": 0,
    "invocation": {
        "module_args": {
            "_async_dir": "/home/semaphore/.ansible_async",
            "jid": "169775103514.1681709",
            "mode": "status"
        }
    },
    "results_file": "/home/semaphore/.ansible_async/169775103514.1681709",
    "retries": 8,
    "started": 1,
    "stderr": "",
    "stderr_lines": [],
    "stdout": "",
    "stdout_lines": []
}
<localhost> EXEC /bin/sh -c 'echo ~semaphore && sleep 0'
<localhost> EXEC /bin/sh -c 'echo ~semaphore && sleep 0'
<localhost> EXEC /bin/sh -c '( umask 77 && mkdir -p "` echo /home/semaphore/.ansible/tmp `"&& mkdir "` echo /home/semaphore/.ansible/tmp/ansible-tmp-1653305101.9730003-1681718-186046040262823 `" && echo ansible-tmp-1653305101.9730003-1681718-186046040262823="` echo /home/semaphore/.ansible/tmp/ansible-tmp-1653305101.9730003-1681718-186046040262823 `" ) && sleep 0'
Using module file /usr/lib/python3/dist-packages/ansible/modules/async_status.py
<localhost> PUT /tmp/ansible-test/ansible-local-1681685kzrx868d/tmpg10k05w3 TO /home/semaphore/.ansible/tmp/ansible-tmp-1653305101.9730003-1681718-186046040262823/AnsiballZ_async_status.py
<localhost> EXEC /bin/sh -c 'chmod u+x /home/semaphore/.ansible/tmp/ansible-tmp-1653305101.9730003-1681718-186046040262823/ /home/semaphore/.ansible/tmp/ansible-tmp-1653305101.9730003-1681718-186046040262823/AnsiballZ_async_status.py && sleep 0'
<localhost> EXEC /bin/sh -c '/usr/bin/python3 /home/semaphore/.ansible/tmp/ansible-tmp-1653305101.9730003-1681718-186046040262823/AnsiballZ_async_status.py && sleep 0'
<localhost> EXEC /bin/sh -c 'rm -f -r /home/semaphore/.ansible/tmp/ansible-tmp-1653305101.9730003-1681718-186046040262823/ > /dev/null 2>&1 && sleep 0'
FAILED - RETRYING: [bar -> localhost]: async_status (6 retries left).Result was: {
    "ansible_job_id": "169775103514.1681709",
    "attempts": 2,
    "changed": false,
    "finished": 0,
    "invocation": {
        "module_args": {
            "_async_dir": "/home/semaphore/.ansible_async",
            "jid": "169775103514.1681709",
            "mode": "status"
        }
    },
    "results_file": "/home/semaphore/.ansible_async/169775103514.1681709",
    "retries": 8,
    "started": 1,
    "stderr": "",
    "stderr_lines": [],
    "stdout": "",
    "stdout_lines": []
}
<localhost> EXEC /bin/sh -c 'echo ~semaphore && sleep 0'
<localhost> EXEC /bin/sh -c 'echo ~semaphore && sleep 0'
<localhost> EXEC /bin/sh -c '( umask 77 && mkdir -p "` echo /home/semaphore/.ansible/tmp `"&& mkdir "` echo /home/semaphore/.ansible/tmp/ansible-tmp-1653305103.1612804-1681718-49635212208866 `" && echo ansible-tmp-1653305103.1612804-1681718-49635212208866="` echo /home/semaphore/.ansible/tmp/ansible-tmp-1653305103.1612804-1681718-49635212208866 `" ) && sleep 0'
Using module file /usr/lib/python3/dist-packages/ansible/modules/async_status.py
<localhost> PUT /tmp/ansible-test/ansible-local-1681685kzrx868d/tmpgo9panxp TO /home/semaphore/.ansible/tmp/ansible-tmp-1653305103.1612804-1681718-49635212208866/AnsiballZ_async_status.py
<localhost> EXEC /bin/sh -c 'chmod u+x /home/semaphore/.ansible/tmp/ansible-tmp-1653305103.1612804-1681718-49635212208866/ /home/semaphore/.ansible/tmp/ansible-tmp-1653305103.1612804-1681718-49635212208866/AnsiballZ_async_status.py && sleep 0'
<localhost> EXEC /bin/sh -c '/usr/bin/python3 /home/semaphore/.ansible/tmp/ansible-tmp-1653305103.1612804-1681718-49635212208866/AnsiballZ_async_status.py && sleep 0'
<localhost> EXEC /bin/sh -c 'rm -f -r /home/semaphore/.ansible/tmp/ansible-tmp-1653305103.1612804-1681718-49635212208866/ > /dev/null 2>&1 && sleep 0'
FAILED - RETRYING: [bar -> localhost]: async_status (5 retries left).Result was: {
    "ansible_job_id": "169775103514.1681709",
    "attempts": 3,
    "changed": false,
    "finished": 0,
    "invocation": {
        "module_args": {
            "_async_dir": "/home/semaphore/.ansible_async",
            "jid": "169775103514.1681709",
            "mode": "status"
        }
    },
    "results_file": "/home/semaphore/.ansible_async/169775103514.1681709",
    "retries": 8,
    "started": 1,
    "stderr": "",
    "stderr_lines": [],
    "stdout": "",
    "stdout_lines": []
}
<localhost> EXEC /bin/sh -c 'echo ~semaphore && sleep 0'
<localhost> EXEC /bin/sh -c 'echo ~semaphore && sleep 0'
<localhost> EXEC /bin/sh -c '( umask 77 && mkdir -p "` echo /home/semaphore/.ansible/tmp `"&& mkdir "` echo /home/semaphore/.ansible/tmp/ansible-tmp-1653305104.337606-1681718-248269275028915 `" && echo ansible-tmp-1653305104.337606-1681718-248269275028915="` echo /home/semaphore/.ansible/tmp/ansible-tmp-1653305104.337606-1681718-248269275028915 `" ) && sleep 0'
Using module file /usr/lib/python3/dist-packages/ansible/modules/async_status.py
<localhost> PUT /tmp/ansible-test/ansible-local-1681685kzrx868d/tmpw59agc30 TO /home/semaphore/.ansible/tmp/ansible-tmp-1653305104.337606-1681718-248269275028915/AnsiballZ_async_status.py
<localhost> EXEC /bin/sh -c 'chmod u+x /home/semaphore/.ansible/tmp/ansible-tmp-1653305104.337606-1681718-248269275028915/ /home/semaphore/.ansible/tmp/ansible-tmp-1653305104.337606-1681718-248269275028915/AnsiballZ_async_status.py && sleep 0'
<localhost> EXEC /bin/sh -c '/usr/bin/python3 /home/semaphore/.ansible/tmp/ansible-tmp-1653305104.337606-1681718-248269275028915/AnsiballZ_async_status.py && sleep 0'
<localhost> EXEC /bin/sh -c 'rm -f -r /home/semaphore/.ansible/tmp/ansible-tmp-1653305104.337606-1681718-248269275028915/ > /dev/null 2>&1 && sleep 0'
FAILED - RETRYING: [bar -> localhost]: async_status (4 retries left).Result was: {
    "ansible_job_id": "169775103514.1681709",
    "attempts": 4,
    "changed": false,
    "finished": 0,
    "invocation": {
        "module_args": {
            "_async_dir": "/home/semaphore/.ansible_async",
            "jid": "169775103514.1681709",
            "mode": "status"
        }
    },
    "results_file": "/home/semaphore/.ansible_async/169775103514.1681709",
    "retries": 8,
    "started": 1,
    "stderr": "",
    "stderr_lines": [],
    "stdout": "",
    "stdout_lines": []
}
<localhost> EXEC /bin/sh -c 'echo ~semaphore && sleep 0'
<localhost> EXEC /bin/sh -c 'echo ~semaphore && sleep 0'
<localhost> EXEC /bin/sh -c '( umask 77 && mkdir -p "` echo /home/semaphore/.ansible/tmp `"&& mkdir "` echo /home/semaphore/.ansible/tmp/ansible-tmp-1653305105.52543-1681718-191361270279238 `" && echo ansible-tmp-1653305105.52543-1681718-191361270279238="` echo /home/semaphore/.ansible/tmp/ansible-tmp-1653305105.52543-1681718-191361270279238 `" ) && sleep 0'
Using module file /usr/lib/python3/dist-packages/ansible/modules/async_status.py
<localhost> PUT /tmp/ansible-test/ansible-local-1681685kzrx868d/tmp_bw8w17r TO /home/semaphore/.ansible/tmp/ansible-tmp-1653305105.52543-1681718-191361270279238/AnsiballZ_async_status.py
<localhost> EXEC /bin/sh -c 'chmod u+x /home/semaphore/.ansible/tmp/ansible-tmp-1653305105.52543-1681718-191361270279238/ /home/semaphore/.ansible/tmp/ansible-tmp-1653305105.52543-1681718-191361270279238/AnsiballZ_async_status.py && sleep 0'
<localhost> EXEC /bin/sh -c '/usr/bin/python3 /home/semaphore/.ansible/tmp/ansible-tmp-1653305105.52543-1681718-191361270279238/AnsiballZ_async_status.py && sleep 0'
<localhost> EXEC /bin/sh -c 'rm -f -r /home/semaphore/.ansible/tmp/ansible-tmp-1653305105.52543-1681718-191361270279238/ > /dev/null 2>&1 && sleep 0'
FAILED - RETRYING: [bar -> localhost]: async_status (3 retries left).Result was: {
    "ansible_job_id": "169775103514.1681709",
    "attempts": 5,
    "changed": false,
    "finished": 0,
    "invocation": {
        "module_args": {
            "_async_dir": "/home/semaphore/.ansible_async",
            "jid": "169775103514.1681709",
            "mode": "status"
        }
    },
    "results_file": "/home/semaphore/.ansible_async/169775103514.1681709",
    "retries": 8,
    "started": 1,
    "stderr": "",
    "stderr_lines": [],
    "stdout": "",
    "stdout_lines": []
}
<localhost> EXEC /bin/sh -c 'echo ~semaphore && sleep 0'
<localhost> EXEC /bin/sh -c 'echo ~semaphore && sleep 0'
<localhost> EXEC /bin/sh -c '( umask 77 && mkdir -p "` echo /home/semaphore/.ansible/tmp `"&& mkdir "` echo /home/semaphore/.ansible/tmp/ansible-tmp-1653305106.6934974-1681718-28012316136652 `" && echo ansible-tmp-1653305106.6934974-1681718-28012316136652="` echo /home/semaphore/.ansible/tmp/ansible-tmp-1653305106.6934974-1681718-28012316136652 `" ) && sleep 0'
Using module file /usr/lib/python3/dist-packages/ansible/modules/async_status.py
<localhost> PUT /tmp/ansible-test/ansible-local-1681685kzrx868d/tmpzzqtbgub TO /home/semaphore/.ansible/tmp/ansible-tmp-1653305106.6934974-1681718-28012316136652/AnsiballZ_async_status.py
<localhost> EXEC /bin/sh -c 'chmod u+x /home/semaphore/.ansible/tmp/ansible-tmp-1653305106.6934974-1681718-28012316136652/ /home/semaphore/.ansible/tmp/ansible-tmp-1653305106.6934974-1681718-28012316136652/AnsiballZ_async_status.py && sleep 0'
<localhost> EXEC /bin/sh -c '/usr/bin/python3 /home/semaphore/.ansible/tmp/ansible-tmp-1653305106.6934974-1681718-28012316136652/AnsiballZ_async_status.py && sleep 0'
<localhost> EXEC /bin/sh -c 'rm -f -r /home/semaphore/.ansible/tmp/ansible-tmp-1653305106.6934974-1681718-28012316136652/ > /dev/null 2>&1 && sleep 0'
changed: [bar -> localhost] => {
    "ansible_job_id": "169775103514.1681709",
    "attempts": 6,
    "changed": true,
    "cmd": [
        "sleep",
        "5"
    ],
    "delta": "0:00:05.003979",
    "end": "2022-05-23 11:25:05.731899",
    "finished": 1,
    "invocation": {
        "module_args": {
            "_raw_params": "sleep 5",
            "_uses_shell": false,
            "argv": null,
            "chdir": null,
            "creates": null,
            "executable": null,
            "removes": null,
            "stdin": null,
            "stdin_add_newline": true,
            "strip_empty_ends": true,
            "warn": false
        }
    },
    "msg": "",
    "rc": 0,
    "results_file": "/home/semaphore/.ansible_async/169775103514.1681709",
    "start": "2022-05-23 11:25:00.727920",
    "started": 1,
    "stderr": "",
    "stderr_lines": [],
    "stdout": "",
    "stdout_lines": []
}
META: ran handlers
META: ran handlers

PLAY RECAP *********************************************************************
bar                        : ok=2    changed=2    unreachable=0    failed=0    skipped=0    rescued=0    ignored=0

But when executing the same playbook via ansible-semaphore UI (no extra variables specified or anything) I get the described error.

ansible-semaphore debug output

1:58:08 PM Task 2147483558 added to queue
1:58:11 PM Preparing: 2147483558
1:58:11 PM Prepare TaskRunner with template: async-delegate-test.yml
1:58:11 PM Updating Repository /home/semaphore/test-repo/.git
1:58:11 PM From /home/semaphore/test-repo/
1:58:11 PM * branch master -> FETCH_HEAD
1:58:11 PM Already up to date.
1:58:11 PM Get current commit hash
1:58:11 PM Get current commit message
1:58:11 PM installing static inventory
1:58:11 PM No roles/requirements.yml file found. Skip galaxy install process.
1:58:16 PM Started: 2147483558
1:58:16 PM Run TaskRunner with template: async-delegate-test.yml
1:58:16 PM ansible-playbook [core 2.12.4]
1:58:16 PM config file = /etc/ansible/ansible.cfg
1:58:16 PM configured module search path = ['/tmp/semaphore/.ansible/plugins/modules', '/usr/share/ansible/plugins/modules']
1:58:16 PM ansible python module location = /usr/lib/python3/dist-packages/ansible
1:58:16 PM ansible collection location = /tmp/semaphore/.ansible/collections:/usr/share/ansible/collections
1:58:16 PM executable location = /usr/bin/ansible-playbook
1:58:16 PM python version = 3.8.10 (default, Mar 15 2022, 12:22:08) [GCC 9.4.0]
1:58:16 PM jinja version = 2.10.1
1:58:16 PM libyaml = True
1:58:16 PM Using /etc/ansible/ansible.cfg as config file
1:58:16 PM setting up inventory plugins
1:58:16 PM host_list declined parsing /tmp/semaphore/inventory_2147483558 as it did not pass its verify_file() method
1:58:16 PM script declined parsing /tmp/semaphore/inventory_2147483558 as it did not pass its verify_file() method
1:58:16 PM auto declined parsing /tmp/semaphore/inventory_2147483558 as it did not pass its verify_file() method
1:58:16 PM Parsed /tmp/semaphore/inventory_2147483558 inventory source with ini plugin
1:58:16 PM Loading callback plugin default of type stdout, v2.0 from /usr/lib/python3/dist-packages/ansible/plugins/callback/default.py
1:58:16 PM Skipping callback 'default', as we already have a stdout callback.
1:58:16 PM Skipping callback 'minimal', as we already have a stdout callback.
1:58:16 PM Skipping callback 'oneline', as we already have a stdout callback.
1:58:16 PM 1:58:16 PM PLAYBOOK: async-delegate-test.yml **********************************************
1:58:16 PM Positional arguments: async-delegate-test.yml
1:58:16 PM verbosity: 4
1:58:16 PM connection: smart
1:58:16 PM timeout: 10
1:58:16 PM become_method: sudo
1:58:16 PM tags: ('all',)
1:58:16 PM inventory: ('/tmp/semaphore/inventory_2147483558',)
1:58:16 PM extra_vars: ('{"semaphore_vars":{"task_details":{"username":"jojoob"}}}',)
1:58:16 PM forks: 5
1:58:16 PM 1 plays in async-delegate-test.yml
1:58:16 PM 1:58:16 PM PLAY [all] *********************************************************************
1:58:16 PM META: ran handlers
1:58:16 PM 1:58:16 PM TASK [command] *****************************************************************
1:58:16 PM task path: /tmp/semaphore/repository_1_1/async-delegate-test.yml:6
1:58:16 PM <localhost> ESTABLISH LOCAL CONNECTION FOR USER: semaphore
1:58:16 PM <localhost> EXEC /bin/sh -c 'echo ~semaphore && sleep 0'
1:58:16 PM <localhost> EXEC /bin/sh -c '( umask 77 && mkdir -p "` echo /home/semaphore/.ansible/tmp `"&& mkdir "` echo /home/semaphore/.ansible/tmp/ansible-tmp-1652961496.8348942-1077874-152839807798810 `" && echo ansible-tmp-1652961496.8348942-1077874-152839807798810="` echo /home/semaphore/.ansible/tmp/ansible-tmp-1652961496.8348942-1077874-152839807798810 `" ) && sleep 0'
1:58:16 PM Using module file /usr/lib/python3/dist-packages/ansible/modules/command.py
1:58:16 PM <localhost> PUT /tmp/semaphore/.ansible/tmp/ansible-local-1077867v5rf2nhz/tmporoxl51c TO /home/semaphore/.ansible/tmp/ansible-tmp-1652961496.8348942-1077874-152839807798810/AnsiballZ_command.py
1:58:17 PM <localhost> PUT /tmp/semaphore/.ansible/tmp/ansible-local-1077867v5rf2nhz/tmp2weahl6o TO /home/semaphore/.ansible/tmp/ansible-tmp-1652961496.8348942-1077874-152839807798810/async_wrapper.py
1:58:17 PM <localhost> EXEC /bin/sh -c 'chmod u+x /home/semaphore/.ansible/tmp/ansible-tmp-1652961496.8348942-1077874-152839807798810/ /home/semaphore/.ansible/tmp/ansible-tmp-1652961496.8348942-1077874-152839807798810/AnsiballZ_command.py /home/semaphore/.ansible/tmp/ansible-tmp-1652961496.8348942-1077874-152839807798810/async_wrapper.py && sleep 0'
1:58:17 PM <localhost> EXEC /bin/sh -c 'ANSIBLE_ASYNC_DIR='"'"'~/.ansible_async'"'"' /usr/bin/python3 /home/semaphore/.ansible/tmp/ansible-tmp-1652961496.8348942-1077874-152839807798810/async_wrapper.py 119402064606 7 /home/semaphore/.ansible/tmp/ansible-tmp-1652961496.8348942-1077874-152839807798810/AnsiballZ_command.py _ && sleep 0'
1:58:17 PM changed: [bar -> localhost] => {
1:58:17 PM "ansible_job_id": "119402064606.1077895",
1:58:17 PM "changed": true,
1:58:17 PM "finished": 0,
1:58:17 PM "results_file": "/tmp/semaphore/.ansible_async/119402064606.1077895",
1:58:17 PM "started": 1
1:58:17 PM }

1:58:17 PM 1:58:17 PM TASK [async_status] ************************************************************
1:58:17 PM task path: /tmp/semaphore/repository_1_1/async-delegate-test.yml:11
1:58:17 PM <localhost> ESTABLISH LOCAL CONNECTION FOR USER: semaphore
1:58:17 PM <localhost> EXEC /bin/sh -c 'echo ~semaphore && sleep 0'
1:58:17 PM <localhost> EXEC /bin/sh -c 'echo ~semaphore && sleep 0'
1:58:17 PM <localhost> EXEC /bin/sh -c '( umask 77 && mkdir -p "` echo /home/semaphore/.ansible/tmp `"&& mkdir "` echo /home/semaphore/.ansible/tmp/ansible-tmp-1652961497.3271375-1077904-221201957350051 `" && echo ansible-tmp-1652961497.3271375-1077904-221201957350051="` echo /home/semaphore/.ansible/tmp/ansible-tmp-1652961497.3271375-1077904-221201957350051 `" ) && sleep 0'
1:58:17 PM Using module file /usr/lib/python3/dist-packages/ansible/modules/async_status.py
1:58:17 PM <localhost> PUT /tmp/semaphore/.ansible/tmp/ansible-local-1077867v5rf2nhz/tmpcc9m8qs3 TO /home/semaphore/.ansible/tmp/ansible-tmp-1652961497.3271375-1077904-221201957350051/AnsiballZ_async_status.py
1:58:17 PM <localhost> EXEC /bin/sh -c 'chmod u+x /home/semaphore/.ansible/tmp/ansible-tmp-1652961497.3271375-1077904-221201957350051/ /home/semaphore/.ansible/tmp/ansible-tmp-1652961497.3271375-1077904-221201957350051/AnsiballZ_async_status.py && sleep 0'
1:58:17 PM <localhost> EXEC /bin/sh -c '/usr/bin/python3 /home/semaphore/.ansible/tmp/ansible-tmp-1652961497.3271375-1077904-221201957350051/AnsiballZ_async_status.py && sleep 0'
1:58:17 PM <localhost> EXEC /bin/sh -c 'rm -f -r /home/semaphore/.ansible/tmp/ansible-tmp-1652961497.3271375-1077904-221201957350051/ > /dev/null 2>&1 && sleep 0'
1:58:17 PM fatal: [bar -> localhost]: FAILED! => {
1:58:17 PM "ansible_job_id": "119402064606.1077895",
1:58:17 PM "attempts": 1,
1:58:17 PM "changed": false,
1:58:17 PM "finished": 1,
1:58:17 PM "invocation": {
1:58:17 PM "module_args": {
1:58:17 PM "_async_dir": "/home/semaphore/.ansible_async",
1:58:17 PM "jid": "119402064606.1077895",
1:58:17 PM "mode": "status"
1:58:17 PM }
1:58:17 PM },
1:58:17 PM "msg": "could not find job",
1:58:17 PM "results_file": "/home/semaphore/.ansible_async/119402064606.1077895",
1:58:17 PM "started": 1,
1:58:17 PM "stderr": "",
1:58:17 PM "stderr_lines": [],
1:58:17 PM "stdout": "",
1:58:17 PM "stdout_lines": []
1:58:17 PM }
1:58:17 PM 1:58:17 PM PLAY RECAP *********************************************************************
1:58:17 PM bar : ok=1 changed=1 unreachable=0 failed=1 skipped=0 rescued=0 ignored=0
1:58:17 PM Running playbook failed: exit status 2

Versions

ansible-playbook [core 2.12.4]
Ubuntu 20.04.4
semaphore version v2.8.53 installed via PPA

May 23 '22 12:05 jojoob

Perhaps there is a difference between your semaphore config (ie config.json) and your ansible config (ie. ansible.cfg). For example the tmp_path semaphore option and the local_tmp ansible option?

May 24 '22 00:05 jdhayes

I was able to reproduce the error just now on command line by setting the HOME environment variable HOME=/tmp/semaphore ansible-playbook -i foo, async-delegate-test.yml. Maybe this is a bug within the ansible async_status module.

May 24 '22 00:05 jojoob

~~I didn't saw the issue happening for a long time now.~~ (Edit: I was confused with another problem. I actually worked around the problem by not using localhost for delegation but connecting to localhost via ssh) And since the problem seems not directly related to semaphore I'm closing this issue.

Jun 27 '23 08:06 jojoob

'could not find job' on async_status with delegate_to localhost