iso_extract module intermittently fails with Read-only file system: '.discinfo'

[ocstjf]

Summary

Intermittently, the iso_extract operation will fail after a successful extraction with this error: OSError: [Errno 30] Read-only file system: '.discinfo' We're almost certain it's because the umount operation isn't finished before attempting to delete tmp_dir at the end of the iso_extract.py module.

Issue Type

Bug Report

Component Name

community.general.iso_extract

Ansible Version

$ ansible --version
ansible [core 2.11.5]
  config file = None
  configured module search path = ['/root/.ansible/plugins/modules', '/usr/share/ansible/plugins/modules']
  ansible python module location = /usr/local/lib/python3.8/site-packages/ansible
  ansible collection location = /root/.ansible/collections:/usr/share/ansible/collections
  executable location = /usr/local/bin/ansible
  python version = 3.8.6 (default, Jan 22 2021, 11:41:28) [GCC 8.4.1 20200928 (Red Hat 8.4.1-1)]
  jinja version = 3.0.2
  libyaml = True

Community.general Version

$ ansible-galaxy collection list community.general

# /usr/share/ansible/collections/ansible_collections
Collection        Version
----------------- -------
community.general 3.7.0

Configuration

$ ansible-config dump --only-changed

(output is empty, no changes)

OS / Environment

$ uname -a
Linux xxx09 4.18.0-305.19.1.el8_4.x86_64 #1 SMP Tue Sep 7 07:07:31 EDT 2021 x86_64 x86_64 x86_64 GNU/Linux

Steps to Reproduce

Run the following playbook repeatedly on busy system until problem occurs

---

- name: test_iso_extract
  hosts: localhost
  gather_facts: no
  vars:
    # iso is large image on nfs-mounted filesystem, any should do
    iso:  /opt/repo/team/test/iso/VMware-ESXi-6.7.0-Update3-18828794-HPE-Gen9plus-670.U3.10.8.1.2-Nov2021.iso
    # dest is on local filesystem different from one used by ansible
    dest: /var/tmp/dest/
    # original source code loops through several files:
    files:
      - efi/boot/bootx64.efi
  tasks:
    - name: Create dest dir if needed
      file:
        path: "{{ dest }}"
        state: directory

    - name: Extract files from iso
      community.general.iso_extract:
        image: "{{ iso }}"
        dest: "{{ dest }}"
        files: "{{ item }}"
      loop: "{{ files }}"
      register: iso_extract_result

    - debug:
        var: iso_extract_result

Expected Results

Expect iso_extract module to report success after successful extraction.

Actual Results

(... applicable portion only is shown ...)

...
TASK [Extract files from iso] **************************************************
task path: /backups/test/test_iso_extract.yml:22
<127.0.0.1> ESTABLISH LOCAL CONNECTION FOR USER: root
<127.0.0.1> EXEC /bin/sh -c 'echo ~root && sleep 0'
<127.0.0.1> EXEC /bin/sh -c '( umask 77 && mkdir -p "` echo /root/.ansible/tmp `"&& mkdir "` echo /root/.ansible/tmp/ansible-tmp-1664898903.0001626-3391264-206907975048156 `" && echo ansible-tmp-1664898903.0001626-3391264-206907975048156="` echo /root/.ansible/tmp/ansible-tmp-1664898903.0001626-3391264-206907975048156 `" ) && sleep 0'
Using module file /usr/share/ansible/collections/ansible_collections/community/general/plugins/modules/iso_extract.py
<127.0.0.1> PUT /root/.ansible/tmp/ansible-local-3391086fo08r6pc/tmpe1sx398o TO /root/.ansible/tmp/ansible-tmp-1664898903.0001626-3391264-206907975048156/AnsiballZ_iso_extract.py
<127.0.0.1> EXEC /bin/sh -c 'chmod u+x /root/.ansible/tmp/ansible-tmp-1664898903.0001626-3391264-206907975048156/ /root/.ansible/tmp/ansible-tmp-1664898903.0001626-3391264-206907975048156/AnsiballZ_iso_extract.py && sleep 0'
<127.0.0.1> EXEC /bin/sh -c '/usr/bin/python3.8 /root/.ansible/tmp/ansible-tmp-1664898903.0001626-3391264-206907975048156/AnsiballZ_iso_extract.py && sleep 0'
<127.0.0.1> EXEC /bin/sh -c 'rm -f -r /root/.ansible/tmp/ansible-tmp-1664898903.0001626-3391264-206907975048156/ > /dev/null 2>&1 && sleep 0'
The full traceback is:
Traceback (most recent call last):
  File "/root/.ansible/tmp/ansible-tmp-1664898903.0001626-3391264-206907975048156/AnsiballZ_iso_extract.py", line 100, in <module>
    _ansiballz_main()
  File "/root/.ansible/tmp/ansible-tmp-1664898903.0001626-3391264-206907975048156/AnsiballZ_iso_extract.py", line 92, in _ansiballz_main
    invoke_module(zipped_mod, temp_path, ANSIBALLZ_PARAMS)
  File "/root/.ansible/tmp/ansible-tmp-1664898903.0001626-3391264-206907975048156/AnsiballZ_iso_extract.py", line 40, in invoke_module
    runpy.run_module(mod_name='ansible_collections.community.general.plugins.modules.iso_extract', init_globals=dict(_module_fqn='ansible_collections.community.general.plugins.modules.iso_extract', _modlib_path=modlib_path),
  File "/usr/lib64/python3.8/runpy.py", line 207, in run_module
    return _run_module_code(code, init_globals, run_name, mod_spec)
  File "/usr/lib64/python3.8/runpy.py", line 97, in _run_module_code
    _run_code(code, mod_globals, init_globals,
  File "/usr/lib64/python3.8/runpy.py", line 87, in _run_code
    exec(code, run_globals)
  File "/tmp/ansible_community.general.iso_extract_payload_ggdqisd2/ansible_community.general.iso_extract_payload.zip/ansible_collections/community/general/plugins/modules/iso_extract.py", line 212, in <module>
  File "/tmp/ansible_community.general.iso_extract_payload_ggdqisd2/ansible_community.general.iso_extract_payload.zip/ansible_collections/community/general/plugins/modules/iso_extract.py", line 206, in main
  File "/usr/lib64/python3.8/shutil.py", line 715, in rmtree
    _rmtree_safe_fd(fd, path, onerror)
  File "/usr/lib64/python3.8/shutil.py", line 672, in _rmtree_safe_fd
    onerror(os.unlink, fullname, sys.exc_info())
  File "/usr/lib64/python3.8/shutil.py", line 670, in _rmtree_safe_fd
    os.unlink(entry.name, dir_fd=topfd)
OSError: [Errno 30] Read-only file system: '.discinfo'
failed: [localhost] (item=efi/boot/bootx64.efi) => {
    "ansible_loop_var": "item",
    "changed": false,
    "item": "efi/boot/bootx64.efi",
    "module_stderr": (not shown, identical to Traceback above)
    "module_stdout": "",
    "msg": "MODULE FAILURE\nSee stdout/stderr for the exact error",
    "rc": 1 
}
...

Code of Conduct

• [X] I agree to follow the Ansible Code of Conduct

[ansibullbot]

Files identified in the description:

• plugins/modules/files/iso_extract.py

If these files are incorrect, please update the component name section of the description or use the !component bot command.

click here for bot help

[ansibullbot]

cc @dagwieers @jhoekx @ribbons click here for bot help

[ansibullbot]

Files identified in the description:

• plugins/modules/iso_extract.py

If these files are incorrect, please update the component name section of the description or use the !component bot command.

click here for bot help

[jnm27]

Some more data:

We added some retries and debug data and found that it is failing to umount the dir because it is busy:

umount_rc: 32 umount_stderr: |- umount: /tmp/tmp1fmemzim: target is busy.

No idea why it is busy, perhaps the copy just before still has a file handle open.

Not sure how you want to handle it, but we added some retries to the umount command after 5 second sleeps. Time will tell if that is enough to workaround it.