raid-1 flake on rhel8, rhel9: "org.fedoraproject.Anaconda.Modules.Storage:gi.overrides.BlockDev.MDRaidError: Process reported exit code 2: mdadm: /dev/vda3 is busy - skipping"

[rvykydal]

First seen on 08-01-2023

23:52:05,763 WARNING org.fedoraproject.Anaconda.Modules.Storage:INFO:program:Running [44] mdadm --assemble /dev/md/1 --run --uuid=c1be631f:9ba4f77c:e0649292:7b4ee300 /dev/vda3 /dev/vdb3 ...
23:52:05,782 WARNING org.fedoraproject.Anaconda.Modules.Storage:INFO:program:stdout[44]:
23:52:05,782 WARNING org.fedoraproject.Anaconda.Modules.Storage:INFO:program:stderr[44]: mdadm: /dev/vda3 is busy - skipping
23:52:05,782 WARNING org.fedoraproject.Anaconda.Modules.Storage:mdadm: /dev/vdb3 is busy - skipping
23:52:05,782 WARNING org.fedoraproject.Anaconda.Modules.Storage:INFO:program:...done [44] (exit code: 2)
23:52:05,783 WARNING org.fedoraproject.Anaconda.Modules.Storage:INFO:anaconda.threading:Thread Failed: AnaTaskThread-ActivateFilesystemsTask-1 (140609254823680)
23:52:05,784 WARNING org.fedoraproject.Anaconda.Modules.Storage:ERROR:anaconda.modules.common.task.task:Thread AnaTaskThread-ActivateFilesystemsTask-1 has failed: Traceback (most recent call last):
23:52:05,784 WARNING org.fedoraproject.Anaconda.Modules.Storage:  File "/usr/lib64/python3.6/site-packages/gi/overrides/BlockDev.py", line 1038, in wrapped
23:52:05,784 WARNING org.fedoraproject.Anaconda.Modules.Storage:    ret = orig_obj(*args, **kwargs)
23:52:05,784 WARNING org.fedoraproject.Anaconda.Modules.Storage:  File "/usr/lib64/python3.6/site-packages/gi/overrides/BlockDev.py", line 758, in md_activate
23:52:05,784 WARNING org.fedoraproject.Anaconda.Modules.Storage:    return _md_activate(raid_spec, members, uuid, start_degraded, extra)
23:52:05,784 WARNING org.fedoraproject.Anaconda.Modules.Storage:GLib.GError: g-bd-utils-exec-error-quark: Process reported exit code 2: mdadm: /dev/vda3 is busy - skipping
23:52:05,784 WARNING org.fedoraproject.Anaconda.Modules.Storage:mdadm: /dev/vdb3 is busy - skipping
23:52:05,784 WARNING org.fedoraproject.Anaconda.Modules.Storage: (0)
23:52:05,784 WARNING org.fedoraproject.Anaconda.Modules.Storage:During handling of the above exception, another exception occurred:
23:52:05,784 WARNING org.fedoraproject.Anaconda.Modules.Storage:Traceback (most recent call last):
23:52:05,784 WARNING org.fedoraproject.Anaconda.Modules.Storage:  File "/usr/lib64/python3.6/site-packages/pyanaconda/threading.py", line 280, in run
23:52:05,784 WARNING org.fedoraproject.Anaconda.Modules.Storage:    threading.Thread.run(self)
23:52:05,784 WARNING org.fedoraproject.Anaconda.Modules.Storage:  File "/usr/lib64/python3.6/threading.py", line 885, in run
23:52:05,784 WARNING org.fedoraproject.Anaconda.Modules.Storage:    self._target(*self._args, **self._kwargs)
23:52:05,784 WARNING org.fedoraproject.Anaconda.Modules.Storage:  File "/usr/lib64/python3.6/site-packages/pyanaconda/modules/common/task/task.py", line 97, in _task_run_callback
23:52:05,784 WARNING org.fedoraproject.Anaconda.Modules.Storage:    self._set_result(self.run())
23:52:05,784 WARNING org.fedoraproject.Anaconda.Modules.Storage:  File "/usr/lib64/python3.6/site-packages/pyanaconda/modules/storage/installation.py", line 84, in run
23:52:05,784 WARNING org.fedoraproject.Anaconda.Modules.Storage:    callbacks=register
23:52:05,784 WARNING org.fedoraproject.Anaconda.Modules.Storage:  File "/usr/lib64/python3.6/site-packages/pyanaconda/modules/storage/installation.py", line 162, in _turn_on_filesystems
23:52:05,784 WARNING org.fedoraproject.Anaconda.Modules.Storage:    storage.do_it(callbacks)
23:52:05,784 WARNING org.fedoraproject.Anaconda.Modules.Storage:  File "/usr/lib/python3.6/site-packages/blivet/threads.py", line 53, in run_with_lock
23:52:05,784 WARNING org.fedoraproject.Anaconda.Modules.Storage:    return m(*args, **kwargs)
23:52:05,784 WARNING org.fedoraproject.Anaconda.Modules.Storage:  File "/usr/lib/python3.6/site-packages/blivet/blivet.py", line 115, in do_it
23:52:05,784 WARNING org.fedoraproject.Anaconda.Modules.Storage:    self.devicetree.actions.process(callbacks=callbacks, devices=self.devices)
23:52:05,784 WARNING org.fedoraproject.Anaconda.Modules.Storage:  File "/usr/lib/python3.6/site-packages/blivet/actionlist.py", line 47, in wrapped_func
23:52:05,784 WARNING org.fedoraproject.Anaconda.Modules.Storage:    return func(obj, *args, **kwargs)
23:52:05,784 WARNING org.fedoraproject.Anaconda.Modules.Storage:  File "/usr/lib/python3.6/site-packages/blivet/actionlist.py", line 284, in process
23:52:05,784 WARNING org.fedoraproject.Anaconda.Modules.Storage:    action.execute(callbacks)
23:52:05,784 WARNING org.fedoraproject.Anaconda.Modules.Storage:  File "/usr/lib/python3.6/site-packages/blivet/threads.py", line 53, in run_with_lock
23:52:05,784 WARNING org.fedoraproject.Anaconda.Modules.Storage:    return m(*args, **kwargs)
23:52:05,785 WARNING org.fedoraproject.Anaconda.Modules.Storage:  File "/usr/lib/python3.6/site-packages/blivet/deviceaction.py", line 661, in execute
23:52:05,785 WARNING org.fedoraproject.Anaconda.Modules.Storage:    self.device.setup()
23:52:05,785 WARNING org.fedoraproject.Anaconda.Modules.Storage:  File "/usr/lib/python3.6/site-packages/blivet/threads.py", line 53, in run_with_lock
23:52:05,785 WARNING org.fedoraproject.Anaconda.Modules.Storage:    return m(*args, **kwargs)
23:52:05,785 WARNING org.fedoraproject.Anaconda.Modules.Storage:  File "/usr/lib/python3.6/site-packages/blivet/devices/storage.py", line 425, in setup
23:52:05,785 WARNING org.fedoraproject.Anaconda.Modules.Storage:    self._setup(orig=orig)
23:52:05,785 WARNING org.fedoraproject.Anaconda.Modules.Storage:  File "/usr/lib/python3.6/site-packages/blivet/threads.py", line 53, in run_with_lock
23:52:05,785 WARNING org.fedoraproject.Anaconda.Modules.Storage:    return m(*args, **kwargs)
23:52:05,785 WARNING org.fedoraproject.Anaconda.Modules.Storage:  File "/usr/lib/python3.6/site-packages/blivet/devices/md.py", line 517, in _setup
23:52:05,785 WARNING org.fedoraproject.Anaconda.Modules.Storage:    blockdev.md.activate(self.path, members=disks, uuid=self.mdadm_format_uuid)
23:52:05,785 WARNING org.fedoraproject.Anaconda.Modules.Storage:  File "/usr/lib64/python3.6/site-packages/gi/overrides/BlockDev.py", line 1060, in wrapped
23:52:05,785 WARNING org.fedoraproject.Anaconda.Modules.Storage:    raise transform[1](msg)
23:52:05,785 WARNING org.fedoraproject.Anaconda.Modules.Storage:gi.overrides.BlockDev.MDRaidError: Process reported exit code 2: mdadm: /dev/vda3 is busy - skipping
23:52:05,785 WARNING org.fedoraproject.Anaconda.Modules.Storage:mdadm: /dev/vdb3 is busy - skipping
23:52:05,786 WARNING org.fedoraproject.Anaconda.Modules.Storage:INFO:anaconda.threading:Thread Done: AnaTaskThread-ActivateFilesystemsTask-1 (140609254823680)

kstest.log virt-install.log

[vojtechtrefny]

Probably same issue as described here: https://bugzilla.redhat.com/show_bug.cgi?id=2166669

[rvykydal]

Seen on rhel9 01-01-2024 kstest.log virt-install.log

Seems to be very rare.