bots icon indicating copy to clipboard operation
bots copied to clipboard

Published 20 hours ago • verified publisher icon cockpit-project

mdadm fails with "Unable to initialize sysfs" since 6.17.0 rc0

Open jelly opened this issue 4 months ago • 7 comments

In rawhide the testNotRemovingDisks test fails

> warn: Error starting RAID array: Process reported exit code 1: mdadm: Unable to initialize sysfs
mdadm: No arrays found in config file or automatically

> error: Error starting RAID array: Process reported exit code 1: mdadm: Unable to initialize sysfs
mdadm: No arrays found in config file or automatically

This clearly comes from mdadm, via a udisks dbus call (which is called in libblockdev).

I can't figure out the exact mdadm reproducer but I gathered strace output: mdadm-strace.log

Downstream report: https://bugzilla.redhat.com/show_bug.cgi?id=2385871

jelly avatar Jul 31 '25 16:07 jelly

So note that we run rc0, it might be a kernel issue... 6.17.0-0.rc0.250730g4b290aae788e.3.fc43.x86_64

jelly avatar Jul 31 '25 16:07 jelly

And it should be mdadm --assemble --scan --run --uuid=f63dc7c1:37a27c6f:bdda1af which fails

[root@fedora-rawhide-127-0-0-2-2201 ~]# mdadm --assemble --scan --run --uuid=982a9d5f:688f94f5:1e81ab1d:339c0cfb
mdadm: Unable to initialize sysfs
mdadm: No arrays found in config file or automatically

strace'd

openat(AT_FDCWD, "/dev/loop0", O_RDONLY|O_EXCL|O_DIRECT) = 5
ioctl(5, BLKSSZGET, [512])              = 0
fstat(5, {st_mode=S_IFBLK|0660, st_rdev=makedev(0x7, 0), ...}) = 0
ioctl(5, BLKGETSIZE64, [49999872])      = 0
lseek(5, 4096, SEEK_SET)                = 4096
read(5, "\374N+\251\1\0\0\0\0\0\0\0\0\0\0\0\230*\235_h\217\224\365\36\201\253\0353\234\f\373"..., 4096) = 4096
lseek(5, 0, SEEK_CUR)                   = 8192
close(5)                                = 0
fstat(4, {st_mode=S_IFBLK|0600, st_rdev=makedev(0x9, 0x7f), ...}) = 0
readlink("/sys/dev/block/9:127", 0x7ffe4f46ce90, 199) = -1 ENOENT (No such file or directory)
newfstatat(AT_FDCWD, "/sys/block/md127/md", 0x7ffe4f46d010, 0) = -1 ENOENT (No such file or directory)
write(2, "mdadm: Unable to initialize sysf"..., 34mdadm: Unable to initialize sysfs

jelly avatar Jul 31 '25 16:07 jelly

Reported by Yu as https://bugzilla.redhat.com/show_bug.cgi?id=2385871

jelly avatar Jul 31 '25 21:07 jelly

Nice! So needs a naughty. Let's move this to bots to use as knownissue.

martinpitt avatar Aug 01 '25 04:08 martinpitt

I sent PR #8070 to add a naughty pattern. Taking off the pilot board.

martinpitt avatar Aug 04 '25 13:08 martinpitt

rhel-9-8 Ooops, it happened again

mdadm: error opening /dev/md/ARR: No such file or directory
/dev/sda:   11593         
umount: /dev/sda: not mounted.
Traceback (most recent call last):
  File "/work/make-checkout-workdir/test/verify/check-storage-mdraid", line 319, in testNotRemovingDisks
    b.wait_text(self.card_desc("MDRAID device", "State"), "Running")
    ~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/work/make-checkout-workdir/test/common/testlib.py", line 904, in wait_text
    self.wait_js_cond("ph_text_is(%s,%s)" % (jsquote(selector), jsquote(text)),
    ~~~~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
                      error_description="() => 'actual text: ' + ph_text(%s)" % jsquote(selector))
                      ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/work/make-checkout-workdir/test/common/testlib.py", line 842, in wait_js_cond
    raise Error(f"timeout\nwait_js_cond({cond}): {last_error.msg}") from None
testlib.Error: timeout
wait_js_cond(ph_text_is("[data-test-card-title='MDRAID device'] [data-test-desc-title='State'] [data-test-value=true]","Running")): Error: actual text: Not running

Traceback (most recent call last):
  File "/work/make-checkout-workdir/bots/machine/machine_core/ssh_connection.py", line 337, in execute
    res = subprocess.run(command_line,
                         input=input.encode("UTF-8") if input else b'',
                         stdout=stdout, check=check)
  File "/usr/lib64/python3.13/subprocess.py", line 577, in run
    raise CalledProcessError(retcode, process.args,
                             output=stdout, stderr=stderr)
subprocess.CalledProcessError: Command '('env', '-u', 'LANGUAGE', 'LC_ALL=C', 'ssh', '-F', 'none', '-o', 'BatchMode=yes', '-o', 'IdentitiesOnly=yes', '-o', 'PKCS11Provider=none', '-o', 'StrictHostKeyChecking=no', '-o', 'UserKnownHostsFile=/dev/null', '-oHostname=127.0.0.2', '-oLogLevel=ERROR', '-oPort=2301', '-oUser=root', '-oControlPath=/tmp/.cockpit-test-resources/ssh-%C-14029', 'vm', 'set -e;', 'mdadm --stop /dev/md/ARR')' returned non-zero exit status 1.

# Result testNotRemovingDisks (__main__.TestStorageMdRaid.testNotRemovingDisks) failed
# 1 TEST FAILED [103s on c69dc8e3616e]
not ok 51 test/verify/check-storage-mdraid TestStorageMdRaid.testNotRemovingDisks [ND@1]

First occurrence: 2025-11-03T09:38:34.150537+00:00 | revision bcce0a55a5e865e47f74c6eb9756f3742c8e3f8d Times recorded: 1 Latest occurrences:

  • 2025-11-03T09:38:34.150537+00:00 | revision bcce0a55a5e865e47f74c6eb9756f3742c8e3f8d

cockpituous avatar Nov 03 '25 09:11 cockpituous

rhel-9-8 Ooops, it happened again

rmmod: ERROR: Module scsi_debug is in use
rmmod: ERROR: Module scsi_debug is in use
rmmod: ERROR: Module scsi_debug is in use
rmmod: ERROR: Module scsi_debug is in use
rmmod: ERROR: Module scsi_debug is in use
rmmod: ERROR: Module scsi_debug is in use
rmmod: ERROR: Module scsi_debug is in use
rmmod: ERROR: Module scsi_debug is in use
rmmod: ERROR: Module scsi_debug is in use
rmmod: ERROR: Module scsi_debug is in use
rmmod: ERROR: Module scsi_debug is in use
rmmod: ERROR: Module scsi_debug is in use
rmmod: ERROR: Module scsi_debug is in use
Traceback (most recent call last):
  File "/work/make-checkout-workdir/test/verify/check-storage-mdraid", line 170, in testRaid
    b.wait_text(self.card_desc("MDRAID device", "State"), "Running")
    ~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/work/make-checkout-workdir/test/common/testlib.py", line 904, in wait_text
    self.wait_js_cond("ph_text_is(%s,%s)" % (jsquote(selector), jsquote(text)),
    ~~~~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
                      error_description="() => 'actual text: ' + ph_text(%s)" % jsquote(selector))
                      ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/work/make-checkout-workdir/test/common/testlib.py", line 842, in wait_js_cond
    raise Error(f"timeout\nwait_js_cond({cond}): {last_error.msg}") from None
testlib.Error: timeout
wait_js_cond(ph_text_is("[data-test-card-title='MDRAID device'] [data-test-desc-title='State'] [data-test-value=true]","Running")): Error: actual text: Not running

# Result testRaid (__main__.TestStorageMdRaid.testRaid) failed
# 1 TEST FAILED [144s on c69dc8e3616e]
not ok 52 test/verify/check-storage-mdraid TestStorageMdRaid.testRaid [ND@2]

First occurrence: 2025-11-03T09:39:15.121852+00:00 | revision bcce0a55a5e865e47f74c6eb9756f3742c8e3f8d Times recorded: 1 Latest occurrences:

  • 2025-11-03T09:39:15.121852+00:00 | revision bcce0a55a5e865e47f74c6eb9756f3742c8e3f8d
File "/work/make-checkout-workdir/test/common/testlib.py", line 842, in wait_js_cond
    raise Error(f"timeout\nwait_js_cond({cond}): {last_error.msg}") from None
testlib.Error: timeout
wait_js_cond(ph_text_is("[data-test-card-title='MDRAID device'] [data-test-desc-title='State'] [data-test-value=true]","Running")): Error: actual text: Not running

Wrote screenshot to TestStorageMdRaid-testRaid-rhel-9-8-127.0.0.2-2701-FAIL.png
Wrote HTML dump to TestStorageMdRaid-testRaid-rhel-9-8-127.0.0.2-2701-FAIL.html
Wrote JS log to TestStorageMdRaid-testRaid-rhel-9-8-127.0.0.2-2701-FAIL.js.log
Journal extracted to TestStorageMdRaid-testRaid-rhel-9-8-127.0.0.2-2701-FAIL.log.gz
> warn: transport closed: disconnected
umount: /run/user/1000: target is busy.
/dev/sda:         14554   
umount: /dev/sda: not mounted.
Traceback (most recent call last):
  File "/work/make-checkout-workdir/test/verify/check-storage-mdraid", line 170, in testRaid
    b.wait_text(self.card_desc("MDRAID device", "State"), "Running")
    ~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/work/make-checkout-workdir/test/common/testlib.py", line 904, in wait_text
    self.wait_js_cond("ph_text_is(%s,%s)" % (jsquote(selector), jsquote(text)),
    ~~~~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
                      error_description="() => 'actual text: ' + ph_text(%s)" % jsquote(selector))
                      ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/work/make-checkout-workdir/test/common/testlib.py", line 842, in wait_js_cond
    raise Error(f"timeout\nwait_js_cond({cond}): {last_error.msg}") from None
testlib.Error: timeout
wait_js_cond(ph_text_is("[data-test-card-title='MDRAID device'] [data-test-desc-title='State'] [data-test-value=true]","Running")): Error: actual text: Not running

# Result testRaid (__main__.TestStorageMdRaid.testRaid) failed
# 1 TEST FAILED [123s on 507b8aad44a7]
not ok 52 test/verify/check-storage-mdraid TestStorageMdRaid.testRaid [ND@4]

First occurrence: 2025-11-03T14:10:39.952652+00:00 | revision db09e9da840d3b71d2f38237d3afde78ae53bf04 Times recorded: 1 Latest occurrences:

  • 2025-11-03T14:10:39.952652+00:00 | revision db09e9da840d3b71d2f38237d3afde78ae53bf04
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/work/make-checkout-workdir/test/common/testlib.py", line 842, in wait_js_cond
    raise Error(f"timeout\nwait_js_cond({cond}): {last_error.msg}") from None
testlib.Error: timeout
wait_js_cond(ph_text_is("[data-test-card-title='MDRAID device'] [data-test-desc-title='State'] [data-test-value=true]","Running")): Error: actual text: Not running

Wrote screenshot to TestStorageMdRaid-testRaid-rhel-9-8-127.0.0.2-2701-FAIL.png
Wrote HTML dump to TestStorageMdRaid-testRaid-rhel-9-8-127.0.0.2-2701-FAIL.html
Wrote JS log to TestStorageMdRaid-testRaid-rhel-9-8-127.0.0.2-2701-FAIL.js.log
Journal extracted to TestStorageMdRaid-testRaid-rhel-9-8-127.0.0.2-2701-FAIL.log.gz
> warn: transport closed: disconnected
/dev/sda:            13343
umount: /dev/sda: not mounted.
Traceback (most recent call last):
  File "/work/make-checkout-workdir/test/verify/check-storage-mdraid", line 170, in testRaid
    b.wait_text(self.card_desc("MDRAID device", "State"), "Running")
    ~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/work/make-checkout-workdir/test/common/testlib.py", line 904, in wait_text
    self.wait_js_cond("ph_text_is(%s,%s)" % (jsquote(selector), jsquote(text)),
    ~~~~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
                      error_description="() => 'actual text: ' + ph_text(%s)" % jsquote(selector))
                      ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/work/make-checkout-workdir/test/common/testlib.py", line 842, in wait_js_cond
    raise Error(f"timeout\nwait_js_cond({cond}): {last_error.msg}") from None
testlib.Error: timeout
wait_js_cond(ph_text_is("[data-test-card-title='MDRAID device'] [data-test-desc-title='State'] [data-test-value=true]","Running")): Error: actual text: Not running

# Result testRaid (__main__.TestStorageMdRaid.testRaid) failed
# 1 TEST FAILED [113s on fa4b90a5a318]
not ok 52 test/verify/check-storage-mdraid TestStorageMdRaid.testRaid [ND@4]

First occurrence: 2025-11-03T15:27:29.503579+00:00 | revision b8dc8db416185cfe2c3f7e2efc4fc4616f816b5f Times recorded: 1 Latest occurrences:

  • 2025-11-03T15:27:29.503579+00:00 | revision b8dc8db416185cfe2c3f7e2efc4fc4616f816b5f
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/work/make-checkout-workdir/test/common/testlib.py", line 842, in wait_js_cond
    raise Error(f"timeout\nwait_js_cond({cond}): {last_error.msg}") from None
testlib.Error: timeout
wait_js_cond(ph_text_is("[data-test-card-title='MDRAID device'] [data-test-desc-title='State'] [data-test-value=true]","Running")): Error: actual text: Not running

Wrote screenshot to TestStorageMdRaid-testRaid-rhel-9-8-127.0.0.2-2701-FAIL.png
Wrote HTML dump to TestStorageMdRaid-testRaid-rhel-9-8-127.0.0.2-2701-FAIL.html
Wrote JS log to TestStorageMdRaid-testRaid-rhel-9-8-127.0.0.2-2701-FAIL.js.log
Journal extracted to TestStorageMdRaid-testRaid-rhel-9-8-127.0.0.2-2701-FAIL.log.gz
> warn: transport closed: disconnected
/dev/sda:          14631  
umount: /dev/sda: not mounted.
Traceback (most recent call last):
  File "/work/make-checkout-workdir/test/verify/check-storage-mdraid", line 170, in testRaid
    b.wait_text(self.card_desc("MDRAID device", "State"), "Running")
    ~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/work/make-checkout-workdir/test/common/testlib.py", line 904, in wait_text
    self.wait_js_cond("ph_text_is(%s,%s)" % (jsquote(selector), jsquote(text)),
    ~~~~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
                      error_description="() => 'actual text: ' + ph_text(%s)" % jsquote(selector))
                      ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/work/make-checkout-workdir/test/common/testlib.py", line 842, in wait_js_cond
    raise Error(f"timeout\nwait_js_cond({cond}): {last_error.msg}") from None
testlib.Error: timeout
wait_js_cond(ph_text_is("[data-test-card-title='MDRAID device'] [data-test-desc-title='State'] [data-test-value=true]","Running")): Error: actual text: Not running

# Result testRaid (__main__.TestStorageMdRaid.testRaid) failed
# 1 TEST FAILED [118s on 7b5d3f1db74d]
not ok 52 test/verify/check-storage-mdraid TestStorageMdRaid.testRaid [ND@4]

First occurrence: 2025-11-03T16:36:00.012953+00:00 | revision 2a3d0e80258eac01ab8e10bd45701fd54dde5b4c Times recorded: 1 Latest occurrences:

  • 2025-11-03T16:36:00.012953+00:00 | revision 2a3d0e80258eac01ab8e10bd45701fd54dde5b4c
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/work/make-checkout-workdir/test/common/testlib.py", line 842, in wait_js_cond
    raise Error(f"timeout\nwait_js_cond({cond}): {last_error.msg}") from None
testlib.Error: timeout
wait_js_cond(ph_text_is("[data-test-card-title='MDRAID device'] [data-test-desc-title='State'] [data-test-value=true]","Running")): Error: actual text: Not running

Wrote screenshot to TestStorageMdRaid-testRaid-rhel-9-8-127.0.0.2-2901-FAIL.png
Wrote HTML dump to TestStorageMdRaid-testRaid-rhel-9-8-127.0.0.2-2901-FAIL.html
Wrote JS log to TestStorageMdRaid-testRaid-rhel-9-8-127.0.0.2-2901-FAIL.js.log
Journal extracted to TestStorageMdRaid-testRaid-rhel-9-8-127.0.0.2-2901-FAIL.log.gz
> warn: transport closed: disconnected
/dev/sda:            13968
umount: /dev/sda: not mounted.
Traceback (most recent call last):
  File "/work/make-checkout-workdir/test/verify/check-storage-mdraid", line 170, in testRaid
    b.wait_text(self.card_desc("MDRAID device", "State"), "Running")
    ~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/work/make-checkout-workdir/test/common/testlib.py", line 904, in wait_text
    self.wait_js_cond("ph_text_is(%s,%s)" % (jsquote(selector), jsquote(text)),
    ~~~~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
                      error_description="() => 'actual text: ' + ph_text(%s)" % jsquote(selector))
                      ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/work/make-checkout-workdir/test/common/testlib.py", line 842, in wait_js_cond
    raise Error(f"timeout\nwait_js_cond({cond}): {last_error.msg}") from None
testlib.Error: timeout
wait_js_cond(ph_text_is("[data-test-card-title='MDRAID device'] [data-test-desc-title='State'] [data-test-value=true]","Running")): Error: actual text: Not running

# Result testRaid (__main__.TestStorageMdRaid.testRaid) failed
# 1 TEST FAILED [110s on 459a793ee8c2]
not ok 52 test/verify/check-storage-mdraid TestStorageMdRaid.testRaid [ND@6]

First occurrence: 2025-11-04T08:52:50.741813+00:00 | revision 3e7d4497e358382ae830d1448ab5454ce9f075cc Times recorded: 1 Latest occurrences:

  • 2025-11-04T08:52:50.741813+00:00 | revision 3e7d4497e358382ae830d1448ab5454ce9f075cc
rmmod: ERROR: Module scsi_debug is in use
rmmod: ERROR: Module scsi_debug is in use
rmmod: ERROR: Module scsi_debug is in use
rmmod: ERROR: Module scsi_debug is in use
rmmod: ERROR: Module scsi_debug is in use
rmmod: ERROR: Module scsi_debug is in use
rmmod: ERROR: Module scsi_debug is in use
rmmod: ERROR: Module scsi_debug is in use
rmmod: ERROR: Module scsi_debug is in use
rmmod: ERROR: Module scsi_debug is in use
rmmod: ERROR: Module scsi_debug is in use
rmmod: ERROR: Module scsi_debug is in use
rmmod: ERROR: Module scsi_debug is in use
Traceback (most recent call last):
  File "/work/make-checkout-workdir/test/verify/check-storage-mdraid", line 170, in testRaid
    b.wait_text(self.card_desc("MDRAID device", "State"), "Running")
    ~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/work/make-checkout-workdir/test/common/testlib.py", line 904, in wait_text
    self.wait_js_cond("ph_text_is(%s,%s)" % (jsquote(selector), jsquote(text)),
    ~~~~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
                      error_description="() => 'actual text: ' + ph_text(%s)" % jsquote(selector))
                      ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/work/make-checkout-workdir/test/common/testlib.py", line 842, in wait_js_cond
    raise Error(f"timeout\nwait_js_cond({cond}): {last_error.msg}") from None
testlib.Error: timeout
wait_js_cond(ph_text_is("[data-test-card-title='MDRAID device'] [data-test-desc-title='State'] [data-test-value=true]","Running")): Error: actual text: Not running

# Result testRaid (__main__.TestStorageMdRaid.testRaid) failed
# 1 TEST FAILED [138s on 8711fbc218e7]
not ok 52 test/verify/check-storage-mdraid TestStorageMdRaid.testRaid [ND@2]

First occurrence: 2025-11-04T09:06:25.746568+00:00 | revision c6a857c9b6eccef4db2e1efebe93ad59ce8aefdb Times recorded: 1 Latest occurrences:

  • 2025-11-04T09:06:25.746568+00:00 | revision c6a857c9b6eccef4db2e1efebe93ad59ce8aefdb
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/work/make-checkout-workdir/test/common/testlib.py", line 857, in wait_js_cond
    raise Error(f"timeout\nwait_js_cond({cond}): {last_error.msg}") from None
testlib.Error: timeout
wait_js_cond(ph_text_is("[data-test-card-title='MDRAID device'] [data-test-desc-title='State'] [data-test-value=true]","Running")): Error: actual text: Not running

Wrote screenshot to TestStorageMdRaid-testRaid-rhel-9-8-127.0.0.2-2201-FAIL.png
Wrote HTML dump to TestStorageMdRaid-testRaid-rhel-9-8-127.0.0.2-2201-FAIL.html
Wrote JS log to TestStorageMdRaid-testRaid-rhel-9-8-127.0.0.2-2201-FAIL.js.log
Journal extracted to TestStorageMdRaid-testRaid-rhel-9-8-127.0.0.2-2201-FAIL.log.gz
> warn: transport closed: disconnected
/dev/sda:      11530      
umount: /dev/sda: not mounted.
Traceback (most recent call last):
  File "/work/make-checkout-workdir/test/verify/check-storage-mdraid", line 170, in testRaid
    b.wait_text(self.card_desc("MDRAID device", "State"), "Running")
    ~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/work/make-checkout-workdir/test/common/testlib.py", line 919, in wait_text
    self.wait_js_cond("ph_text_is(%s,%s)" % (jsquote(selector), jsquote(text)),
    ~~~~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
                      error_description="() => 'actual text: ' + ph_text(%s)" % jsquote(selector))
                      ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/work/make-checkout-workdir/test/common/testlib.py", line 857, in wait_js_cond
    raise Error(f"timeout\nwait_js_cond({cond}): {last_error.msg}") from None
testlib.Error: timeout
wait_js_cond(ph_text_is("[data-test-card-title='MDRAID device'] [data-test-desc-title='State'] [data-test-value=true]","Running")): Error: actual text: Not running

# Result testRaid (__main__.TestStorageMdRaid.testRaid) failed
# 1 TEST FAILED [110s on 27fa958b4963]
not ok 30 test/verify/check-storage-mdraid TestStorageMdRaid.testRaid [ND@0]

First occurrence: 2025-11-04T10:36:27.231659+00:00 | revision 41217dd04d6bca6293e1286343f5011cc28fa34b Times recorded: 1 Latest occurrences:

  • 2025-11-04T10:36:27.231659+00:00 | revision 41217dd04d6bca6293e1286343f5011cc28fa34b
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/work/make-checkout-workdir/test/common/testlib.py", line 842, in wait_js_cond
    raise Error(f"timeout\nwait_js_cond({cond}): {last_error.msg}") from None
testlib.Error: timeout
wait_js_cond(ph_text_is("[data-test-card-title='MDRAID device'] [data-test-desc-title='State'] [data-test-value=true]","Running")): Error: actual text: Not running

Wrote screenshot to TestStorageMdRaid-testRaid-rhel-9-8-127.0.0.2-2901-FAIL.png
Wrote HTML dump to TestStorageMdRaid-testRaid-rhel-9-8-127.0.0.2-2901-FAIL.html
Wrote JS log to TestStorageMdRaid-testRaid-rhel-9-8-127.0.0.2-2901-FAIL.js.log
Journal extracted to TestStorageMdRaid-testRaid-rhel-9-8-127.0.0.2-2901-FAIL.log.gz
> warn: transport closed: disconnected
 14269/dev/sda:           
umount: /dev/sda: not mounted.
Traceback (most recent call last):
  File "/work/make-checkout-workdir/test/verify/check-storage-mdraid", line 170, in testRaid
    b.wait_text(self.card_desc("MDRAID device", "State"), "Running")
    ~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/work/make-checkout-workdir/test/common/testlib.py", line 904, in wait_text
    self.wait_js_cond("ph_text_is(%s,%s)" % (jsquote(selector), jsquote(text)),
    ~~~~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
                      error_description="() => 'actual text: ' + ph_text(%s)" % jsquote(selector))
                      ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/work/make-checkout-workdir/test/common/testlib.py", line 842, in wait_js_cond
    raise Error(f"timeout\nwait_js_cond({cond}): {last_error.msg}") from None
testlib.Error: timeout
wait_js_cond(ph_text_is("[data-test-card-title='MDRAID device'] [data-test-desc-title='State'] [data-test-value=true]","Running")): Error: actual text: Not running

# Result testRaid (__main__.TestStorageMdRaid.testRaid) failed
# 1 TEST FAILED [115s on afcf3b099322]
not ok 52 test/verify/check-storage-mdraid TestStorageMdRaid.testRaid [ND@6]

First occurrence: 2025-11-04T12:55:42.722526+00:00 | revision ce23dbdfcce6a17fba63d36104788ba60e885914 Times recorded: 1 Latest occurrences:

  • 2025-11-04T12:55:42.722526+00:00 | revision ce23dbdfcce6a17fba63d36104788ba60e885914
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/work/make-checkout-workdir/test/common/testlib.py", line 842, in wait_js_cond
    raise Error(f"timeout\nwait_js_cond({cond}): {last_error.msg}") from None
testlib.Error: timeout
wait_js_cond(ph_text_is("[data-test-card-title='MDRAID device'] [data-test-desc-title='State'] [data-test-value=true]","Running")): Error: actual text: Not running

Wrote screenshot to TestStorageMdRaid-testRaid-rhel-9-8-127.0.0.2-2501-FAIL.png
Wrote HTML dump to TestStorageMdRaid-testRaid-rhel-9-8-127.0.0.2-2501-FAIL.html
Wrote JS log to TestStorageMdRaid-testRaid-rhel-9-8-127.0.0.2-2501-FAIL.js.log
Journal extracted to TestStorageMdRaid-testRaid-rhel-9-8-127.0.0.2-2501-FAIL.log.gz
> warn: transport closed: disconnected
/dev/sda:   13828         
umount: /dev/sda: not mounted.
Traceback (most recent call last):
  File "/work/make-checkout-workdir/test/verify/check-storage-mdraid", line 170, in testRaid
    b.wait_text(self.card_desc("MDRAID device", "State"), "Running")
    ~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/work/make-checkout-workdir/test/common/testlib.py", line 904, in wait_text
    self.wait_js_cond("ph_text_is(%s,%s)" % (jsquote(selector), jsquote(text)),
    ~~~~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
                      error_description="() => 'actual text: ' + ph_text(%s)" % jsquote(selector))
                      ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/work/make-checkout-workdir/test/common/testlib.py", line 842, in wait_js_cond
    raise Error(f"timeout\nwait_js_cond({cond}): {last_error.msg}") from None
testlib.Error: timeout
wait_js_cond(ph_text_is("[data-test-card-title='MDRAID device'] [data-test-desc-title='State'] [data-test-value=true]","Running")): Error: actual text: Not running

# Result testRaid (__main__.TestStorageMdRaid.testRaid) failed
# 1 TEST FAILED [114s on f3d7044f9bfe]
not ok 52 test/verify/check-storage-mdraid TestStorageMdRaid.testRaid [ND@3]

First occurrence: 2025-11-04T13:32:46.930338+00:00 | revision 786d4332fd5604401f2b92ba6bc99545bccb4976 Times recorded: 1 Latest occurrences:

  • 2025-11-04T13:32:46.930338+00:00 | revision 786d4332fd5604401f2b92ba6bc99545bccb4976
rmmod: ERROR: Module scsi_debug is in use
rmmod: ERROR: Module scsi_debug is in use
rmmod: ERROR: Module scsi_debug is in use
rmmod: ERROR: Module scsi_debug is in use
rmmod: ERROR: Module scsi_debug is in use
rmmod: ERROR: Module scsi_debug is in use
rmmod: ERROR: Module scsi_debug is in use
rmmod: ERROR: Module scsi_debug is in use
rmmod: ERROR: Module scsi_debug is in use
rmmod: ERROR: Module scsi_debug is in use
rmmod: ERROR: Module scsi_debug is in use
rmmod: ERROR: Module scsi_debug is in use
rmmod: ERROR: Module scsi_debug is in use
Traceback (most recent call last):
  File "/work/make-checkout-workdir/test/verify/check-storage-mdraid", line 170, in testRaid
    b.wait_text(self.card_desc("MDRAID device", "State"), "Running")
    ~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/work/make-checkout-workdir/test/common/testlib.py", line 921, in wait_text
    self.wait_js_cond("ph_text_is(%s,%s)" % (jsquote(selector), jsquote(text)),
    ~~~~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
                      error_description="() => 'actual text: ' + ph_text(%s)" % jsquote(selector))
                      ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/work/make-checkout-workdir/test/common/testlib.py", line 859, in wait_js_cond
    raise Error(f"timeout\nwait_js_cond({cond}): {last_error.msg}") from None
testlib.Error: timeout
wait_js_cond(ph_text_is("[data-test-card-title='MDRAID device'] [data-test-desc-title='State'] [data-test-value=true]","Running")): Error: actual text: Not running

# Result testRaid (__main__.TestStorageMdRaid.testRaid) failed
# 1 TEST FAILED [146s on 99f91eba8e0a]
not ok 52 test/verify/check-storage-mdraid TestStorageMdRaid.testRaid [ND@2]

First occurrence: 2025-11-19T08:31:46.612822+00:00 | revision 95f0f822a6ef09ffa2a2c43599da5a4399a4f16e Times recorded: 1 Latest occurrences:

  • 2025-11-19T08:31:46.612822+00:00 | revision 95f0f822a6ef09ffa2a2c43599da5a4399a4f16e

cockpituous avatar Nov 03 '25 09:11 cockpituous

centos-9-bootc Ooops, it happened again

mdadm: error opening /dev/md/ARR: No such file or directory
/dev/sda:             3512
umount: /dev/sda: not mounted.
Traceback (most recent call last):
  File "/work/make-checkout-workdir/test/verify/check-storage-mdraid", line 319, in testNotRemovingDisks
    b.wait_text(self.card_desc("MDRAID device", "State"), "Running")
    ~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/work/make-checkout-workdir/test/common/testlib.py", line 921, in wait_text
    self.wait_js_cond("ph_text_is(%s,%s)" % (jsquote(selector), jsquote(text)),
    ~~~~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
                      error_description="() => 'actual text: ' + ph_text(%s)" % jsquote(selector))
                      ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/work/make-checkout-workdir/test/common/testlib.py", line 859, in wait_js_cond
    raise Error(f"timeout\nwait_js_cond({cond}): {last_error.msg}") from None
testlib.Error: timeout
wait_js_cond(ph_text_is("[data-test-card-title='MDRAID device'] [data-test-desc-title='State'] [data-test-value=true]","Running")): Error: actual text: Not running

Traceback (most recent call last):
  File "/work/make-checkout-workdir/bots/machine/machine_core/ssh_connection.py", line 337, in execute
    res = subprocess.run(command_line,
                         input=input.encode("UTF-8") if input else b'',
                         stdout=stdout, check=check)
  File "/usr/lib64/python3.14/subprocess.py", line 577, in run
    raise CalledProcessError(retcode, process.args,
                             output=stdout, stderr=stderr)
subprocess.CalledProcessError: Command '('env', '-u', 'LANGUAGE', 'LC_ALL=C', 'ssh', '-F', 'none', '-o', 'BatchMode=yes', '-o', 'IdentitiesOnly=yes', '-o', 'PKCS11Provider=none', '-o', 'StrictHostKeyChecking=no', '-o', 'UserKnownHostsFile=/dev/null', '-oHostname=127.0.0.2', '-oLogLevel=ERROR', '-oPort=2501', '-oUser=root', '-oControlPath=/tmp/.cockpit-test-resources/ssh-%C-12052', 'vm', 'set -e;', 'mdadm --stop /dev/md/ARR')' returned non-zero exit status 1.

# Result testNotRemovingDisks (__main__.TestStorageMdRaid.testNotRemovingDisks) failed
# 1 TEST FAILED [96s on 02de698b99a2]
not ok 51 test/verify/check-storage-mdraid TestStorageMdRaid.testNotRemovingDisks [ND@3]

First occurrence: 2025-11-14T23:19:24.254619+00:00 | revision 766dd6dc82dfab8430881424cb6f601761038df9 Times recorded: 1 Latest occurrences:

  • 2025-11-14T23:19:24.254619+00:00 | revision 766dd6dc82dfab8430881424cb6f601761038df9

cockpituous avatar Nov 14 '25 23:11 cockpituous

centos-9-bootc Ooops, it happened again

rmmod: ERROR: Module scsi_debug is in use
rmmod: ERROR: Module scsi_debug is in use
rmmod: ERROR: Module scsi_debug is in use
rmmod: ERROR: Module scsi_debug is in use
rmmod: ERROR: Module scsi_debug is in use
rmmod: ERROR: Module scsi_debug is in use
rmmod: ERROR: Module scsi_debug is in use
rmmod: ERROR: Module scsi_debug is in use
rmmod: ERROR: Module scsi_debug is in use
rmmod: ERROR: Module scsi_debug is in use
rmmod: ERROR: Module scsi_debug is in use
rmmod: ERROR: Module scsi_debug is in use
rmmod: ERROR: Module scsi_debug is in use
Traceback (most recent call last):
  File "/work/make-checkout-workdir/test/verify/check-storage-mdraid", line 170, in testRaid
    b.wait_text(self.card_desc("MDRAID device", "State"), "Running")
    ~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/work/make-checkout-workdir/test/common/testlib.py", line 921, in wait_text
    self.wait_js_cond("ph_text_is(%s,%s)" % (jsquote(selector), jsquote(text)),
    ~~~~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
                      error_description="() => 'actual text: ' + ph_text(%s)" % jsquote(selector))
                      ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/work/make-checkout-workdir/test/common/testlib.py", line 859, in wait_js_cond
    raise Error(f"timeout\nwait_js_cond({cond}): {last_error.msg}") from None
testlib.Error: timeout
wait_js_cond(ph_text_is("[data-test-card-title='MDRAID device'] [data-test-desc-title='State'] [data-test-value=true]","Running")): Error: actual text: Not running

# Result testRaid (__main__.TestStorageMdRaid.testRaid) failed
# 1 TEST FAILED [135s on 02de698b99a2]
not ok 52 test/verify/check-storage-mdraid TestStorageMdRaid.testRaid [ND@6]

First occurrence: 2025-11-14T23:19:56.891556+00:00 | revision 766dd6dc82dfab8430881424cb6f601761038df9 Times recorded: 1 Latest occurrences:

  • 2025-11-14T23:19:56.891556+00:00 | revision 766dd6dc82dfab8430881424cb6f601761038df9
rmmod: ERROR: Module scsi_debug is in use
rmmod: ERROR: Module scsi_debug is in use
rmmod: ERROR: Module scsi_debug is in use
rmmod: ERROR: Module scsi_debug is in use
rmmod: ERROR: Module scsi_debug is in use
rmmod: ERROR: Module scsi_debug is in use
rmmod: ERROR: Module scsi_debug is in use
rmmod: ERROR: Module scsi_debug is in use
rmmod: ERROR: Module scsi_debug is in use
rmmod: ERROR: Module scsi_debug is in use
rmmod: ERROR: Module scsi_debug is in use
rmmod: ERROR: Module scsi_debug is in use
rmmod: ERROR: Module scsi_debug is in use
Traceback (most recent call last):
  File "/work/make-checkout-workdir/test/verify/check-storage-mdraid", line 170, in testRaid
    b.wait_text(self.card_desc("MDRAID device", "State"), "Running")
    ~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/work/make-checkout-workdir/test/common/testlib.py", line 921, in wait_text
    self.wait_js_cond("ph_text_is(%s,%s)" % (jsquote(selector), jsquote(text)),
    ~~~~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
                      error_description="() => 'actual text: ' + ph_text(%s)" % jsquote(selector))
                      ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/work/make-checkout-workdir/test/common/testlib.py", line 859, in wait_js_cond
    raise Error(f"timeout\nwait_js_cond({cond}): {last_error.msg}") from None
testlib.Error: timeout
wait_js_cond(ph_text_is("[data-test-card-title='MDRAID device'] [data-test-desc-title='State'] [data-test-value=true]","Running")): Error: actual text: Not running

# Result testRaid (__main__.TestStorageMdRaid.testRaid) failed
# 1 TEST FAILED [131s on 39f46222959e]
not ok 52 test/verify/check-storage-mdraid TestStorageMdRaid.testRaid [ND@4]

First occurrence: 2025-11-16T04:14:56.662830+00:00 | revision 86735e244833fa07cd172fbb5236f8805622b2f7 Times recorded: 1 Latest occurrences:

  • 2025-11-16T04:14:56.662830+00:00 | revision 86735e244833fa07cd172fbb5236f8805622b2f7
rmmod: ERROR: Module scsi_debug is in use
rmmod: ERROR: Module scsi_debug is in use
rmmod: ERROR: Module scsi_debug is in use
rmmod: ERROR: Module scsi_debug is in use
rmmod: ERROR: Module scsi_debug is in use
rmmod: ERROR: Module scsi_debug is in use
rmmod: ERROR: Module scsi_debug is in use
rmmod: ERROR: Module scsi_debug is in use
rmmod: ERROR: Module scsi_debug is in use
rmmod: ERROR: Module scsi_debug is in use
rmmod: ERROR: Module scsi_debug is in use
rmmod: ERROR: Module scsi_debug is in use
rmmod: ERROR: Module scsi_debug is in use
Traceback (most recent call last):
  File "/work/make-checkout-workdir/test/verify/check-storage-mdraid", line 170, in testRaid
    b.wait_text(self.card_desc("MDRAID device", "State"), "Running")
    ~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/work/make-checkout-workdir/test/common/testlib.py", line 921, in wait_text
    self.wait_js_cond("ph_text_is(%s,%s)" % (jsquote(selector), jsquote(text)),
    ~~~~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
                      error_description="() => 'actual text: ' + ph_text(%s)" % jsquote(selector))
                      ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/work/make-checkout-workdir/test/common/testlib.py", line 859, in wait_js_cond
    raise Error(f"timeout\nwait_js_cond({cond}): {last_error.msg}") from None
testlib.Error: timeout
wait_js_cond(ph_text_is("[data-test-card-title='MDRAID device'] [data-test-desc-title='State'] [data-test-value=true]","Running")): Error: actual text: Not running

# Result testRaid (__main__.TestStorageMdRaid.testRaid) failed
# 1 TEST FAILED [141s on d638db9e3fd7]
not ok 52 test/verify/check-storage-mdraid TestStorageMdRaid.testRaid [ND@0]

First occurrence: 2025-11-17T02:51:58.618209+00:00 | revision 2a4466d5fe915cb0d67481c0f063f0f100ff6645 Times recorded: 1 Latest occurrences:

  • 2025-11-17T02:51:58.618209+00:00 | revision 2a4466d5fe915cb0d67481c0f063f0f100ff6645
rmmod: ERROR: Module scsi_debug is in use
rmmod: ERROR: Module scsi_debug is in use
rmmod: ERROR: Module scsi_debug is in use
rmmod: ERROR: Module scsi_debug is in use
rmmod: ERROR: Module scsi_debug is in use
rmmod: ERROR: Module scsi_debug is in use
rmmod: ERROR: Module scsi_debug is in use
rmmod: ERROR: Module scsi_debug is in use
rmmod: ERROR: Module scsi_debug is in use
rmmod: ERROR: Module scsi_debug is in use
rmmod: ERROR: Module scsi_debug is in use
rmmod: ERROR: Module scsi_debug is in use
rmmod: ERROR: Module scsi_debug is in use
Traceback (most recent call last):
  File "/work/make-checkout-workdir/test/verify/check-storage-mdraid", line 170, in testRaid
    b.wait_text(self.card_desc("MDRAID device", "State"), "Running")
    ~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/work/make-checkout-workdir/test/common/testlib.py", line 921, in wait_text
    self.wait_js_cond("ph_text_is(%s,%s)" % (jsquote(selector), jsquote(text)),
    ~~~~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
                      error_description="() => 'actual text: ' + ph_text(%s)" % jsquote(selector))
                      ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/work/make-checkout-workdir/test/common/testlib.py", line 859, in wait_js_cond
    raise Error(f"timeout\nwait_js_cond({cond}): {last_error.msg}") from None
testlib.Error: timeout
wait_js_cond(ph_text_is("[data-test-card-title='MDRAID device'] [data-test-desc-title='State'] [data-test-value=true]","Running")): Error: actual text: Not running

# Result testRaid (__main__.TestStorageMdRaid.testRaid) failed
# 1 TEST FAILED [136s on a78792f9d970]
not ok 52 test/verify/check-storage-mdraid TestStorageMdRaid.testRaid [ND@6]

First occurrence: 2025-11-17T08:54:52.485135+00:00 | revision 3725f1ba8a8170f4132868ff7efdaec9623b36be Times recorded: 1 Latest occurrences:

  • 2025-11-17T08:54:52.485135+00:00 | revision 3725f1ba8a8170f4132868ff7efdaec9623b36be
rmmod: ERROR: Module scsi_debug is in use
rmmod: ERROR: Module scsi_debug is in use
rmmod: ERROR: Module scsi_debug is in use
rmmod: ERROR: Module scsi_debug is in use
rmmod: ERROR: Module scsi_debug is in use
rmmod: ERROR: Module scsi_debug is in use
rmmod: ERROR: Module scsi_debug is in use
rmmod: ERROR: Module scsi_debug is in use
rmmod: ERROR: Module scsi_debug is in use
rmmod: ERROR: Module scsi_debug is in use
rmmod: ERROR: Module scsi_debug is in use
rmmod: ERROR: Module scsi_debug is in use
rmmod: ERROR: Module scsi_debug is in use
Traceback (most recent call last):
  File "/work/make-checkout-workdir/test/verify/check-storage-mdraid", line 170, in testRaid
    b.wait_text(self.card_desc("MDRAID device", "State"), "Running")
    ~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/work/make-checkout-workdir/test/common/testlib.py", line 921, in wait_text
    self.wait_js_cond("ph_text_is(%s,%s)" % (jsquote(selector), jsquote(text)),
    ~~~~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
                      error_description="() => 'actual text: ' + ph_text(%s)" % jsquote(selector))
                      ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/work/make-checkout-workdir/test/common/testlib.py", line 859, in wait_js_cond
    raise Error(f"timeout\nwait_js_cond({cond}): {last_error.msg}") from None
testlib.Error: timeout
wait_js_cond(ph_text_is("[data-test-card-title='MDRAID device'] [data-test-desc-title='State'] [data-test-value=true]","Running")): Error: actual text: Not running

# Result testRaid (__main__.TestStorageMdRaid.testRaid) failed
# 1 TEST FAILED [133s on 7e52e5afd543]
not ok 52 test/verify/check-storage-mdraid TestStorageMdRaid.testRaid [ND@6]

First occurrence: 2025-11-17T23:17:51.104113+00:00 | revision e990f97501dc69e0d5dfdc53427bc207e907c1b5 Times recorded: 1 Latest occurrences:

  • 2025-11-17T23:17:51.104113+00:00 | revision e990f97501dc69e0d5dfdc53427bc207e907c1b5
mdadm: error opening /dev/md/ARR: No such file or directory
/dev/sda:          3672   
umount: /dev/sda: not mounted.
Traceback (most recent call last):
  File "/work/make-checkout-workdir/test/verify/check-storage-mdraid", line 319, in testNotRemovingDisks
    b.wait_text(self.card_desc("MDRAID device", "State"), "Running")
    ~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/work/make-checkout-workdir/test/common/testlib.py", line 921, in wait_text
    self.wait_js_cond("ph_text_is(%s,%s)" % (jsquote(selector), jsquote(text)),
    ~~~~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
                      error_description="() => 'actual text: ' + ph_text(%s)" % jsquote(selector))
                      ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/work/make-checkout-workdir/test/common/testlib.py", line 859, in wait_js_cond
    raise Error(f"timeout\nwait_js_cond({cond}): {last_error.msg}") from None
testlib.Error: timeout
wait_js_cond(ph_text_is("[data-test-card-title='MDRAID device'] [data-test-desc-title='State'] [data-test-value=true]","Running")): Error: actual text: Not running

Traceback (most recent call last):
  File "/work/make-checkout-workdir/bots/machine/machine_core/ssh_connection.py", line 337, in execute
    res = subprocess.run(command_line,
                         input=input.encode("UTF-8") if input else b'',
                         stdout=stdout, check=check)
  File "/usr/lib64/python3.14/subprocess.py", line 577, in run
    raise CalledProcessError(retcode, process.args,
                             output=stdout, stderr=stderr)
subprocess.CalledProcessError: Command '('env', '-u', 'LANGUAGE', 'LC_ALL=C', 'ssh', '-F', 'none', '-o', 'BatchMode=yes', '-o', 'IdentitiesOnly=yes', '-o', 'PKCS11Provider=none', '-o', 'StrictHostKeyChecking=no', '-o', 'UserKnownHostsFile=/dev/null', '-oHostname=127.0.0.2', '-oLogLevel=ERROR', '-oPort=2501', '-oUser=root', '-oControlPath=/tmp/.cockpit-test-resources/ssh-%C-12166', 'vm', 'set -e;', 'mdadm --stop /dev/md/ARR')' returned non-zero exit status 1.

# Result testNotRemovingDisks (__main__.TestStorageMdRaid.testNotRemovingDisks) failed
# 1 TEST FAILED [97s on 2800b5e220aa]
not ok 51 test/verify/check-storage-mdraid TestStorageMdRaid.testNotRemovingDisks [ND@3]

First occurrence: 2025-11-18T08:08:43.976916+00:00 | revision 9219a89d5c50c2b8f71b4101e44c58397cbb6b94 Times recorded: 1 Latest occurrences:

  • 2025-11-18T08:08:43.976916+00:00 | revision 9219a89d5c50c2b8f71b4101e44c58397cbb6b94
rmmod: ERROR: Module scsi_debug is in use
rmmod: ERROR: Module scsi_debug is in use
rmmod: ERROR: Module scsi_debug is in use
rmmod: ERROR: Module scsi_debug is in use
rmmod: ERROR: Module scsi_debug is in use
rmmod: ERROR: Module scsi_debug is in use
rmmod: ERROR: Module scsi_debug is in use
rmmod: ERROR: Module scsi_debug is in use
rmmod: ERROR: Module scsi_debug is in use
rmmod: ERROR: Module scsi_debug is in use
rmmod: ERROR: Module scsi_debug is in use
rmmod: ERROR: Module scsi_debug is in use
rmmod: ERROR: Module scsi_debug is in use
Traceback (most recent call last):
  File "/work/make-checkout-workdir/test/verify/check-storage-mdraid", line 170, in testRaid
    b.wait_text(self.card_desc("MDRAID device", "State"), "Running")
    ~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/work/make-checkout-workdir/test/common/testlib.py", line 921, in wait_text
    self.wait_js_cond("ph_text_is(%s,%s)" % (jsquote(selector), jsquote(text)),
    ~~~~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
                      error_description="() => 'actual text: ' + ph_text(%s)" % jsquote(selector))
                      ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/work/make-checkout-workdir/test/common/testlib.py", line 859, in wait_js_cond
    raise Error(f"timeout\nwait_js_cond({cond}): {last_error.msg}") from None
testlib.Error: timeout
wait_js_cond(ph_text_is("[data-test-card-title='MDRAID device'] [data-test-desc-title='State'] [data-test-value=true]","Running")): Error: actual text: Not running

# Result testRaid (__main__.TestStorageMdRaid.testRaid) failed
# 1 TEST FAILED [134s on e243a4fc1b0f]
not ok 52 test/verify/check-storage-mdraid TestStorageMdRaid.testRaid [ND@0]

First occurrence: 2025-11-18T11:23:31.838923+00:00 | revision 4c3dc51ec1e41e939ee7b5a6493556e7577afcd2 Times recorded: 1 Latest occurrences:

  • 2025-11-18T11:23:31.838923+00:00 | revision 4c3dc51ec1e41e939ee7b5a6493556e7577afcd2
rmmod: ERROR: Module scsi_debug is in use
rmmod: ERROR: Module scsi_debug is in use
rmmod: ERROR: Module scsi_debug is in use
rmmod: ERROR: Module scsi_debug is in use
rmmod: ERROR: Module scsi_debug is in use
rmmod: ERROR: Module scsi_debug is in use
rmmod: ERROR: Module scsi_debug is in use
rmmod: ERROR: Module scsi_debug is in use
rmmod: ERROR: Module scsi_debug is in use
rmmod: ERROR: Module scsi_debug is in use
rmmod: ERROR: Module scsi_debug is in use
rmmod: ERROR: Module scsi_debug is in use
rmmod: ERROR: Module scsi_debug is in use
Traceback (most recent call last):
  File "/work/make-checkout-workdir/test/verify/check-storage-mdraid", line 170, in testRaid
    b.wait_text(self.card_desc("MDRAID device", "State"), "Running")
    ~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/work/make-checkout-workdir/test/common/testlib.py", line 921, in wait_text
    self.wait_js_cond("ph_text_is(%s,%s)" % (jsquote(selector), jsquote(text)),
    ~~~~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
                      error_description="() => 'actual text: ' + ph_text(%s)" % jsquote(selector))
                      ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/work/make-checkout-workdir/test/common/testlib.py", line 859, in wait_js_cond
    raise Error(f"timeout\nwait_js_cond({cond}): {last_error.msg}") from None
testlib.Error: timeout
wait_js_cond(ph_text_is("[data-test-card-title='MDRAID device'] [data-test-desc-title='State'] [data-test-value=true]","Running")): Error: actual text: Not running

# Result testRaid (__main__.TestStorageMdRaid.testRaid) failed
# 1 TEST FAILED [133s on 4c3ff1b68c5f]
not ok 52 test/verify/check-storage-mdraid TestStorageMdRaid.testRaid [ND@4]

First occurrence: 2025-11-18T15:06:45.260668+00:00 | revision f9022dfb6e4ca2e3cdb95a5008a8d035918af025 Times recorded: 1 Latest occurrences:

  • 2025-11-18T15:06:45.260668+00:00 | revision f9022dfb6e4ca2e3cdb95a5008a8d035918af025
rmmod: ERROR: Module scsi_debug is in use
rmmod: ERROR: Module scsi_debug is in use
rmmod: ERROR: Module scsi_debug is in use
rmmod: ERROR: Module scsi_debug is in use
rmmod: ERROR: Module scsi_debug is in use
rmmod: ERROR: Module scsi_debug is in use
rmmod: ERROR: Module scsi_debug is in use
rmmod: ERROR: Module scsi_debug is in use
rmmod: ERROR: Module scsi_debug is in use
rmmod: ERROR: Module scsi_debug is in use
rmmod: ERROR: Module scsi_debug is in use
rmmod: ERROR: Module scsi_debug is in use
rmmod: ERROR: Module scsi_debug is in use
Traceback (most recent call last):
  File "/work/make-checkout-workdir/test/verify/check-storage-mdraid", line 170, in testRaid
    b.wait_text(self.card_desc("MDRAID device", "State"), "Running")
    ~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/work/make-checkout-workdir/test/common/testlib.py", line 921, in wait_text
    self.wait_js_cond("ph_text_is(%s,%s)" % (jsquote(selector), jsquote(text)),
    ~~~~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
                      error_description="() => 'actual text: ' + ph_text(%s)" % jsquote(selector))
                      ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/work/make-checkout-workdir/test/common/testlib.py", line 859, in wait_js_cond
    raise Error(f"timeout\nwait_js_cond({cond}): {last_error.msg}") from None
testlib.Error: timeout
wait_js_cond(ph_text_is("[data-test-card-title='MDRAID device'] [data-test-desc-title='State'] [data-test-value=true]","Running")): Error: actual text: Not running

# Result testRaid (__main__.TestStorageMdRaid.testRaid) failed
# 1 TEST FAILED [130s on 584cdd74f233]
not ok 52 test/verify/check-storage-mdraid TestStorageMdRaid.testRaid [ND@1]

First occurrence: 2025-11-19T08:33:29.569422+00:00 | revision 95f0f822a6ef09ffa2a2c43599da5a4399a4f16e Times recorded: 1 Latest occurrences:

  • 2025-11-19T08:33:29.569422+00:00 | revision 95f0f822a6ef09ffa2a2c43599da5a4399a4f16e
rmmod: ERROR: Module scsi_debug is in use
rmmod: ERROR: Module scsi_debug is in use
rmmod: ERROR: Module scsi_debug is in use
rmmod: ERROR: Module scsi_debug is in use
rmmod: ERROR: Module scsi_debug is in use
rmmod: ERROR: Module scsi_debug is in use
rmmod: ERROR: Module scsi_debug is in use
rmmod: ERROR: Module scsi_debug is in use
rmmod: ERROR: Module scsi_debug is in use
rmmod: ERROR: Module scsi_debug is in use
rmmod: ERROR: Module scsi_debug is in use
rmmod: ERROR: Module scsi_debug is in use
rmmod: ERROR: Module scsi_debug is in use
Traceback (most recent call last):
  File "/work/make-checkout-workdir/test/verify/check-storage-mdraid", line 170, in testRaid
    b.wait_text(self.card_desc("MDRAID device", "State"), "Running")
    ~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/work/make-checkout-workdir/test/common/testlib.py", line 921, in wait_text
    self.wait_js_cond("ph_text_is(%s,%s)" % (jsquote(selector), jsquote(text)),
    ~~~~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
                      error_description="() => 'actual text: ' + ph_text(%s)" % jsquote(selector))
                      ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/work/make-checkout-workdir/test/common/testlib.py", line 859, in wait_js_cond
    raise Error(f"timeout\nwait_js_cond({cond}): {last_error.msg}") from None
testlib.Error: timeout
wait_js_cond(ph_text_is("[data-test-card-title='MDRAID device'] [data-test-desc-title='State'] [data-test-value=true]","Running")): Error: actual text: Not running

# Result testRaid (__main__.TestStorageMdRaid.testRaid) failed
# 1 TEST FAILED [136s on 892698ff9d28]
not ok 52 test/verify/check-storage-mdraid TestStorageMdRaid.testRaid [ND@0]

First occurrence: 2025-11-23T04:14:34.164800+00:00 | revision 35634d4cfa958cbe9300a9c72833902a0d1394da Times recorded: 1 Latest occurrences:

  • 2025-11-23T04:14:34.164800+00:00 | revision 35634d4cfa958cbe9300a9c72833902a0d1394da

cockpituous avatar Nov 14 '25 23:11 cockpituous