qubes-core-admin icon indicating copy to clipboard operation
qubes-core-admin copied to clipboard

Published 20 hours ago • verified publisher icon QubesOS

Reproduce bug in qrexec tests

Open DemiMarie opened this issue 9 months ago • 3 comments

The qrexec tests are broken and fail intermittently. This patch makes the bug obvious: by adding a sleep, it causes the test to fail right away, reliably.

DemiMarie avatar Feb 22 '25 07:02 DemiMarie

Codecov Report

All modified and coverable lines are covered by tests :white_check_mark:

Project coverage is 69.72%. Comparing base (5de215d) to head (87f187d). Report is 51 commits behind head on main.

Additional details and impacted files 
@@            Coverage Diff             @@
##             main     #661      +/-   ##
==========================================
- Coverage   69.79%   69.72%   -0.07%     
==========================================
  Files          58       58              
  Lines       12546    12546              
==========================================
- Hits         8756     8748       -8     
- Misses       3790     3798       +8
Flag Coverage Δ
unittests 69.72% <ø> (-0.07%) :arrow_down:

Flags with carried forward coverage won't be shown. Click here to find out more.

:umbrella: View full report in Codecov by Sentry.
:loudspeaker: Have feedback on the report? Share it here.

:rocket: New features to boost your workflow:
  • :snowflake: Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

codecov[bot] avatar Feb 22 '25 07:02 codecov[bot]

OpenQA test summary

Complete test suite and dependencies: https://openqa.qubes-os.org/tests/overview?distri=qubesos&version=4.3&build=2025041215-4.3&flavor=pull-requests

Test run included the following:

  • https://github.com/QubesOS/qubes-manager/pull/419 (https://github.com/QubesOS/qubes-manager/pull/419/commits/9cd7bec0d2cf3db93696a8296b4e534da0b01a4f)
  • https://github.com/QubesOS/qubes-core-admin-client/pull/332 (https://github.com/QubesOS/qubes-core-admin-client/pull/332/commits/e22c531c6eb2e0b5404b58295e499ca758627009)
  • https://github.com/QubesOS/qubes-core-admin/pull/672 (https://github.com/QubesOS/qubes-core-admin/pull/672/commits/b8ee4518ba7941afaf8056cd582cd81e60d6dbfa)
  • https://github.com/QubesOS/qubes-core-admin/pull/667 (https://github.com/QubesOS/qubes-core-admin/pull/667/commits/69cea577bab1e31a666424f31310ba14a0c45f76)
  • https://github.com/QubesOS/qubes-core-admin/pull/661 (https://github.com/QubesOS/qubes-core-admin/pull/661/commits/87f187dafb4de4b88f4d27b9e4779567ca3fa043)
  • https://github.com/QubesOS/qubes-core-admin-linux/pull/189 (https://github.com/QubesOS/qubes-core-admin-linux/pull/189/commits/ab927ec4f0642e967e1ee59f160469e283b2b402)
  • https://github.com/QubesOS/qubes-vmm-xen/pull/205 (https://github.com/QubesOS/qubes-vmm-xen/pull/205/commits/4426d1da870f493606542004d3d9249f7d822763)
  • https://github.com/QubesOS/qubes-core-admin-client/pull/347 (https://github.com/QubesOS/qubes-core-admin-client/pull/347/commits/6dd1258a59c8f92651e9d27f1e15c098f0d51951)
  • https://github.com/QubesOS/qubes-core-admin/pull/674 (https://github.com/QubesOS/qubes-core-admin/pull/674/commits/0918bc0b890374cdd8c9a9f5df4bb50a034444ff)
  • https://github.com/QubesOS/qubes-desktop-linux-manager/pull/249 (https://github.com/QubesOS/qubes-desktop-linux-manager/pull/249/commits/203be8c1079512cafca4786eba0063874a2b5c00)
  • https://github.com/QubesOS/qubes-core-admin-linux/pull/190 (https://github.com/QubesOS/qubes-core-admin-linux/pull/190/commits/7806f5714558040e80d7b825806d7e8e83e54990)
  • https://github.com/QubesOS/qubes-core-qrexec/pull/196 (https://github.com/QubesOS/qubes-core-qrexec/pull/196/commits/e5a5b0ae61a85482c668d28b2574740a9c829b43)
  • https://github.com/QubesOS/qubes-core-admin/pull/670 (https://github.com/QubesOS/qubes-core-admin/pull/670/commits/20aa5490396bda016ec8e63581a97d8d38860414)

New failures, excluding unstable

Compared to: https://openqa.qubes-os.org/tests/overview?distri=qubesos&version=4.3&build=2025031804-4.3&flavor=update

  • system_tests_qwt_win11@hw13

    • windows_install: wait_serial (wait serial expected) # wait_serial expected: qr/dcWzE-\d+-/...

    • windows_install: Failed (test died + timed out) # Test died: command 'script -e -c 'bash -x /usr/bin/qvm-create-win...

  • system_tests_suspend@hw1

    • suspend: unnamed test (unknown)
    • suspend: Failed (test died) # Test died: no candidate needle with tag(s) 'SUSPEND-FAILED' match...

  • system_tests_whonix

    • whonixcheck: unnamed test (unknown)

    • whonixcheck: wait_serial (wait serial expected) # wait_serial expected: qr/TBvsr-\d+-/...

    • whonixcheck: Failed (test died + timed out) # Test died: command 'qvm-run -ap whonix-gateway-17 'LC_ALL=C whon...

  • system_tests_gui_tools

    • qui_widgets_update: unnamed test (unknown)
    • qui_widgets_update: Failed (test died) # Test died: no candidate needle with tag(s) 'qui-updates-widget-op...

  • system_tests_suspend

    • suspend: wait_serial (wait serial expected) # wait_serial expected: qr/wG5VX-\d+-/...

  • system_tests_extra

    • TC_00_QVCTest_whonix-workstation-17: test_010_screenshare (failure + cleanup) ~~~~~~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^... AssertionError: 0 == 0

  • system_tests_qrexec

  • system_tests_kde_gui_interactive

    • gui_keyboard_layout: wait_serial (wait serial expected) # wait_serial expected: "echo -e '[Layout]\nLayoutList=us,de' | sud...

    • gui_keyboard_layout: Failed (test died) # Test died: command 'test "$(cd ~user;ls e1*)" = "$(qvm-run -p wor...

  • system_tests_pvgrub_salt_storage

    • TC_10_VMSalt_fedora-41-xfce: test_003_update (failure) AssertionError: Salt command '['qubesctl', '--skip-dom0', '--show-o...

  • system_tests_audio

  • system_tests_guivm_vnc_gui_interactive

    • guivm_manager: unnamed test (unknown)
    • guivm_manager: Failed (test died) # Test died: no candidate needle with tag(s) 'menu-qubes-tools-subm...

  • system_tests_qwt_win10@hw13

    • windows_install: wait_serial (wait serial expected) # wait_serial expected: qr/LDWI1-\d+-/...

    • windows_install: Failed (test died + timed out) # Test died: command 'script -e -c 'bash -x /usr/bin/qvm-create-win...

  • system_tests_qwt_win10_seamless@hw13

    • windows_clipboard_and_filecopy: unnamed test (unknown)
    • windows_clipboard_and_filecopy: Failed (test died) # Test died: no candidate needle with tag(s) 'windows-Explorer-empt...

Failed tests

37 failures

  • system_tests_whonix@hw7

    • whonixcheck: fail (unknown) Whonixcheck for sys-whonix failed...

    • whonixcheck: unnamed test (unknown)

  • system_tests_qwt_win11@hw13

    • windows_install: wait_serial (wait serial expected) # wait_serial expected: qr/dcWzE-\d+-/...

    • windows_install: Failed (test died + timed out) # Test died: command 'script -e -c 'bash -x /usr/bin/qvm-create-win...

  • system_tests_suspend@hw1

    • suspend: unnamed test (unknown)
    • suspend: Failed (test died) # Test died: no candidate needle with tag(s) 'SUSPEND-FAILED' match...

  • system_tests_whonix

    • whonixcheck: fail (unknown) Whonixcheck for anon-whonix failed...

    • whonixcheck: unnamed test (unknown)

    • whonixcheck: wait_serial (wait serial expected) # wait_serial expected: qr/TBvsr-\d+-/...

    • whonixcheck: Failed (test died + timed out) # Test died: command 'qvm-run -ap whonix-gateway-17 'LC_ALL=C whon...

  • system_tests_gui_tools

    • qui_widgets_update: unnamed test (unknown)
    • qui_widgets_update: Failed (test died) # Test died: no candidate needle with tag(s) 'qui-updates-widget-op...

  • system_tests_suspend

    • suspend: wait_serial (wait serial expected) # wait_serial expected: qr/wG5VX-\d+-/...

    • suspend: Failed (test died + timed out) # Test died: command 'qvm-run -ap sys-whonix 'LC_ALL=C whonixcheck ...

  • system_tests_extra

    • TC_00_QVCTest_whonix-workstation-17: test_010_screenshare (failure + cleanup) ~~~~~~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^... AssertionError: 0 == 0

  • system_tests_qrexec

  • system_tests_kde_gui_interactive

    • gui_keyboard_layout: wait_serial (wait serial expected) # wait_serial expected: "echo -e '[Layout]\nLayoutList=us,de' | sud...

    • gui_keyboard_layout: Failed (test died) # Test died: command 'test "$(cd ~user;ls e1*)" = "$(qvm-run -p wor...

  • system_tests_pvgrub_salt_storage

    • TC_10_VMSalt_fedora-41-xfce: test_003_update (failure) AssertionError: Salt command '['qubesctl', '--skip-dom0', '--show-o...

  • system_tests_audio

  • system_tests_guivm_vnc_gui_interactive

    • guivm_manager: unnamed test (unknown)
    • guivm_manager: Failed (test died) # Test died: no candidate needle with tag(s) 'menu-qubes-tools-subm...

  • system_tests_qwt_win10@hw13

    • windows_install: wait_serial (wait serial expected) # wait_serial expected: qr/LDWI1-\d+-/...

    • windows_install: Failed (test died + timed out) # Test died: command 'script -e -c 'bash -x /usr/bin/qvm-create-win...

  • system_tests_qwt_win10_seamless@hw13

    • windows_clipboard_and_filecopy: unnamed test (unknown)
    • windows_clipboard_and_filecopy: Failed (test died) # Test died: no candidate needle with tag(s) 'windows-Explorer-empt...

Fixed failures

Compared to: https://openqa.qubes-os.org/tests/132953#dependencies

9 fixed

  • system_tests_whonix

  • system_tests_suspend

  • system_tests_basic_vm_qrexec_gui

  • system_tests_kde_gui_interactive

    • clipboard_and_web: unnamed test (unknown)

    • clipboard_and_web: Failed (test died) # Test died: no candidate needle with tag(s) 'qubes-website' matche...

    • clipboard_and_web: wait_serial (wait serial expected) # wait_serial expected: "lspci; echo 2E8vz-\$?-"...

  • system_tests_audio

  • system_tests_guivm_vnc_gui_interactive

    • gui_filecopy: unnamed test (unknown)
    • gui_filecopy: Failed (test died) # Test died: no candidate needle with tag(s) 'files-work' matched...

Unstable tests

Performance Tests

Performance degradation:

16 performance degradations
  • debian-12-xfce_exec: 8.37 :small_red_triangle_up: ( previous job: 7.12, degradation: 117.52%)
  • whonix-gateway-17_exec: 8.90 :small_red_triangle_up: ( previous job: 6.82, degradation: 130.50%)
  • whonix-gateway-17_socket-root: 8.50 :small_red_triangle_up: ( previous job: 7.65, degradation: 111.16%)
  • whonix-workstation-17_socket-root: 9.04 :small_red_triangle_up: ( previous job: 8.13, degradation: 111.15%)
  • dom0_root_seq1m_q8t1_read 3:read_bandwidth_kb: 116150.00 :small_red_triangle_up: ( previous job: 446963.00, degradation: 25.99%)
  • dom0_root_seq1m_q1t1_read 3:read_bandwidth_kb: 31030.00 :small_red_triangle_up: ( previous job: 294295.00, degradation: 10.54%)
  • dom0_root_rnd4k_q32t1_read 3:read_bandwidth_kb: 24355.00 :small_red_triangle_up: ( previous job: 79803.00, degradation: 30.52%)
  • dom0_root_rnd4k_q32t1_write 3:write_bandwidth_kb: 950.00 :small_red_triangle_up: ( previous job: 6149.00, degradation: 15.45%)
  • dom0_root_rnd4k_q1t1_write 3:write_bandwidth_kb: 2198.00 :small_red_triangle_up: ( previous job: 4826.00, degradation: 45.54%)
  • dom0_varlibqubes_seq1m_q8t1_write 3:write_bandwidth_kb: 149908.00 :small_red_triangle_up: ( previous job: 250795.00, degradation: 59.77%)
  • dom0_varlibqubes_rnd4k_q32t1_write 3:write_bandwidth_kb: 5338.00 :small_red_triangle_up: ( previous job: 6479.00, degradation: 82.39%)
  • dom0_varlibqubes_rnd4k_q1t1_write 3:write_bandwidth_kb: 3556.00 :small_red_triangle_up: ( previous job: 4903.00, degradation: 72.53%)
  • fedora-41-xfce_root_rnd4k_q32t1_write 3:write_bandwidth_kb: 2140.00 :small_red_triangle_up: ( previous job: 3599.00, degradation: 59.46%)
  • fedora-41-xfce_private_seq1m_q1t1_read 3:read_bandwidth_kb: 275000.00 :small_red_triangle_up: ( previous job: 334687.00, degradation: 82.17%)
  • fedora-41-xfce_volatile_rnd4k_q32t1_write 3:write_bandwidth_kb: 1888.00 :small_red_triangle_up: ( previous job: 5672.00, degradation: 33.29%)
  • fedora-41-xfce_volatile_rnd4k_q1t1_write 3:write_bandwidth_kb: 1478.00 :small_red_triangle_up: ( previous job: 1953.00, degradation: 75.68%)

Remaining performance tests:

56 tests
  • debian-12-xfce_exec-root: 28.79 :small_red_triangle_up: ( previous job: 28.65, degradation: 100.45%)
  • debian-12-xfce_socket: 8.23 :green_circle: ( previous job: 8.60, improvement: 95.65%)
  • debian-12-xfce_socket-root: 8.92 :small_red_triangle_up: ( previous job: 8.52, degradation: 104.63%)
  • debian-12-xfce_exec-data-simplex: 69.28 :green_circle: ( previous job: 71.62, improvement: 96.72%)
  • debian-12-xfce_exec-data-duplex: 68.10 :green_circle: ( previous job: 70.34, improvement: 96.81%)
  • debian-12-xfce_exec-data-duplex-root: 86.76 :small_red_triangle_up: ( previous job: 82.72, degradation: 104.88%)
  • debian-12-xfce_socket-data-duplex: 163.59 :small_red_triangle_up: ( previous job: 156.96, degradation: 104.23%)
  • fedora-41-xfce_exec: 9.33 :small_red_triangle_up: ( previous job: 9.27, degradation: 100.66%)
  • fedora-41-xfce_exec-root: 63.23 :small_red_triangle_up: ( previous job: 61.51, degradation: 102.80%)
  • fedora-41-xfce_socket: 8.33 :green_circle: ( previous job: 8.63, improvement: 96.50%)
  • fedora-41-xfce_socket-root: 9.00 :small_red_triangle_up: ( previous job: 8.71, degradation: 103.34%)
  • fedora-41-xfce_exec-data-simplex: 64.75 :green_circle: ( previous job: 75.53, improvement: 85.73%)
  • fedora-41-xfce_exec-data-duplex: 66.18 :green_circle: ( previous job: 71.56, improvement: 92.48%)
  • fedora-41-xfce_exec-data-duplex-root: 105.60 :green_circle: ( previous job: 109.13, improvement: 96.77%)
  • fedora-41-xfce_socket-data-duplex: 149.04 :green_circle: ( previous job: 150.61, improvement: 98.96%)
  • whonix-gateway-17_exec-root: 40.34 :green_circle: ( previous job: 40.43, improvement: 99.76%)
  • whonix-gateway-17_socket: 7.41 :small_red_triangle_up: ( previous job: 7.24, degradation: 102.40%)
  • whonix-gateway-17_exec-data-simplex: 70.65 :green_circle: ( previous job: 78.32, improvement: 90.22%)
  • whonix-gateway-17_exec-data-duplex: 75.15 :green_circle: ( previous job: 76.65, improvement: 98.04%)
  • whonix-gateway-17_exec-data-duplex-root: 92.29 :small_red_triangle_up: ( previous job: 88.52, degradation: 104.26%)
  • whonix-gateway-17_socket-data-duplex: 170.79 :green_circle: ( previous job: 171.76, improvement: 99.43%)
  • whonix-workstation-17_exec: 8.03 :small_red_triangle_up: ( previous job: 7.67, degradation: 104.72%)
  • whonix-workstation-17_exec-root: 62.43 :small_red_triangle_up: ( previous job: 58.26, degradation: 107.15%)
  • whonix-workstation-17_socket: 8.50 :small_red_triangle_up: ( previous job: 8.19, degradation: 103.77%)
  • whonix-workstation-17_exec-data-simplex: 76.51 :small_red_triangle_up: ( previous job: 74.99, degradation: 102.02%)
  • whonix-workstation-17_exec-data-duplex: 64.36 :green_circle: ( previous job: 72.71, improvement: 88.52%)
  • whonix-workstation-17_exec-data-duplex-root: 102.71 :small_red_triangle_up: ( previous job: 99.82, degradation: 102.89%)
  • whonix-workstation-17_socket-data-duplex: 165.93 :green_circle: ( previous job: 169.50, improvement: 97.89%)
  • dom0_root_seq1m_q8t1_write 3:write_bandwidth_kb: 168170.00 :green_circle: ( previous job: 129298.00, improvement: 130.06%)
  • dom0_root_seq1m_q1t1_write 3:write_bandwidth_kb: 104179.00 :green_circle: ( previous job: 95454.00, improvement: 109.14%)
  • dom0_root_rnd4k_q1t1_read 3:read_bandwidth_kb: 11162.00 :green_circle: ( previous job: 10795.00, improvement: 103.40%)
  • dom0_varlibqubes_seq1m_q8t1_read 3:read_bandwidth_kb: 466033.00 :green_circle: ( previous job: 382273.00, improvement: 121.91%)
  • dom0_varlibqubes_seq1m_q1t1_read 3:read_bandwidth_kb: 438734.00 :green_circle: ( previous job: 437636.00, improvement: 100.25%)
  • dom0_varlibqubes_seq1m_q1t1_write 3:write_bandwidth_kb: 180273.00 :small_red_triangle_up: ( previous job: 184752.00, degradation: 97.58%)
  • dom0_varlibqubes_rnd4k_q32t1_read 3:read_bandwidth_kb: 102966.00 :green_circle: ( previous job: 62195.00, improvement: 165.55%)
  • dom0_varlibqubes_rnd4k_q1t1_read 3:read_bandwidth_kb: 7688.00 :green_circle: ( previous job: 7669.00, improvement: 100.25%)
  • fedora-41-xfce_root_seq1m_q8t1_read 3:read_bandwidth_kb: 367277.00 :small_red_triangle_up: ( previous job: 368309.00, degradation: 99.72%)
  • fedora-41-xfce_root_seq1m_q8t1_write 3:write_bandwidth_kb: 280143.00 :green_circle: ( previous job: 162081.00, improvement: 172.84%)
  • fedora-41-xfce_root_seq1m_q1t1_read 3:read_bandwidth_kb: 323434.00 :green_circle: ( previous job: 318716.00, improvement: 101.48%)
  • fedora-41-xfce_root_seq1m_q1t1_write 3:write_bandwidth_kb: 111370.00 :green_circle: ( previous job: 87940.00, improvement: 126.64%)
  • fedora-41-xfce_root_rnd4k_q32t1_read 3:read_bandwidth_kb: 93009.00 :green_circle: ( previous job: 82694.00, improvement: 112.47%)
  • fedora-41-xfce_root_rnd4k_q1t1_read 3:read_bandwidth_kb: 7792.00 :small_red_triangle_up: ( previous job: 8485.00, degradation: 91.83%)
  • fedora-41-xfce_root_rnd4k_q1t1_write 3:write_bandwidth_kb: 906.00 :green_circle: ( previous job: 542.00, improvement: 167.16%)
  • fedora-41-xfce_private_seq1m_q8t1_read 3:read_bandwidth_kb: 393609.00 :green_circle: ( previous job: 373957.00, improvement: 105.26%)
  • fedora-41-xfce_private_seq1m_q8t1_write 3:write_bandwidth_kb: 174844.00 :green_circle: ( previous job: 170062.00, improvement: 102.81%)
  • fedora-41-xfce_private_seq1m_q1t1_write 3:write_bandwidth_kb: 89487.00 :green_circle: ( previous job: 61534.00, improvement: 145.43%)
  • fedora-41-xfce_private_rnd4k_q32t1_read 3:read_bandwidth_kb: 74301.00 :small_red_triangle_up: ( previous job: 80283.00, degradation: 92.55%)
  • fedora-41-xfce_private_rnd4k_q32t1_write 3:write_bandwidth_kb: 3220.00 :green_circle: ( previous job: 2215.00, improvement: 145.37%)
  • fedora-41-xfce_private_rnd4k_q1t1_read 3:read_bandwidth_kb: 8522.00 :green_circle: ( previous job: 7540.00, improvement: 113.02%)
  • fedora-41-xfce_private_rnd4k_q1t1_write 3:write_bandwidth_kb: 1452.00 :green_circle: ( previous job: 1130.00, improvement: 128.50%)
  • fedora-41-xfce_volatile_seq1m_q8t1_read 3:read_bandwidth_kb: 356416.00 :small_red_triangle_up: ( previous job: 369868.00, degradation: 96.36%)
  • fedora-41-xfce_volatile_seq1m_q8t1_write 3:write_bandwidth_kb: 230456.00 :green_circle: ( previous job: 179949.00, improvement: 128.07%)
  • fedora-41-xfce_volatile_seq1m_q1t1_read 3:read_bandwidth_kb: 356173.00 :green_circle: ( previous job: 324737.00, improvement: 109.68%)
  • fedora-41-xfce_volatile_seq1m_q1t1_write 3:write_bandwidth_kb: 46638.00 :green_circle: ( previous job: 17567.00, improvement: 265.49%)
  • fedora-41-xfce_volatile_rnd4k_q32t1_read 3:read_bandwidth_kb: 75436.00 :small_red_triangle_up: ( previous job: 79021.00, degradation: 95.46%)
  • fedora-41-xfce_volatile_rnd4k_q1t1_read 3:read_bandwidth_kb: 8299.00 :green_circle: ( previous job: 7867.00, improvement: 105.49%)

qubesos-bot avatar Feb 23 '25 12:02 qubesos-bot

I don't see any failure with this PR (but not https://github.com/QubesOS/qubes-core-qrexec/pull/186): https://openqa.qubes-os.org/tests/129865

marmarek avatar Feb 23 '25 16:02 marmarek

I finally got the correct stderr logged (https://github.com/QubesOS/qubes-core-admin/pull/670, https://github.com/QubesOS/qubes-core-qrexec/pull/196/files) and got this:

2025-04-12 18:47:10.164 qrexec-fork-server[1073]: exec.c:165:exec_qubes_rpc2: execve: Exec format error

It's a progress I guess...

marmarek avatar Apr 13 '25 00:04 marmarek

Ok, this is clearly bug in tests - they create rpc services without shebang, so it works only when called via shell - which not always is the case.

marmarek avatar Apr 13 '25 01:04 marmarek

Fix: https://github.com/QubesOS/qubes-core-admin/pull/675

marmarek avatar Apr 13 '25 01:04 marmarek