qubes-issues icon indicating copy to clipboard operation
qubes-issues copied to clipboard

[Build 2024111416-4.2] openQA test fails in upload_packages

Open marmarek opened this issue 1 year ago • 5 comments

Observation

openQA test in scenario qubesos-4.2-securedrop-x86_64-securedrop_install@64bit fails in upload_packages

curl: (28) Operation timed out after 10003 miliseconds with 0 bytes received

Test suite description

Install SecureDrop

Reproducible

Fails since (at least) Build 2024111416-4.2 (current job)

Expected result

Last good: 2024111309-4.2 (or more recent)

Further details

Always latest result in this scenario: latest

See more info at https://github.com/QubesOS/openqa-tests-qubesos/pull/25#issuecomment-2476280844

marmarek avatar Nov 14 '24 20:11 marmarek

Adding some context captured on an old commit that won't make it into main:

In a specific set of circumstances dropped packets were causing the log uploads to take a long time and eventually fail. Even though the client would send retries, they would not reach the host.

A failok flag was added to the logs-uploading method, as a a temporary measure to let the tests proceed even though the log uploading failed.

deeplow avatar Nov 20 '24 10:11 deeplow

@deeplow here is the issue I mentioned - if you hit such issue, add the issue URL as the job comment

marmarek avatar Sep 02 '25 21:09 marmarek

Recent related failures:

?column? t_finished state result test host
https://openqa.qubes-os.org/tests/159898 2025-11-19 10:08:45 done failed system_tests_basic_vm_qrexec_gui_xfs ramen
https://openqa.qubes-os.org/tests/159885 2025-11-19 07:25:53 done failed system_tests_basic_vm_qrexec_gui_xfs ramen
https://openqa.qubes-os.org/tests/159851 2025-11-19 06:24:35 done failed system_tests_guivm_gui_interactive herring
https://openqa.qubes-os.org/tests/159763 2025-11-18 13:25:23 done failed system_tests_basic_vm_qrexec_gui_ext4 herring
https://openqa.qubes-os.org/tests/159739 2025-11-18 12:51:48 done failed system_tests_basic_vm_qrexec_gui herring
https://openqa.qubes-os.org/tests/159757 2025-11-18 12:31:32 done failed system_tests_guivm_vnc_gui_interactive onigiri
https://openqa.qubes-os.org/tests/159670 2025-11-18 07:14:30 done failed system_tests_qrexec herring
https://openqa.qubes-os.org/tests/159681 2025-11-18 05:59:27 done failed system_tests_update salmon.qubes-os.org
https://openqa.qubes-os.org/tests/159532 2025-11-16 04:43:17 done failed system_tests_network_ipv6 herring
https://openqa.qubes-os.org/tests/159561 2025-11-16 04:12:40 done failed system_tests_basic_vm_qrexec_gui_btrfs salmon.qubes-os.org
https://openqa.qubes-os.org/tests/159562 2025-11-16 03:06:23 done failed system_tests_basic_vm_qrexec_gui_ext4 ramen
https://openqa.qubes-os.org/tests/159469 2025-11-14 17:16:55 done failed system_tests_basic_vm_qrexec_gui_xfs onigiri
https://openqa.qubes-os.org/tests/159468 2025-11-14 17:14:38 done failed system_tests_basic_vm_qrexec_gui_ext4 ramen
https://openqa.qubes-os.org/tests/159467 2025-11-14 17:11:09 done failed system_tests_basic_vm_qrexec_gui_btrfs ramen
https://openqa.qubes-os.org/tests/159350 2025-11-13 05:48:57 done failed system_tests_basic_vm_qrexec_gui_ext4 herring
https://openqa.qubes-os.org/tests/159160 2025-11-11 16:02:53 done failed system_tests_basic_vm_qrexec_gui_ext4 ramen
https://openqa.qubes-os.org/tests/159153 2025-11-11 13:46:03 done failed system_tests_basic_vm_qrexec_gui_ext4 herring
https://openqa.qubes-os.org/tests/159140 2025-11-11 12:15:19 done failed system_tests_basic_vm_qrexec_gui_btrfs herring
https://openqa.qubes-os.org/tests/159141 2025-11-11 12:05:07 done failed system_tests_basic_vm_qrexec_gui_ext4 ramen
https://openqa.qubes-os.org/tests/159152 2025-11-11 11:46:07 done failed system_tests_guivm_vnc_gui_interactive salmon.qubes-os.org
https://openqa.qubes-os.org/tests/159130 2025-11-11 11:45:36 done failed system_tests_backupdispvm onigiri
https://openqa.qubes-os.org/tests/159021 2025-11-11 06:08:12 done failed system_tests_basic_vm_qrexec_gui_ext4 herring
https://openqa.qubes-os.org/tests/159060 2025-11-11 06:07:31 done failed system_tests_update salmon.qubes-os.org
https://openqa.qubes-os.org/tests/158845 2025-11-10 14:32:03 done failed system_tests_network herring
https://openqa.qubes-os.org/tests/158811 2025-11-10 11:57:38 done failed system_tests_update salmon.qubes-os.org
https://openqa.qubes-os.org/tests/158796 2025-11-09 13:25:30 done failed system_tests_splitgpg ramen
https://openqa.qubes-os.org/tests/158674 2025-11-05 16:59:43 done failed system_tests_dispvm herring
https://openqa.qubes-os.org/tests/158605 2025-11-05 05:35:32 done failed system_tests_dispvm herring
https://openqa.qubes-os.org/tests/158632 2025-11-05 04:42:32 done failed system_tests_basic_vm_qrexec_gui_xfs ramen
https://openqa.qubes-os.org/tests/158494 2025-11-04 05:29:51 done failed system_tests_whonix ramen
https://openqa.qubes-os.org/tests/158489 2025-11-03 22:28:56 done failed system_tests_dispvm onigiri
https://openqa.qubes-os.org/tests/158490 2025-11-03 22:15:46 done failed system_tests_basic_vm_qrexec_gui_zfs herring
https://openqa.qubes-os.org/tests/158456 2025-11-03 21:24:03 done failed system_tests_dispvm herring
https://openqa.qubes-os.org/tests/158483 2025-11-03 20:28:16 done failed system_tests_basic_vm_qrexec_gui_xfs onigiri
https://openqa.qubes-os.org/tests/158481 2025-11-03 20:23:46 done failed system_tests_basic_vm_qrexec_gui_btrfs ramen
https://openqa.qubes-os.org/tests/158443 2025-11-03 20:11:58 done failed system_tests_network ramen
https://openqa.qubes-os.org/tests/158004 2025-10-30 07:04:33 done passed system_tests_gui_interactive_preloaded onigiri
https://openqa.qubes-os.org/tests/157976 2025-10-30 06:59:25 done passed system_tests_gui_interactive ramen
https://openqa.qubes-os.org/tests/157787 2025-10-28 12:09:56 done failed system_tests_extra ramen
https://openqa.qubes-os.org/tests/157762 2025-10-28 07:08:39 done failed system_tests_extra ramen
https://openqa.qubes-os.org/tests/157755 2025-10-28 06:20:42 done failed system_tests_basic_vm_qrexec_gui ramen
https://openqa.qubes-os.org/tests/157757 2025-10-28 06:04:18 done failed system_tests_basic_vm_qrexec_gui_ext4 ramen
https://openqa.qubes-os.org/tests/157756 2025-10-28 05:58:34 done failed system_tests_basic_vm_qrexec_gui_btrfs ramen
https://openqa.qubes-os.org/tests/157737 2025-10-28 05:26:09 done failed system_tests_whonix onigiri
https://openqa.qubes-os.org/tests/157685 2025-10-27 08:05:05 done failed system_tests_basic_vm_qrexec_gui_zfs ramen
https://openqa.qubes-os.org/tests/157674 2025-10-27 06:01:33 done failed system_tests_guivm_gui_interactive ramen
https://openqa.qubes-os.org/tests/157379 2025-10-24 09:57:06 done failed system_tests_basic_vm_qrexec_gui salmon.qubes-os.org
https://openqa.qubes-os.org/tests/157308 2025-10-23 15:16:53 done failed system_tests_network_updates salmon.qubes-os.org
https://openqa.qubes-os.org/tests/157195 2025-10-22 20:53:59 done failed system_tests_guivm_gui_interactive ramen
https://openqa.qubes-os.org/tests/156736 2025-10-21 08:37:34 done failed system_tests_network_updates salmon.qubes-os.org
https://openqa.qubes-os.org/tests/156730 2025-10-21 08:25:51 done failed system_tests_extra salmon.qubes-os.org
https://openqa.qubes-os.org/tests/156734 2025-10-21 07:07:23 done failed system_tests_network onigiri
https://openqa.qubes-os.org/tests/156725 2025-10-21 06:36:55 done failed system_tests_basic_vm_qrexec_gui_ext4 ramen
https://openqa.qubes-os.org/tests/156427 2025-10-16 22:17:16 done failed system_tests_network ramen
https://openqa.qubes-os.org/tests/156227 2025-10-14 19:27:09 done failed system_tests_basic_vm_qrexec_gui_zfs ramen
https://openqa.qubes-os.org/tests/156228 2025-10-14 17:32:03 done failed system_tests_guivm_vnc_gui_interactive onigiri
https://openqa.qubes-os.org/tests/156170 2025-10-14 07:40:39 done failed system_tests_network_updates ramen
https://openqa.qubes-os.org/tests/156157 2025-10-14 06:31:44 done failed system_tests_basic_vm_qrexec_gui ramen
https://openqa.qubes-os.org/tests/155873 2025-10-11 10:19:25 done failed system_tests_network ramen
https://openqa.qubes-os.org/tests/155571 2025-10-07 17:34:21 done failed system_tests_guivm_vnc_gui_interactive ramen
https://openqa.qubes-os.org/tests/155519 2025-10-07 09:06:53 done failed system_tests_guivm_vnc_gui_interactive onigiri
https://openqa.qubes-os.org/tests/155192 2025-10-05 01:24:30 done failed system_tests_splitgpg salmon.qubes-os.org
https://openqa.qubes-os.org/tests/155193 2025-10-05 01:14:46 done failed system_tests_extra onigiri
https://openqa.qubes-os.org/tests/154818 2025-10-02 20:41:32 done failed system_tests_basic_vm_qrexec_gui_ext4 onigiri
https://openqa.qubes-os.org/tests/154792 2025-10-02 20:17:36 done failed system_tests_dispvm onigiri
https://openqa.qubes-os.org/tests/154503 2025-09-30 06:48:46 done failed system_tests_basic_vm_qrexec_gui_ext4 onigiri
https://openqa.qubes-os.org/tests/154175 2025-09-23 06:31:14 done failed system_tests_basic_vm_qrexec_gui_btrfs ramen
https://openqa.qubes-os.org/tests/154176 2025-09-23 06:10:21 done failed system_tests_basic_vm_qrexec_gui_ext4 ramen
https://openqa.qubes-os.org/tests/153865 2025-09-19 01:59:23 done failed system_tests_basic_vm_qrexec_gui_xfs onigiri
https://openqa.qubes-os.org/tests/153729 2025-09-18 08:00:15 done failed system_tests_basic_vm_qrexec_gui_btrfs onigiri
https://openqa.qubes-os.org/tests/153696 2025-09-18 07:34:39 done failed system_tests_extra onigiri
https://openqa.qubes-os.org/tests/153589 2025-09-17 21:53:16 done failed system_tests_basic_vm_qrexec_gui_ext4 onigiri
https://openqa.qubes-os.org/tests/153581 2025-09-17 19:10:07 done failed system_tests_basic_vm_qrexec_gui_ext4 ramen
https://openqa.qubes-os.org/tests/153555 2025-09-17 18:31:17 done failed system_tests_dispvm ramen
https://openqa.qubes-os.org/tests/153547 2025-09-17 18:18:36 done failed system_tests_extra ramen
https://openqa.qubes-os.org/tests/153362 2025-09-16 07:51:41 done failed system_tests_network_updates salmon.qubes-os.org
https://openqa.qubes-os.org/tests/153364 2025-09-16 07:17:08 done failed system_tests_qrexec ramen
https://openqa.qubes-os.org/tests/153351 2025-09-16 06:39:40 done failed system_tests_basic_vm_qrexec_gui_ext4 onigiri
https://openqa.qubes-os.org/tests/153350 2025-09-16 06:23:58 done failed system_tests_basic_vm_qrexec_gui_btrfs salmon.qubes-os.org
https://openqa.qubes-os.org/tests/153340 2025-09-16 05:35:07 done failed system_tests_guivm_vnc_gui_interactive ramen
https://openqa.qubes-os.org/tests/153286 2025-09-16 03:54:16 done failed system_tests_network ramen
https://openqa.qubes-os.org/tests/153266 2025-09-15 19:16:16 done failed system_tests_basic_vm_qrexec_gui_xfs onigiri
https://openqa.qubes-os.org/tests/153236 2025-09-15 18:27:09 done failed system_tests_qrexec ramen
https://openqa.qubes-os.org/tests/152831 2025-09-12 08:28:13 done failed system_tests_network_updates salmon.qubes-os.org
https://openqa.qubes-os.org/tests/152833 2025-09-12 07:45:15 done failed system_tests_dispvm ramen
https://openqa.qubes-os.org/tests/152828 2025-09-12 07:10:09 done failed system_tests_usbproxy ramen
https://openqa.qubes-os.org/tests/152238 2025-09-07 02:52:05 done failed system_tests_basic_vm_qrexec_gui_ext4 salmon.qubes-os.org
https://openqa.qubes-os.org/tests/152212 2025-09-07 01:35:18 done failed system_tests_backupdispvm salmon.qubes-os.org
https://openqa.qubes-os.org/tests/152218 2025-09-07 01:28:12 done failed system_tests_basic_vm_qrexec_gui_zfs onigiri
https://openqa.qubes-os.org/tests/152199 2025-09-07 00:15:42 done failed system_tests_basic_vm_qrexec_gui salmon.qubes-os.org
https://openqa.qubes-os.org/tests/152150 2025-09-06 04:51:24 done failed system_tests_basic_vm_qrexec_gui_ext4 onigiri
https://openqa.qubes-os.org/tests/152130 2025-09-06 04:35:24 done failed system_tests_basic_vm_qrexec_gui_zfs ramen
https://openqa.qubes-os.org/tests/152111 2025-09-06 03:38:04 done failed system_tests_basic_vm_qrexec_gui ramen

marmarek avatar Nov 19 '25 18:11 marmarek

In a very limited testing, it appears as ethtool -K ens6 sg off in sys-net helps.

marmarek avatar Dec 07 '25 23:12 marmarek

The ethtool change helps quite a lot, but the issue still happens, especially in collect_logs stage at the end of the the test job.

marmarek avatar Dec 12 '25 19:12 marmarek