qubes-core-agent-linux icon indicating copy to clipboard operation
qubes-core-agent-linux copied to clipboard

Published 20 hours ago • verified publisher icon QubesOS

Add support for IPv6 Virtual DNS (#462)

Open 3nprob opened this issue 4 months ago • 20 comments

  • #462 resurrected
  • Fix merge conflicts
  • Address outstanding review feedback
  • Unify logic for generating firewall rules for ipv4/ipv6

Fixes https://github.com/QubesOS/qubes-issues/issues/10038

3nprob avatar Jul 05 '25 01:07 3nprob

Codecov Report

All modified and coverable lines are covered by tests :white_check_mark:

Project coverage is 71.10%. Comparing base (4c5c36e) to head (c0c7d42).

Additional details and impacted files 
@@           Coverage Diff           @@
##             main     #592   +/-   ##
=======================================
  Coverage   71.10%   71.10%           
=======================================
  Files           3        3           
  Lines         481      481           
=======================================
  Hits          342      342           
  Misses        139      139

:umbrella: View full report in Codecov by Sentry.
:loudspeaker: Have feedback on the report? Share it here.

:rocket: New features to boost your workflow:
  • :snowflake: Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

codecov[bot] avatar Jul 05 '25 03:07 codecov[bot]

Broke out some tangential smaller changes that can be merged independently in order to reduce PR size and keeping diff scoped:

  • #593
  • #594
  • #595

3nprob avatar Jul 05 '25 06:07 3nprob

Now that the smaller PRs are in (the last one is still going through CI), this will need a rebase (dropping those already merged commits) to resolve conflicts.

marmarek avatar Jul 06 '25 03:07 marmarek

Now that the smaller PRs are in (the last one is still going through CI), this will need a rebase (dropping those already merged commits) to resolve conflicts.

Rebased. Ended up squashing all the existing commits on this branch. (lmk if you prefer to retain history during review in situations like this)

3nprob avatar Jul 06 '25 03:07 3nprob

openQA run is still in progress, but I already see some failures: https://openqa.qubes-os.org/tests/145922#step/clipboard_and_web/19

[2025-07-07 07:09:42] [   11.559294] resolvectl[699]: [0;1;31mFailed to parse DNS server address: [0m
[2025-07-07 07:09:42] [   11.560408] resolvectl[699]: [0;1;31mFailed to set DNS configuration: Invalid argument[0m
[2025-07-07 07:09:42] [   11.563336] systemd[1]: [0;1;[email protected]: Main process exited, code=exited, status=1/FAILURE[0m
[2025-07-07 07:09:42] [   11.564178] systemd[1]: [0;1;38:5:[email protected]: Failed with result 'exit-code'.[0m
[2025-07-07 07:09:42] [[0;1;31mFAILED[0m] Failed to start [0;1;39mqubes-network-upli…[0m Qubes network uplink (eth0) setup.

[2025-07-07 07:09:42] [   11.566328] systemd[1]: [0;1;31mFailed to start [email protected] - Qubes network uplink (eth0) setup.[0m
[2025-07-07 07:09:42] See 'systemctl status [email protected]' for details.

[2025-07-07 07:09:42] [   11.569483] systemctl[590]: [0;1;31mJob for [email protected] failed because the control process exited with error code.[0m
[2025-07-07 07:09:42] [   11.569982] systemctl[590]: [0;1;31mSee "systemctl status [email protected]" and "journalctl -xeu [email protected]" for details.[0m
[2025-07-07 07:09:42] [   11.573268] (udev-worker)[387]: [0;1;38:5:185meth0: Process '/usr/bin/systemctl restart --job-mode=replace [email protected]' failed with exit code 1.[0m
[2025-07-07 07:09:42] [   11.576306] systemd[1]: [0;1;39mqubes-network-uplink.service: Main process exited, code=exited, status=1/FAILURE[0m
[2025-07-07 07:09:42] [[0;1;31mFAILED[0m] Failed to start [0;1;39mqubes-network-upli…ervice[0m - Qubes network uplink wait.

[2025-07-07 07:09:42] [   11.576905] systemd[1]: [0;1;38:5:185mqubes-network-uplink.service: Failed with result 'exit-code'.[0m
[2025-07-07 07:09:42] [   11.577185] systemd[1]: [0;1;31mFailed to start qubes-network-uplink.service - Qubes network uplink wait.[0m
[2025-07-07 07:09:42] See 'systemctl status qubes-network-uplink.service' for details.

This job doesn't have IPv6 enabled.

marmarek avatar Jul 07 '25 11:07 marmarek

OpenQA test summary

Complete test suite and dependencies: https://openqa.qubes-os.org/tests/overview?distri=qubesos&version=4.3&build=2025090403-4.3&flavor=pull-requests

Test run included the following:

  • https://github.com/QubesOS/qubes-core-admin/pull/695 (https://github.com/QubesOS/qubes-core-admin/pull/695/commits/4135aed4594e05055ff50a6bc75b1b7d082edb02)
  • https://github.com/QubesOS/qubes-core-admin/pull/696 (https://github.com/QubesOS/qubes-core-admin/pull/696/commits/d510b59adc6e716aa000a3df9f9ce2d1d8930f55)
  • https://github.com/QubesOS/qubes-core-agent-linux/pull/592 (https://github.com/QubesOS/qubes-core-agent-linux/pull/592/commits/c0c7d420ec656b4035ea328da4bc101a26ae8153)

New failures, excluding unstable

Compared to: https://openqa.qubes-os.org/tests/overview?distri=qubesos&version=4.3&build=2025081011-4.3&flavor=update

  • system_tests_basic_vm_qrexec_gui

    • startup: unnamed test (unknown)
    • startup: Failed (test died) # Test died: no candidate needle with tag(s) 'nm-connection-establi...

  • system_tests_network

    • VmNetworking_fedora-42-xfce: test_010_simple_proxyvm (failure) AssertionError: 2 != 0 : Ping by IP from AppVM failed

    • VmNetworking_fedora-42-xfce: test_030_firewallvm_firewall (failure) AssertionError: 2 != 0 : Ping by name failed (should be allowed now)

    • VmNetworking_fedora-42-xfce: test_202_fake_ip_firewall (failure) AssertionError: 2 != 0 : Ping by name should be allowed

    • VmNetworking_fedora-42-xfce: test_204_fake_ip_proxy (failure) ~~~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^^^^... AssertionError: 2 != 0

    • VmNetworking_fedora-42-xfce: test_211_custom_ip_proxy (failure) ~~~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^^^^... AssertionError: 2 != 0

    • VmNetworking_fedora-42-xfce: test_212_custom_ip_firewall (failure) AssertionError: 2 != 0 : Ping by name should be allowed

  • system_tests_gui_interactive

    • clipboard_and_web: unnamed test (unknown)
    • clipboard_and_web: Failed (test died) # Test died: no candidate needle with tag(s) 'qubes-website' matche...

  • system_tests_guivm_gui_interactive

    • clipboard_and_web: unnamed test (unknown)
    • clipboard_and_web: Failed (test died) # Test died: no candidate needle with tag(s) 'qubes-website' matche...

  • system_tests_network_ipv6

    • VmIPv6Networking_debian-13-xfce: test_530_ipv6_firewallvm_firewall (failure) AssertionError: 1 != 0 : TCP (IPv4) connection failed (should be al...

    • VmIPv6Networking_debian-13-xfce: test_560_ipv6_dns_only (failure) ~~~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^^^^... AssertionError: 2 != 0

  • system_tests_dispvm

  • system_tests_kde_gui_interactive

    • clipboard_and_web: unnamed test (unknown)
    • clipboard_and_web: Failed (test died) # Test died: no candidate needle with tag(s) 'qubes-website' matche...

  • system_tests_guivm_vnc_gui_interactive

    • clipboard_and_web: unnamed test (unknown)
    • clipboard_and_web: Failed (test died) # Test died: no candidate needle with tag(s) 'qubes-website' matche...

  • system_tests_qwt_win10@hw13

    • windows_install: Failed (test died) # Test died: command './install.sh' failed at /usr/lib/os-autoinst/...

  • system_tests_qwt_win10_seamless@hw13

    • windows_install: Failed (test died) # Test died: command './install.sh' failed at /usr/lib/os-autoinst/...

  • system_tests_qwt_win11@hw13

    • windows_install: Failed (test died) # Test died: command './install.sh' failed at /usr/lib/os-autoinst/...

  • system_tests_gui_interactive_preloaded

    • clipboard_and_web: unnamed test (unknown)
    • clipboard_and_web: Failed (test died) # Test died: no candidate needle with tag(s) 'qubes-website' matche...

  • system_tests_guivm_gpu_gui_interactive@hw13

    • clipboard_and_web: unnamed test (unknown)
    • clipboard_and_web: Failed (test died) # Test died: no candidate needle with tag(s) 'qubes-website' matche...

  • install_default_upload@hw7

    • install_do_user: unnamed test (unknown)

    • install_do_user: Failed (test died) # Test died: no candidate needle with tag(s) 'installer-post-instal...

    • install_do_user: wait_serial (wait serial expected) # wait_serial expected: qr/4tG2W-\d+-/...

Failed tests

36 failures

  • system_tests_basic_vm_qrexec_gui

    • startup: unnamed test (unknown)
    • startup: Failed (test died) # Test died: no candidate needle with tag(s) 'nm-connection-establi...

  • system_tests_network

    • VmNetworking_fedora-42-xfce: test_010_simple_proxyvm (failure) AssertionError: 2 != 0 : Ping by IP from AppVM failed

    • VmNetworking_fedora-42-xfce: test_030_firewallvm_firewall (failure) AssertionError: 2 != 0 : Ping by name failed (should be allowed now)

    • VmNetworking_fedora-42-xfce: test_202_fake_ip_firewall (failure) AssertionError: 2 != 0 : Ping by name should be allowed

    • VmNetworking_fedora-42-xfce: test_204_fake_ip_proxy (failure) ~~~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^^^^... AssertionError: 2 != 0

    • VmNetworking_fedora-42-xfce: test_211_custom_ip_proxy (failure) ~~~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^^^^... AssertionError: 2 != 0

    • VmNetworking_fedora-42-xfce: test_212_custom_ip_firewall (failure) AssertionError: 2 != 0 : Ping by name should be allowed

  • system_tests_extra

    • TC_00_QVCTest_whonix-workstation-17: test_010_screenshare (failure) AssertionError: 1 != 0 : Timeout waiting for /dev/video0 in test-in...

  • system_tests_gui_interactive

    • clipboard_and_web: unnamed test (unknown)
    • clipboard_and_web: Failed (test died) # Test died: no candidate needle with tag(s) 'qubes-website' matche...

  • system_tests_guivm_gui_interactive

    • clipboard_and_web: unnamed test (unknown)
    • clipboard_and_web: Failed (test died) # Test died: no candidate needle with tag(s) 'qubes-website' matche...

  • system_tests_network_ipv6

    • VmIPv6Networking_debian-13-xfce: test_530_ipv6_firewallvm_firewall (failure) AssertionError: 1 != 0 : TCP (IPv4) connection failed (should be al...

    • VmIPv6Networking_debian-13-xfce: test_560_ipv6_dns_only (failure) ~~~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^^^^... AssertionError: 2 != 0

  • system_tests_dispvm

  • system_tests_kde_gui_interactive

    • clipboard_and_web: unnamed test (unknown)
    • clipboard_and_web: Failed (test died) # Test died: no candidate needle with tag(s) 'qubes-website' matche...

  • system_tests_guivm_vnc_gui_interactive

    • clipboard_and_web: unnamed test (unknown)
    • clipboard_and_web: Failed (test died) # Test died: no candidate needle with tag(s) 'qubes-website' matche...

  • system_tests_qwt_win10@hw13

    • windows_install: Failed (test died) # Test died: command './install.sh' failed at /usr/lib/os-autoinst/...

  • system_tests_qwt_win10_seamless@hw13

    • windows_install: Failed (test died) # Test died: command './install.sh' failed at /usr/lib/os-autoinst/...

  • system_tests_qwt_win11@hw13

    • windows_install: Failed (test died) # Test died: command './install.sh' failed at /usr/lib/os-autoinst/...

  • system_tests_gui_interactive_preloaded

    • clipboard_and_web: unnamed test (unknown)
    • clipboard_and_web: Failed (test died) # Test died: no candidate needle with tag(s) 'qubes-website' matche...

  • system_tests_guivm_gpu_gui_interactive@hw13

    • clipboard_and_web: unnamed test (unknown)
    • clipboard_and_web: Failed (test died) # Test died: no candidate needle with tag(s) 'qubes-website' matche...

  • install_default_upload@hw7

    • install_do_user: unnamed test (unknown)

    • install_do_user: Failed (test died) # Test died: no candidate needle with tag(s) 'installer-post-instal...

    • install_do_user: wait_serial (wait serial expected) # wait_serial expected: qr/4tG2W-\d+-/...

Fixed failures

Compared to: https://openqa.qubes-os.org/tests/149225#dependencies

73 fixed

Unstable tests

Performance Tests

Performance degradation:

15 performance degradations
  • whonix-gateway-17_exec-data-duplex: 81.57 :small_red_triangle: ( previous job: 68.38, degradation: 119.28%)
  • whonix-workstation-17_exec-data-simplex: 73.82 :small_red_triangle: ( previous job: 66.80, degradation: 110.50%)
  • whonix-workstation-17_socket-data-duplex: 163.36 :small_red_triangle: ( previous job: 147.97, degradation: 110.40%)
  • dom0_root_seq1m_q8t1_read 3:read_bandwidth_kb: 332986.00 :small_red_triangle: ( previous job: 497426.00, degradation: 66.94%)
  • dom0_root_seq1m_q8t1_write 3:write_bandwidth_kb: 225403.00 :small_red_triangle: ( previous job: 265260.00, degradation: 84.97%)
  • dom0_root_seq1m_q1t1_read 3:read_bandwidth_kb: 146811.00 :small_red_triangle: ( previous job: 431512.00, degradation: 34.02%)
  • dom0_root_seq1m_q1t1_write 3:write_bandwidth_kb: 84737.00 :small_red_triangle: ( previous job: 196254.00, degradation: 43.18%)
  • fedora-42-xfce_root_rnd4k_q32t1_read 3:read_bandwidth_kb: 77202.00 :small_red_triangle: ( previous job: 86001.00, degradation: 89.77%)
  • fedora-42-xfce_root_rnd4k_q1t1_write 3:write_bandwidth_kb: 1054.00 :small_red_triangle: ( previous job: 1368.00, degradation: 77.05%)
  • fedora-42-xfce_private_seq1m_q1t1_write 3:write_bandwidth_kb: 65353.00 :small_red_triangle: ( previous job: 79539.00, degradation: 82.16%)
  • fedora-42-xfce_private_rnd4k_q1t1_write 3:write_bandwidth_kb: 624.00 :small_red_triangle: ( previous job: 1251.00, degradation: 49.88%)
  • fedora-42-xfce_volatile_seq1m_q1t1_write 3:write_bandwidth_kb: 43336.00 :small_red_triangle: ( previous job: 64217.00, degradation: 67.48%)
  • fedora-42-xfce_volatile_rnd4k_q32t1_read 3:read_bandwidth_kb: 76879.00 :small_red_triangle: ( previous job: 87141.00, degradation: 88.22%)
  • fedora-42-xfce_volatile_rnd4k_q32t1_write 3:write_bandwidth_kb: 2521.00 :small_red_triangle: ( previous job: 4098.00, degradation: 61.52%)
  • fedora-42-xfce_volatile_rnd4k_q1t1_write 3:write_bandwidth_kb: 1882.00 :small_red_triangle: ( previous job: 2384.00, degradation: 78.94%)

Remaining performance tests:

57 tests
  • debian-13-xfce_exec: 8.19 :green_circle: ( previous job: 8.36, improvement: 97.91%)
  • debian-13-xfce_exec-root: 27.72 :small_red_triangle: ( previous job: 27.36, degradation: 101.30%)
  • debian-13-xfce_socket: 8.43 :green_circle: ( previous job: 8.57, improvement: 98.34%)
  • debian-13-xfce_socket-root: 7.89 :green_circle: ( previous job: 8.26, improvement: 95.58%)
  • debian-13-xfce_exec-data-simplex: 60.86 :green_circle: ( previous job: 72.43, improvement: 84.02%)
  • debian-13-xfce_exec-data-duplex: 74.44 :green_circle: ( previous job: 76.65, improvement: 97.12%)
  • debian-13-xfce_exec-data-duplex-root: 78.90 :green_circle: ( previous job: 91.79, improvement: 85.96%)
  • debian-13-xfce_socket-data-duplex: 135.88 :small_red_triangle: ( previous job: 133.45, degradation: 101.83%)
  • fedora-42-xfce_exec: 9.15 :small_red_triangle: ( previous job: 9.06, degradation: 101.00%)
  • fedora-42-xfce_exec-root: 59.01 :small_red_triangle: ( previous job: 58.19, degradation: 101.42%)
  • fedora-42-xfce_socket: 7.49 :green_circle: ( previous job: 8.48, improvement: 88.37%)
  • fedora-42-xfce_socket-root: 8.37 :small_red_triangle: ( previous job: 8.18, degradation: 102.27%)
  • fedora-42-xfce_exec-data-simplex: 77.69 :green_circle: ( previous job: 78.48, improvement: 98.98%)
  • fedora-42-xfce_exec-data-duplex: 69.05 :small_red_triangle: ( previous job: 67.92, degradation: 101.67%)
  • fedora-42-xfce_exec-data-duplex-root: 104.99 :small_red_triangle: ( previous job: 96.36, degradation: 108.95%)
  • fedora-42-xfce_socket-data-duplex: 138.27 :green_circle: ( previous job: 142.58, improvement: 96.97%)
  • whonix-gateway-17_exec: 7.63 :green_circle: ( previous job: 8.12, improvement: 94.05%)
  • whonix-gateway-17_exec-root: 39.08 :green_circle: ( previous job: 41.05, improvement: 95.20%)
  • whonix-gateway-17_socket: 7.48 :green_circle: ( previous job: 8.52, improvement: 87.70%)
  • whonix-gateway-17_socket-root: 8.17 :small_red_triangle: ( previous job: 8.12, degradation: 100.59%)
  • whonix-gateway-17_exec-data-simplex: 81.20 :green_circle: ( previous job: 83.60, improvement: 97.13%)
  • whonix-gateway-17_exec-data-duplex-root: 88.19 :green_circle: ( previous job: 99.37, improvement: 88.75%)
  • whonix-gateway-17_socket-data-duplex: 163.85 :green_circle: ( previous job: 167.12, improvement: 98.05%)
  • whonix-workstation-17_exec: 7.87 :small_red_triangle: ( previous job: 7.57, degradation: 103.97%)
  • whonix-workstation-17_exec-root: 57.97 :small_red_triangle: ( previous job: 56.76, degradation: 102.13%)
  • whonix-workstation-17_socket: 8.65 :small_red_triangle: ( previous job: 8.59, degradation: 100.77%)
  • whonix-workstation-17_socket-root: 9.18 :small_red_triangle: ( previous job: 8.89, degradation: 103.29%)
  • whonix-workstation-17_exec-data-duplex: 73.28 :green_circle: ( previous job: 74.50, improvement: 98.35%)
  • whonix-workstation-17_exec-data-duplex-root: 96.49 :green_circle: ( previous job: 102.34, improvement: 94.28%)
  • dom0_root_rnd4k_q32t1_read 3:read_bandwidth_kb: 31305.00 :green_circle: ( previous job: 23940.00, improvement: 130.76%)
  • dom0_root_rnd4k_q32t1_write 3:write_bandwidth_kb: 2228.00 :small_red_triangle: ( previous job: 2446.00, degradation: 91.09%)
  • dom0_root_rnd4k_q1t1_read 3:read_bandwidth_kb: 11661.00 :green_circle: ( previous job: 5874.00, improvement: 198.52%)
  • dom0_root_rnd4k_q1t1_write 3:write_bandwidth_kb: 1524.00 :green_circle: ( previous job: 29.00, improvement: 5255.17%)
  • dom0_varlibqubes_seq1m_q8t1_read 3:read_bandwidth_kb: 461927.00 :green_circle: ( previous job: 292489.00, improvement: 157.93%)
  • dom0_varlibqubes_seq1m_q8t1_write 3:write_bandwidth_kb: 138148.00 :green_circle: ( previous job: 110817.00, improvement: 124.66%)
  • dom0_varlibqubes_seq1m_q1t1_read 3:read_bandwidth_kb: 446772.00 :green_circle: ( previous job: 137802.00, improvement: 324.21%)
  • dom0_varlibqubes_seq1m_q1t1_write 3:write_bandwidth_kb: 165777.00 :green_circle: ( previous job: 121719.00, improvement: 136.20%)
  • dom0_varlibqubes_rnd4k_q32t1_read 3:read_bandwidth_kb: 105073.00 :green_circle: ( previous job: 103932.00, improvement: 101.10%)
  • dom0_varlibqubes_rnd4k_q32t1_write 3:write_bandwidth_kb: 8145.00 :green_circle: ( previous job: 6356.00, improvement: 128.15%)
  • dom0_varlibqubes_rnd4k_q1t1_read 3:read_bandwidth_kb: 7082.00 :small_red_triangle: ( previous job: 7695.00, degradation: 92.03%)
  • dom0_varlibqubes_rnd4k_q1t1_write 3:write_bandwidth_kb: 4890.00 :green_circle: ( previous job: 3925.00, improvement: 124.59%)
  • fedora-42-xfce_root_seq1m_q8t1_read 3:read_bandwidth_kb: 402833.00 :green_circle: ( previous job: 366891.00, improvement: 109.80%)
  • fedora-42-xfce_root_seq1m_q8t1_write 3:write_bandwidth_kb: 195542.00 :green_circle: ( previous job: 140215.00, improvement: 139.46%)
  • fedora-42-xfce_root_seq1m_q1t1_read 3:read_bandwidth_kb: 327680.00 :green_circle: ( previous job: 299764.00, improvement: 109.31%)
  • fedora-42-xfce_root_seq1m_q1t1_write 3:write_bandwidth_kb: 79319.00 :green_circle: ( previous job: 47575.00, improvement: 166.72%)
  • fedora-42-xfce_root_rnd4k_q32t1_write 3:write_bandwidth_kb: 4039.00 :green_circle: ( previous job: 3020.00, improvement: 133.74%)
  • fedora-42-xfce_root_rnd4k_q1t1_read 3:read_bandwidth_kb: 9106.00 :green_circle: ( previous job: 9042.00, improvement: 100.71%)
  • fedora-42-xfce_private_seq1m_q8t1_read 3:read_bandwidth_kb: 351635.00 :small_red_triangle: ( previous job: 387500.00, degradation: 90.74%)
  • fedora-42-xfce_private_seq1m_q8t1_write 3:write_bandwidth_kb: 234633.00 :green_circle: ( previous job: 136640.00, improvement: 171.72%)
  • fedora-42-xfce_private_seq1m_q1t1_read 3:read_bandwidth_kb: 342224.00 :green_circle: ( previous job: 325139.00, improvement: 105.25%)
  • fedora-42-xfce_private_rnd4k_q32t1_read 3:read_bandwidth_kb: 91381.00 :green_circle: ( previous job: 87396.00, improvement: 104.56%)
  • fedora-42-xfce_private_rnd4k_q32t1_write 3:write_bandwidth_kb: 3768.00 :green_circle: ( previous job: 3765.00, improvement: 100.08%)
  • fedora-42-xfce_private_rnd4k_q1t1_read 3:read_bandwidth_kb: 8582.00 :small_red_triangle: ( previous job: 8992.00, degradation: 95.44%)
  • fedora-42-xfce_volatile_seq1m_q8t1_read 3:read_bandwidth_kb: 373957.00 :small_red_triangle: ( previous job: 383531.00, degradation: 97.50%)
  • fedora-42-xfce_volatile_seq1m_q8t1_write 3:write_bandwidth_kb: 147247.00 :small_red_triangle: ( previous job: 157382.00, degradation: 93.56%)
  • fedora-42-xfce_volatile_seq1m_q1t1_read 3:read_bandwidth_kb: 294295.00 :green_circle: ( previous job: 293225.00, improvement: 100.36%)
  • fedora-42-xfce_volatile_rnd4k_q1t1_read 3:read_bandwidth_kb: 8293.00 :small_red_triangle: ( previous job: 8804.00, degradation: 94.20%)

qubesos-bot avatar Jul 07 '25 16:07 qubesos-bot

Not all of the failures are caused by this PR (or the core-admin one), but it seems most of them are.

marmarek avatar Jul 07 '25 17:07 marmarek

  • TypeError: sequence item 0: expected str instance, list found

Fixed in 7658b78. I guess I would expect this one to have been caught in run-tests. This looks like a good regression case to check for if anyone is ever adding unit tests here (I guess separate from this PR).

Also added in a little bit of typings for this here.

3nprob avatar Jul 07 '25 22:07 3nprob

openQA run is still in progress, but I already see some failures: https://openqa.qubes-os.org/tests/145922#step/clipboard_and_web/19

[2025-07-07 07:09:42] [   11.559294] resolvectl[699]: �[0;1;31mFailed to parse DNS server address: �[0m
[2025-07-07 07:09:42] [   11.560408] resolvectl[699]: �[0;1;31mFailed to set DNS configuration: Invalid argument�[0m
[2025-07-07 07:09:42] [   11.563336] systemd[1]: �[0;1;[email protected]: Main process exited, code=exited, status=1/FAILURE�[0m
[2025-07-07 07:09:42] [   11.564178] systemd[1]: �[0;1;38:5:[email protected]: Failed with result 'exit-code'.�[0m
[2025-07-07 07:09:42] [�[0;1;31mFAILED�[0m] Failed to start �[0;1;39mqubes-network-upli…�[0m Qubes network uplink (eth0) setup.

[2025-07-07 07:09:42] [   11.566328] systemd[1]: �[0;1;31mFailed to start [email protected] - Qubes network uplink (eth0) setup.�[0m
[2025-07-07 07:09:42] See 'systemctl status [email protected]' for details.

[2025-07-07 07:09:42] [   11.569483] systemctl[590]: �[0;1;31mJob for [email protected] failed because the control process exited with error code.�[0m
[2025-07-07 07:09:42] [   11.569982] systemctl[590]: �[0;1;31mSee "systemctl status [email protected]" and "journalctl -xeu [email protected]" for details.�[0m
[2025-07-07 07:09:42] [   11.573268] (udev-worker)[387]: �[0;1;38:5:185meth0: Process '/usr/bin/systemctl restart --job-mode=replace [email protected]' failed with exit code 1.�[0m
[2025-07-07 07:09:42] [   11.576306] systemd[1]: �[0;1;39mqubes-network-uplink.service: Main process exited, code=exited, status=1/FAILURE�[0m
[2025-07-07 07:09:42] [�[0;1;31mFAILED�[0m] Failed to start �[0;1;39mqubes-network-upli…ervice�[0m - Qubes network uplink wait.

[2025-07-07 07:09:42] [   11.576905] systemd[1]: �[0;1;38:5:185mqubes-network-uplink.service: Failed with result 'exit-code'.�[0m
[2025-07-07 07:09:42] [   11.577185] systemd[1]: �[0;1;31mFailed to start qubes-network-uplink.service - Qubes network uplink wait.�[0m
[2025-07-07 07:09:42] See 'systemctl status qubes-network-uplink.service' for details.

This job doesn't have IPv6 enabled.

This one is still the case.

marmarek avatar Jul 08 '25 01:07 marmarek

@3nprob @1cho1ce are any of you still working on this? the current version results in no DNS working at all (neither v4 nor v6)...

marmarek avatar Sep 04 '25 12:09 marmarek

@3nprob Are you still interested in getting this through, or would you mind if I pick up where you left off? Getting IPv6 DNS working is desirable for Whonix (which I help develop), so if you're busy elsewhere, I'd like to continue work on these PRs. I'm happy to just provide some extra testing if you want to keep working on this though.

ArrayBolt3 avatar Oct 05 '25 01:10 ArrayBolt3

@3nprob Are you still interested in getting this through, or would you mind if I pick up where you left off? Getting IPv6 DNS working is desirable for Whonix (which I help develop), so if you're busy elsewhere, I'd like to continue work on these PRs. I'm happy to just provide some extra testing if you want to keep working on this though.

It would be super sweet if you'd like to collaborate on or even pick this up! I'll see if I can give you push rights to this branch so we can continue on the same PR.

Unfortunately I don't currently have a great setup to test this properly at the moment. Noting from above that it doesn't seem to be in a properly working state currently. Testing (both manual and additional coverage) would be great!

3nprob avatar Oct 05 '25 02:10 3nprob

Thanks! I think I can test this since my network supports both IPv4 and IPv6 and I can disable IPv6 for testing. I don't need push rights, I'll just open new PRs, but thank you :) And thanks for everything you and 1cho1ce did to get things this far!

ArrayBolt3 avatar Oct 05 '25 02:10 ArrayBolt3

Unfortunately, as promising as this situation looked, it looks like it fundamentally won't work. Some routers (including mine) advertise a link-local IPv6 address to devices on the LAN, so /etc/resolv.conf in sys-net ends up containing a line like nameserver fe80::XXXX:XXXX:XXXX:XXXX%wls6 rather than a "normal" IPv6 address. Based on my research and arguing with some folks in #networking on libera.chat, it looks like there isn't a way to use DNAT to target a link-local address like this, at least not an easy way. What ends up happening is that sys-net successfully translates the virtual IPv6 DNS address to the link-local address of the router, except it's missing the bit that specifies which interface the link-local address is on. sys-net then attempts to route the request to sys-firewall rather than routing it to the router (why it chooses that, I have no idea, maybe random chance), then the forward chain configured in nft detects that the packet is trying to go out interface group 2 and the packet is dropped.

We may have to resort to something along the lines of socat to get this to work right.

ArrayBolt3 avatar Oct 05 '25 05:10 ArrayBolt3

Unfortunately, as promising as this situation looked, it looks like it fundamentally won't work. Some routers (including mine) advertise a link-local IPv6 address to devices on the LAN, so /etc/resolv.conf in sys-net ends up containing a line like nameserver fe80::XXXX:XXXX:XXXX:XXXX%wls6 rather than a "normal" IPv6 address. Based on my research and arguing with some folks in #networking on libera.chat, it looks like there isn't a way to use DNAT to target a link-local address like this, at least not an easy way.

Thanks for digging! So not that easy, eh..

What ends up happening is that sys-net successfully translates the virtual IPv6 DNS address to the link-local address of the router, except it's missing the bit that specifies which interface the link-local address is on. sys-net then attempts to route the request to sys-firewall rather than routing it to the router (why it chooses that, I have no idea, maybe random chance), then the forward chain configured in nft detects that the packet is trying to go out interface group 2 and the packet is dropped.

So the request comes from (or via) sys-firewall so sys-net sends it "back in" where it should go "out".

We may have to resort to something along the lines of socat to get this to work right.

Just a thought: If going all the way to socat, it makes me think might as well use a more purpose-made software... In the distant past, there was dnsmasq running in netvm but this is no longer considered an option (at least not with ports exposed) due to its attack surface.

I wonder if it could be an option to run systemd-resolved (rather than socat) inside netvm, and route dns requests to a localhost address it listens on, with systemd-resolved then taking care of relaying?

3nprob avatar Oct 05 '25 06:10 3nprob

@3nprob From some discussion in the Qubes OS Matrix room, currently systemd-resolved is intentionally being bypassed as much as possible, because upstream doesn't recommend exposing it to the LAN, and so it's unclear whether it can safely handle potentially malicious clients. A different DNS resolver that is designed to work in this way might be an option, but then you have to run a DNS resolver in sys-net which is painful because sys-net is already severely resource-constrained by design.

IIUC, socat would allow us to simply DNAT the virtual IPv6 DNS address to sys-net's internal network adapter's IP address, then we can forward it to the real DNS server thereafter. It's crude, but as long as all socat does is pass bytes from point A to point B and then back again, it should work and not introduce any further attack surface. (Then again, I don't know what socat's attack surface looks like; I'd assume it's pretty close to zero, but maybe not?)

@marmarek Since it doesn't look like DNAT alone is going to work, what are your thoughts on adding socat to the solution?

ArrayBolt3 avatar Oct 05 '25 14:10 ArrayBolt3

Alright, this is slightly horrible but after some fighting I have things working kinda...

In order to use socat to forward the DNS queries, we have to have an IP address to have socat listen on. The loopback address would be near-ideal if it weren't for the fact that the IPv6 protocol itself intentionally prevents DNAT to loopback. We can't use a link-local address for the same reason we can't trivially forward packets to the router in all situations, so we have to use a "real" IP. We can either use an IP on one of sys-nets external listening interfaces (which sounds like a distinctly bad idea if you ask me, since then anyone on the LAN can use the Qubes machine as a DNS proxy), or we can use an internal virtual interface (which sounds much better). The problem with using an internal virtual interface is that those interfaces don't even exist unless sys-firewall (or something else that uses sys-net as its NetVM) is powered on, and if there are multiple VMs using sys-net as a NetVM at the same time, there are going to be multiple internal interfaces, only one of which we can choose as the DNAT destination. For the initial proof of concept, I didn't bother trying to write code to handle this and instead just manually punched in the nftables rules to DNAT anything coming from one of the virtual IPv6 DNS addresses to fd09:24ef:4179::a89:7, the IP address my sys-net happens to have on the interface that connects it to sys-firewall. I think for this to work in a real-world scenario, we'd have to create a "dangling" virtual network interface that exists only to provide an IP address that can be DNAT'ed to.

The internal-facing interface is part of if group 2, and so the default input chain will see that we have a packet incoming to if group 2 and will then reject it because it isn't one of two particular router-related ICMPv6 packets. To overcome that, I added a rule to the input chain right before iifgroup 2 goto _icmpv6, which reads meta l4proto { tcp, udp } th dport 53 accept (translation: allow traffic to port 53 with both TCP and UDP). I'm not really thrilled with this, because it means anything can try to talk to port 53 on any interface and have it be accepted. A better solution would be to rename the _icmpv6 chain to something like internalif and then add the rule there, so that DNS packets are only allowed if they are attempting to come in through the internal-facing interface. But anyway, adding that rule managed to stop packets from getting dropped. If we went with the dangling interface solution suggested above, we'll instead need a rule similar to the iif "lo" accept one that will allow TCP and UDP coming in through the dangling interface on port 53 to be accepted.

At this point I was able to fire up socat with sudo socat -T1 UDP6-RECVFROM:53,fork,bind=[fd09:24ef:4179::a89:7] UDP6-SENDTO:[fe80::5463:eaff:fe3a:6398%wls6]:53 - this basically forwards any UDP packet from the internal-facing interface to the link-local address of the router on the external-facing interface, which is what we want. Once this is done, I'm able to run dig -6 google.com in sys-firewall and get a valid reply back from the router, i.e. DNS is working, but the problem with this is that this literally forks off a new process every single time a DNS packet is received from the internal-facing interface. It will die as soon as it's done sending the response back to the client, but having the overhead of a fork for every DNS query is not great performance-wise.

Putting all the above together:

  • We'll need a dedicated virtual interface so that we have an IP address we can DNAT IPv6 DNS packets to.
  • We'll need to DNAT IPv6 DNS packets to that rather than to the router.
  • socat will incur a fork for every DNS query. It might be possible to make a dedicated tool that won't incur this cost, which might be a good idea anyway.

As ugly as this solution seems to be, it does work, so it looks like this will be possible!

ArrayBolt3 avatar Oct 05 '25 23:10 ArrayBolt3

Using socat enables one more trick - it doesn't care about protocol on the listening and sending side, so you can translate IPv4 to IPv6 and back. So, technically we could have DNS1 and DNS2 addresses in v4 and v6 flavors, and they would translate to upstream DNS1 and DNS2 regardless of which protocol they use. On one hand, it's tempting, since it would avoid needing every qube to try IPv6 first just to learn it isn't available at the time (when you're connected to IPv4-only network). But on the other hand, it feels like a lot could go wrong...

I did used something similar before, to be able to use IPv6-only network on qubes - simply socat listening on 10.139.1.1 and forwarding to whatever IPv6 DNS was there. But I don't have that script anymore, so I can't check what I did about listening address... I guess went with a dummy interface for it.

marmarek avatar Oct 06 '25 00:10 marmarek

If we want to do that, maybe it would be easier to forget all of the IPv6 DNAT stuff, make internal DNS always go over IPv4, and then use socat to forward those DNS requests to an IPv6 server if that was the only server available? That could come with substantially simpler code, and we wouldn't need a dummy interface since we can just do what we're already doing with IPv4 DNAT. Granted, applications that refuse to even try using IPv4 DNS for whatever reason wouldn't work with that, but I don't know of any such applications, and in a pinch one could use a second socat instance running directly in the AppVM with the problematic application to translate IPv6 DNS requests to IPv4 for forwarding to sys-net.

ArrayBolt3 avatar Oct 06 '25 00:10 ArrayBolt3

While adding this feature, it would be nice to consider already a case where there is only IPv6 traffic. Just because it's rather awkward to have need IPv4 just for (internal) DNS when everything else is on IPv6 already. On the other hand, there s still a very long way to IPv6-only internet, so maybe it doesn't really matter yet...

marmarek avatar Oct 06 '25 00:10 marmarek