Flatcar icon indicating copy to clipboard operation
Flatcar copied to clipboard
verified publisher icon flatcar

Not generating correct VLAN netdev files from kernel args in early user space

Open tcldr opened this issue 8 months ago • 2 comments

Description

When a VLAN configuration is passed in via dracut kernel args (vlan=, ip=) the parse-ip-for-networkd.service fails to generate the vlan interface .netdev and .network files.

Impact

Can't boot Flatcar instances that require remote resources in ignition phase.

Environment and steps to reproduce

  1. Set-up:
  • Create minimal butane config that requires a remote resource and VLAN based network:
variant: flatcar
version: 1.1.0

storage:
  files:
    - path: /etc/sysupdate.d/noop.conf
      contents:
        source: https://extensions.flatcar.org/extensions/noop.conf     
kernel_arguments:
  should_exist:
    - vlan=enp9s0.4000:enp9s0
    - ip=172.16.2.101::172.16.2.1:24:worker-1:enp9s0.4000:off:1400
    - nameserver=185.12.64.1
    - nameserver=185.12.64.2
  1. Task:
  • Attempt to boot instance on described network
  1. Action(s):
  • Instance fails to boot
  1. Error:
  • None. As ignition fails, no journal is produced.

Expected behavior

Instance fetches remote resources and completes ignition successfully.

Additional information

Although I haven't been able to retrieve logs from an instance that fails to get through the ignition stage, it is possible to get an idea of what's happening behind the scenes by booting an instance without a remote file dependency that boots successfully. From here it's possible to see how the network is being configured through the various boot stages.

AFAICT, looking at the timeline of a successful boot, you can see that network configuration based on dracut kernel args actually occurs twice. Once during in early user space (ignition phase) and then again once ignition is complete and the instance enters user space (regular operations).

The configuration created in user space now works correctly after fix #1706. These are the files that end up in /run/systemd/network/70-*.{.network,.netdev}.

However, the configuration created in early user space (ignition phase) appears to be based on an earlier version of systemd-network-generator that doesn't incorporate this fix. This configuration is a temporary file placed at /etc/systemd/network/10-dracut-cmdline-99.network and is generated by a script found in the bootengine repo.

Gathering the critical events of the timeline below:

  • 13:47:49: Finished parse-ip-for-networkd.service - Write systemd-networkd units from cmdline. (Assume this is the creation of /etc/systemd/network/10-dracut-cmdline-99.network)
  • 13:47:49: Configuring with /etc/systemd/network/10-dracut-cmdline-99.network. (Configured with the faulty network file prior to the ignition phase)
  • 13:47:49: Ignition parsing seems to start here
  • 13:47:51: /sysroot/run/systemd/network/70-*{.network,.netdev} created (After the original network is already up and ignition has begun)
  • 13:47:51: /sysroot/etc/systemd/network/10-*{.network,.netdev} created

And here's the logs/filesystem timestamps:

core@localhost ~ $ journalctl -g network --no-pager
Apr 02 13:47:46 localhost systemd[1]: Listening on systemd-networkd.socket - Network Service Netlink Socket.
Apr 02 13:47:46 localhost systemd[1]: Finished network-cleanup.service - Network Cleanup.
Apr 02 13:47:46 localhost systemd[1]: Starting systemd-resolved.service - Network Name Resolution...
Apr 02 13:47:46 localhost systemd[1]: Started systemd-resolved.service - Network Name Resolution.
Apr 02 13:47:46 localhost systemd[1]: Reached target nss-lookup.target - Host and Network Name Lookups.
Apr 02 13:47:49 localhost systemd[1]: afterburn-network-kargs.service - Afterburn Initrd Setup Network Kernel Arguments was skipped because no trigger condition checks were met.
Apr 02 13:47:49 localhost systemd[1]: Starting parse-ip-for-networkd.service - Write systemd-networkd units from cmdline...
Apr 02 13:47:49 localhost systemd[1]: Finished parse-ip-for-networkd.service - Write systemd-networkd units from cmdline.
Apr 02 13:47:49 localhost systemd[1]: Starting systemd-networkd.service - Network Configuration...
Apr 02 13:47:49 localhost systemd[1]: Started systemd-networkd.service - Network Configuration.
Apr 02 13:47:49 localhost systemd-networkd[893]: enp9s0: Configuring with /etc/systemd/network/10-dracut-cmdline-99.network.
Apr 02 13:47:49 localhost systemd[1]: Reached target network.target - Network.
Apr 02 13:47:51 localhost ignition[1119]: INFO     : files: createFilesystemsFiles: createFiles: op(3): [started]  writing file "/sysroot/etc/systemd/network/10-enp9s0.network"
Apr 02 13:47:51 localhost ignition[1119]: INFO     : files: createFilesystemsFiles: createFiles: op(3): [finished] writing file "/sysroot/etc/systemd/network/10-enp9s0.network"
Apr 02 13:47:51 localhost ignition[1119]: INFO     : files: createFilesystemsFiles: createFiles: op(4): [started]  writing file "/sysroot/etc/systemd/network/10-enp9s0.4000.netdev"
Apr 02 13:47:51 localhost ignition[1119]: INFO     : files: createFilesystemsFiles: createFiles: op(4): [finished] writing file "/sysroot/etc/systemd/network/10-enp9s0.4000.netdev"
Apr 02 13:47:51 localhost ignition[1119]: INFO     : files: createFilesystemsFiles: createFiles: op(5): [started]  writing file "/sysroot/etc/systemd/network/10-enp9s0.4000.network"
Apr 02 13:47:51 localhost ignition[1119]: INFO     : files: createFilesystemsFiles: createFiles: op(5): [finished] writing file "/sysroot/etc/systemd/network/10-enp9s0.4000.network"
Apr 02 13:47:51 localhost systemd[1]: Stopped target nss-lookup.target - Host and Network Name Lookups.
Apr 02 13:47:51 localhost systemd[1]: Stopped target network.target - Network.

core@localhost ~ $ ls -al --full-time /run/systemd/network
total 12
drwxr-xr-x.  2 root root 120 2025-04-02 13:47:52.184584585 +0000 .
drwxr-xr-x. 27 root root 740 2025-04-02 13:49:00.178106969 +0000 ..
-rw-r--r--.  1 root root 108 2025-04-02 13:47:51.918584557 +0000 70-enp9s0.4000.netdev
-rw-r--r--.  1 root root 282 2025-04-02 13:47:51.918584557 +0000 70-enp9s0.4000.network
-rw-r--r--.  1 root root 284 2025-04-02 13:47:51.918584557 +0000 70-enp9s0.network
lrwxrwxrwx.  1 root root  22 2025-04-02 13:47:52.184584585 +0000 resolv.conf -> ../resolve/resolv.conf

core@localhost ~ $ ls -al --full-time /etc/systemd/network
total 28
drwxr-xr-x. 1 root root 4096 2025-04-02 13:47:51.121584474 +0000 .
drwxr-xr-x. 1 root root 4096 2025-03-19 10:50:41.596661508 +0000 ..
-rw-r--r--. 1 root root    0 2025-03-19 10:06:18.000000000 +0000 .keep_sys-apps_systemd-0
-rw-r--r--. 1 root root   52 2025-04-02 13:47:51.120584474 +0000 10-enp9s0.4000.netdev
-rw-r--r--. 1 root root  200 2025-04-02 13:47:51.121584474 +0000 10-enp9s0.4000.network
-rw-r--r--. 1 root root   48 2025-04-02 13:47:51.120584474 +0000 10-enp9s0.network

EDIT: Clarifications

tcldr avatar Apr 02 '25 15:04 tcldr

You're looking at the wrong bootengine branch. This is the file you mean, although it hasn't changed much since.

chewi avatar Apr 03 '25 10:04 chewi

You're looking at the wrong bootengine branch. This is the file you mean, although it hasn't changed much since.

Ah, yes. Will update the link. Thanks!

tcldr avatar Apr 03 '25 10:04 tcldr