Flatcar icon indicating copy to clipboard operation
Flatcar copied to clipboard
verified publisher icon flatcar

Broken early networking with dual-stack configuration

Open jqueuniet opened this issue 3 months ago • 9 comments

Description

When configuring early networking for Ignition using Dracut-style ip= kernel cmdline arguments, attempting a dual-stack setup fails and generates mangled systemd network unit files. The same kernel arguments used separately for single-stack configuration (either IPv4 or IPv6) leads to a functioning network.

I've seen another IPv6-related issue at #1696 , but unless I'm mistaken, it seems provider-specific.

Impact

As early networking configuration is not functional, further network-dependant Ignition steps either hang indefinitely or timeout after the configured ignition.timeouts.httpTotal.

Environment and steps to reproduce

  1. Set-up: KVM/qemu virtual machine, using current beta qcow2 file. Ignition done with attached Butane sample.
  2. Task: Configuring Ignition early networking
  3. Action(s): a. Boot the image with attached butane file
  4. Error: crash from the systemd-network-generator.service and systemd-networkd.service systemd units

The systemd-network-generator.service seems like a red herring, the mangled configuration file is actually generated by parse-ip-for-networkd.service. Removing the /etc/systemd/network/10-dracut-cmdline-99.network file and running APPLY=1 /lib/dracut/hooks/cmdline/99-parse-ip-for-networkd.sh leads to an identical result.

Expected behavior

Network is properly configured, Ignition HTTP requests complete successfully.

Additional information

Sample Butane file:

version: 1.0.0
variant: flatcar
ignition:
  timeouts:
    http_total: 10
kernel_arguments:
  should_exist:
    - "ip=100.76.3.2::100.76.3.254:255.255.255.0:test-flatcar-ds:eth0:none:172.18.254.200"
    - "ip=[fd00:c0db:2::1]::[fd00:c0db:2:0:ffff:ffff:ffff:ffff]:[ffff:ffff:ffff:ffff::]:test-flatcar-ds:eth0:none:172.18.254.200"
storage:
  files:
    - path: /etc/sysupdate.kubernetes.d/kubernetes-v1.33.conf
      contents:
        source: https://extensions.flatcar.org/extensions/kubernetes/kubernetes-v1.33.conf

Relevant kernel args in generated Ignition:

{
  "kernelArguments": {
    "shouldExist": [
      "ip=100.76.3.2::100.76.3.254:255.255.255.0:test-flatcar-ds:eth0:none:172.18.254.200",
      "ip=[fd00:c0db:2::1]::[fd00:c0db:2:0:ffff:ffff:ffff:ffff]:[ffff:ffff:ffff:ffff::]:test-flatcar-ds:eth0:none:172.18.254.200"
    ]
  }
}

Resulting cmdline:

rootflags=rw mount.usrflags=ro BOOT_IMAGE=/flatcar/vmlinuz-a mount.usr=/dev/mapper/usr verity.usr=PARTUUID=7130c94a-213a-4e5a-8e26-6cce9662f132 rootflags=rw mount.usrflags=ro consoleblank=0 root=LABEL=ROOT console=ttyS0,115200n8 console=tty0 flatcar.first_boot=detected flatcar.oem.id=qemu flatcar.autologin ip=100.76.3.2::100.76.3.254:255.255.255.0:test-flatcar-ds.lab.plf.mgt.proxad.net:eth0:none:172.18.254.200 ip=[fd00:c0db:2::1]::[fd00:c0db:2:0:ffff:ffff:ffff:ffff]:[ffff:ffff:ffff:ffff::]:test-flatcar-ds.lab.plf.mgt.proxad.net:eth0:none:172.18.254.200  verity.usrhash=271a44cc8ea1639cfb6fdf777202a5f025fda0b3ce9b293cc4e0e7047aecb858

parse-ip-for-networkd.service logs:

Sep 24 04:20:15 localhost systemd[1]: Starting parse-ip-for-networkd.service - Write systemd-networkd units from cmdline...
Sep 24 04:20:15 localhost systemd[1]: Finished parse-ip-for-networkd.service - Write systemd-networkd units from cmdline.
Sep 24 04:21:19 localhost systemd[1]: parse-ip-for-networkd.service: Deactivated successfully.
Sep 24 04:21:19 localhost systemd[1]: Stopped parse-ip-for-networkd.service - Write systemd-networkd units from cmdline.

systemd-network-generator.service logs:

Sep 24 03:45:58 localhost systemd[1]: Starting systemd-network-generator.service - Generate network units from Kernel command line...
Sep 24 03:45:58 localhost systemd-network-generator[349]: Failed to parse kernel command line: Operation not supported
Sep 24 03:45:58 localhost systemd[1]: systemd-network-generator.service: Main process exited, code=exited, status=1/FAILURE
Sep 24 03:45:58 localhost systemd[1]: systemd-network-generator.service: Failed with result 'exit-code'.
Sep 24 03:45:58 localhost systemd[1]: Failed to start systemd-network-generator.service - Generate network units from Kernel command line.

systemd-networkd.service logs:

Sep 24 03:46:00 localhost systemd[1]: Starting systemd-networkd.service - Network Configuration...
Sep 24 03:46:00 localhost systemd-networkd[818]: /etc/systemd/network/10-dracut-cmdline-99.network:8: Invalid section header '[fd00'
Sep 24 03:46:00 localhost systemd-networkd[818]: /etc/systemd/network/10-dracut-cmdline-99.network:8: Failed to parse file: Bad message
Sep 24 03:46:00 localhost systemd-networkd[818]: lo: Link UP
Sep 24 03:46:00 localhost systemd-networkd[818]: lo: Gained carrier
Sep 24 03:46:00 localhost systemd-networkd[818]: Enumeration completed
Sep 24 03:46:00 localhost systemd[1]: Started systemd-networkd.service - Network Configuration.
Sep 24 03:46:00 localhost systemd-networkd[818]: eth0: found matching network '/usr/lib/systemd/network/zz-default.network', based on potentially unpredictable interface name.
Sep 24 03:46:00 localhost systemd-networkd[818]: eth0: Configuring with /usr/lib/systemd/network/zz-default.network.
Sep 24 03:46:00 localhost systemd-networkd[818]: eth0: Link UP
Sep 24 03:46:00 localhost systemd-networkd[818]: eth0: Gained carrier
Sep 24 03:46:00 localhost systemd-networkd[818]: eth0: found matching network '/usr/lib/systemd/network/zz-default.network', based on potentially unpredictable interface name.
Sep 24 03:46:01 localhost systemd-networkd[818]: eth0: Gained IPv6LL
Sep 24 03:47:03 localhost systemd-networkd[818]: lo: Link DOWN
Sep 24 03:47:03 localhost systemd-networkd[818]: lo: Lost carrier
Sep 24 03:47:03 localhost systemd-networkd[818]: eth0: Link DOWN
Sep 24 03:47:03 localhost systemd-networkd[818]: eth0: Lost carrier

Mangled networkd unit generated to /etc/systemd/network/10-dracut-cmdline-99.network:

[Match]
Name=eth0
[Link]
[Network]
DHCP=no
Gateway=100.76.3.254
DNS=172.18.254.200
[fd00
DNS=c0db
[Address]
Address=100.76.3.2/24
[DHCP]
Hostname=test-flatcar-ds

Target networking configured through Ignition using storage.files for /etc/systemd/network/00-net0.network:

[Match]
MACAddress=BC:24:11:BA:53:11

[Link]
MTUBytes=9000

[Network]
Address=100.76.3.2/24
Address=fd00:c0db:2::1/64
Gateway=100.76.3.254
Gateway=fd00:c0db:2:0:ffff:ffff:ffff:ffff
DNS=172.18.254.200
DNS=172.18.253.200

Note that using the same ip= arguments with Fedora CoreOS works without issue and leads to a functioning dual-stack configuration. If I understand correctly their initrd implementation, Dracut-style early networking is handled by Afterburner at the moment instead of Dracut itself, as I could not find the network-legacy script in their Dracut modules.

jqueuniet avatar Sep 24 '25 04:09 jqueuniet

I might be wrong, but reading the Dracut script it just does not seem to have been planned for dual-stack support, as it would imply an ability to collapse multiple ip= arguments in a single network unit that seem to be missing here. Would a rework MR to add such a feature be acceptable, or is dual-stack deemed unnecessary for early networking and I should just work it out using single-stack networking?

jqueuniet avatar Sep 24 '25 09:09 jqueuniet

Thank you for the detailed report! Right, as always I think IPv6 wasn't really considered when this code was written. Adding support for dual stack now would be really nice, and if you want, you can start the work now and ask for help when needed. Questions are good here and in the Flatcar Matrix channel, too.

pothos avatar Sep 24 '25 12:09 pothos

I have a few questions actually now that I've started diving into it.

  • First, it's unlikely I'll be able to add dual-stack without heavily rewriting this script as a lot of things seem to break unexpectedly and the very act of merging ip arguments will lead to separating the parsing and writing loops, rather than doing everything in one go. Also, splitting everything in proper functions will make it much easier to debug and later maintain. Is that still ok with you?
  • I've noticed IPv6 RA autoconfiguration returns an error even though systemd-networkd supports it. Can I try to add support for it on the way, or should I keep the feature set and overall behaviour as close as possible?
  • Regarding systemd-network-generator, reading the doc it seems it should basically do what this script does. Is there a reason not to use it instead of a bash script where doing things such as netmask calculations and IP validation can be rather involved affairs? Like, is it too limited at the moment, not taking into accounts additional parameters than ip, lack of integration with other functions, etc?

jqueuniet avatar Sep 26 '25 16:09 jqueuniet

Using systemd-network-generator instead sounds good! I'm not sure if this directly works or needs a bit of preprocessing if there are slight differences (one could then invoke it with a custom path for the kernel cmdline).

pothos avatar Sep 29 '25 01:09 pothos

Initial investigation shows that systemd-network-generator does not work with IPv6 either.

With systemd 256.8 from Flatcar images, I get an error message mentioning a netmask issue:

# SYSTEMD_LOG_LEVEL=debug /usr/lib/systemd/systemd-network-generator
Found cgroup2 on /sys/fs/cgroup/, full unified hierarchy
Found container virtualization none.
Invalid MTU '172.18.254.200' for 'eth0': Invalid argument
IPv6 prefix length is not supported yet
Failed to parse kernel command line: Operation not supported

With systemd 258.0 from Debian sid, I get a different error message:

SYSTEMD_LOG_LEVEL=debug /usr/lib/systemd/systemd-network-generator --root=tmp/systemd-network-generator/ -- ip=100.76.3.2::100.76.3.254:255.255.255.0:test-flatcar-ds:eth0:none:172.18.254.200 ip=[fd00:c0db:2::1]
::[fd00:c0db:2:0:ffff:ffff:ffff:ffff]:[ffff:ffff:ffff:ffff::]:test-flatcar-ds:eth0:none:172.18.254.200
Invalid MTU '172.18.254.200' for 'eth0': Invalid argument
Failed to parse netmask in ip=[fd00:c0db:2::1]::[fd00:c0db:2:0:ffff:ffff:ffff:ffff]:[ffff:ffff:ffff:ffff::]:test-flatcar-ds:eth0:none:172.18.254.200: Invalid argument
Failed to parse command line "ip=[fd00:c0db:2::1]::[fd00:c0db:2:0:ffff:ffff:ffff:ffff]:[ffff:ffff:ffff:ffff::]:test-flatcar-ds:eth0:none:172.18.254.200": Invalid argument

It does work with multiple ip arguments when using only IPv4 though, units get written to /run/systemd/network/.

SYSTEMD_LOG_LEVEL=debug /usr/lib/systemd/systemd-network-generator --root=tmp/systemd-network-generator/ -- ip=100.76.3.2::100.76.3.254:255.255.255.0:test-flatcar-ds:net0:none:172.18.254.200 ip=100.76.4.2::100.76.4.254:255.255.255.0:test-flatcar-ds:net1:none:172.18.254.200
Invalid MTU '172.18.254.200' for 'net0': Invalid argument
Invalid MTU '172.18.254.200' for 'net1': Invalid argument
No credentials found.

ls -l tmp/systemd-network-generator/run/systemd/network/
total 8
-rw-r--r-- 1 jqueuniet jqueuniet 255 Sep 29 13:04 70-net0.network
-rw-r--r-- 1 jqueuniet jqueuniet 255 Sep 29 13:04 70-net1.network

# Automatically generated by systemd-network-generator

[Match]
Name=net0

[Link]

[Network]
DHCP=no
LinkLocalAddressing=no
IPv6AcceptRA=no
DNS=172.18.254.200

[DHCP]
Hostname=test-flatcar-ds

[Address]
Address=100.76.3.2/24

[Route]
Gateway=100.76.3.254

The MTU and credential messages are from the debug output and do not lead to actual errors.

jqueuniet avatar Sep 29 '25 11:09 jqueuniet

I did end up finding a syntax that works with IPv6, using a CIDR prefix number instead of an actual netmask:

With systemd 258.0:

SYSTEMD_LOG_LEVEL=debug /usr/lib/systemd/systemd-network-generator --root=tmp/systemd-network-generator/ -- ip=100.76.3.2::100.76.3.254:255.255.255.0:test-flatcar-ds:eth0:none:172.18.254.200 ip=[fd00:c0db:2::1]::[fd00:c0db:2:0:ffff:ffff:ffff:ffff]:64:test-flatcar-ds:eth0:none:172.18.254.200
Invalid MTU '172.18.254.200' for 'eth0': Invalid argument
Invalid MTU '172.18.254.200' for 'eth0': Invalid argument
No credentials found.

# Automatically generated by systemd-network-generator

[Match]
Name=eth0

[Link]

[Network]
DHCP=no
LinkLocalAddressing=no
IPv6AcceptRA=no
DNS=172.18.254.200
DNS=172.18.254.200

[DHCP]
Hostname=test-flatcar-ds

[Address]
Address=fd00:c0db:2::1/64

[Address]
Address=100.76.3.2/24

[Route]
Gateway=fd00:c0db:2:0:ffff:ffff:ffff:ffff

[Route]
Gateway=100.76.3.254

With systemd 256.8:

SYSTEMD_LOG_LEVEL=debug /usr/lib/systemd/systemd-network-generator -- ip=100.76.3.2::100.76.3.254:255.255.255.0:test-flatcar-ds:eth0:none:172.18.254.200 ip=[fd00:c0db:2::1]::[fd00:c0db:2:0:ffff:ffff:ffff:ffff]:64:test-flatcar-ds:eth0:none:172.18.254.200
Invalid MTU '172.18.254.200' for 'eth0': Invalid argument
Invalid IPv6 address '64:test-flatcar-ds:eth0:none:172.18.254.200'
Invalid MTU '172.18.254.200' for 'eth0': Invalid argument
No credentials found.

# Automatically generated by systemd-network-generator

[Match]
Name=eth0

[Link]

[Network]
DHCP=no
LinkLocalAddressing=no
IPv6AcceptRA=no
DNS=172.18.254.200
DNS=172.18.254.200

[DHCP]
Hostname=test-flatcar-ds

[Address]
Address=fd00:c0db:2::1/64

[Address]
Address=100.76.3.2/24

[Route]
Gateway=fd00:c0db:2:0:ffff:ffff:ffff:ffff

[Route]
Gateway=100.76.3.254

In both case, error messages are from the debug output and do not return actual errors.

jqueuniet avatar Sep 29 '25 11:09 jqueuniet

Thanks for looking into this! Sounds like we should drop the shell script (unless it's still needed for some other parts of the config).

pothos avatar Sep 29 '25 11:09 pothos

I modified a bit my automation to output CIDR prefixes for both IPv4 and IPv6, and Flatcar provisioning actually passed this time, I guess thanks to systemd-network-generator since I can see its output network unit file in /run/systemd/network. Once a reboot is done, it seems the unit I wrote with Ignition to /etc/systemd/network properly took over and the generator file is now ignored.

So yeah, it looks like Flatcar has two tools doing the same thing at the moment and one of them should probably go. After some time looking at the shell implementation, it seems it has a few additional features like taking additional parameters from /etc/cmdline and /etc/cmdline.d/*.conf through the getcmdline function from dracut-lib.sh but that's all I found that could be relevant. The could be additional things I did not notice though.

jqueuniet avatar Sep 29 '25 12:09 jqueuniet

Somehow, keeping both configurator active worked with standard cases and with community sysexts, but breaks when I try to use official sysexts. Then both configs are generated and systemd-networkd is back to raising errors on /etc/systemd/network/10-dracut-cmdline-99.network.

The solution I found was to mask the parse-ip-for-networkd.service unit by adding rd.systemd.mask=parse-ip-for-networkd.service to the cmdline. I'm now running into https://github.com/flatcar/Flatcar/issues/1520 since our network is proxified, but I guess I'll use the workaround mentioned over there.

jqueuniet avatar Sep 30 '25 16:09 jqueuniet