Newest Ubuntu could images have a networking issue

[coesensbert]

Recent LTS cloud image builds for Ubuntu seem to fail setting up networking on the grid. None of the provided IP's (wg, ygg or pub) respond to any request or ping.

https://hub.grid.tf/linkmark.3bot/ubuntu-22.04-LTS-amd64-20220902.flist

[  OK  ] Finished Set console font and keymap.M
[  OK  ] Finished Create final runt…e dir for shutdown pivot root.M
[  OK  ] Finished Tell Plymouth To Write Out Runtime Data.M
[  OK  ] Finished Uncomplicated firewall.M
[  OK  ] Finished Create Volatile Files and Directories.M
         Starting Network Time Synchronization...M
         Starting Record System Boot/Shutdown in UTMP...M
[  OK  ] Finished Record System Boot/Shutdown in UTMP.M
[  OK  ] Finished Commit a transient machine-id on disk.M
[  OK  ] Started Network Time Synchronization.M
[  OK  ] Reached target System Time Set.M
[  OK  ] Finished Load AppArmor profiles.M
         Starting Load AppArmor pro…managed internally by snapd...M
         Starting Initial cloud-init job (pre-networking)...M
[    6.110483] cloud-init[557]: Cloud-init v. 22.2-0ubuntu1~22.04.3 running 'init-local' at Wed, 21 Sep 2022 15:20:56 +0000. Up 6.07 seconds.M
[    6.309820] cloud-init[557]: 2022-09-21 15:20:56,288 - util.py[WARNING]: failed stage init-localM
[    6.315841] cloud-init[557]: failed run of stage init-localM
[    6.316690] cloud-init[557]: ------------------------------------------------------------M
[    6.317924] cloud-init[557]: Traceback (most recent call last):M
[    6.318785] cloud-init[557]:   File "/usr/lib/python3/dist-packages/cloudinit/cmd/main.py", line 740, in status_wrapperM
[    6.320282] cloud-init[557]:     ret = functor(name, args)M
[    6.321130] cloud-init[557]:   File "/usr/lib/python3/dist-packages/cloudinit/cmd/main.py", line 410, in main_initM
[    6.322456] cloud-init[557]:     init.apply_network_config(bring_up=bring_up_interfaces)M
[    6.323498] cloud-init[557]:   File "/usr/lib/python3/dist-packages/cloudinit/stages.py", line 937, in apply_network_configM
[    6.324938] cloud-init[557]:     return self.distro.apply_network_config(M
[    6.325869] cloud-init[557]:   File "/usr/lib/python3/dist-packages/cloudinit/distros/__init__.py", line 231, in apply_network_configM
[    6.330152] cloud-init[557]:     network_state = parse_net_config_data(netconfig)M
[    6.331192] cloud-init[557]:   File "/usr/lib/python3/dist-packages/cloudinit/net/network_state.py", line 1056, in parse_net_config_dataM
[    6.332906] cloud-init[557]:     nsi.parse_config(skip_broken=skip_broken)M
[    6.333898] cloud-init[557]:   File "/usr/lib/python3/dist-packages/cloudinit/net/network_state.py", line 278, in parse_configM
[    6.335521] cloud-init[557]:     self.parse_config_v2(skip_broken=skip_broken)M
[    6.336465] cloud-init[557]:   File "/usr/lib/python3/dist-packages/cloudinit/net/network_state.py", line 328, in parse_config_v2M
[    6.341985] cloud-init[557]:     self._v2_common(command)M
[    6.342745] cloud-init[557]:   File "/usr/lib/python3/dist-packages/cloudinit/net/network_state.py", line 782, in _v2_commonM
[    6.344063] cloud-init[557]:     self._handle_individual_nameserver(name_cmd, iface)M
[    6.345049] cloud-init[557]:   File "/usr/lib/python3/dist-packages/cloudinit/net/network_state.py", line 110, in decoratorM
[    6.346297] cloud-init[557]:     return func(self, command, *args, **kwargs)M
[    6.347161] cloud-init[557]:   File "/usr/lib/python3/dist-packages/cloudinit/net/network_state.py", line 570, in _handle_individual_nameserverM
[    6.348619] cloud-init[557]:     _iface[iface]["dns"] = {"nameservers": nameservers, "search": search}M
[    6.349633] cloud-init[557]: KeyError: 'eth0'M
[    6.350282] cloud-init[557]: ------------------------------------------------------------M
[FAILED] Failed to start Initial cloud-init job (pre-networking).M
See 'systemctl status cloud-init-local.service' for details.M
[  OK  ] Reached target Preparation for Network.M
         Starting Network Configuration...M
[  OK  ] Started Network Configuration.M
         Starting Wait for Network to be Configured...M
         Starting Network Name Resolution...M
[  OK  ] Finished Wait for Network to be Configured.M
         Starting Initial cloud-ini… (metadata service crawler)...M
[  OK  ] Started Network Name Resolution.M
[  OK  ] Reached target Network.M
[  OK  ] Reached target Host and Network Name Lookups.M
[  OK  ] Finished Load AppArmor pro…s managed internally by snapd.M
[    6.706267] cloud-init[580]: Cloud-init v. 22.2-0ubuntu1~22.04.3 running 'init' at Wed, 21 Sep 2022 15:20:56 +0000. Up 6.68 seconds.M

https://hub.grid.tf/linkmark.3bot/ubuntu-20.04-LTS-amd64-20220920.flist

[  OK  ] Finished Create final runt…e dir for shutdown pivot root.M
[  OK  ] Finished Set console font and keymap.M
[  OK  ] Finished Tell Plymouth To Write Out Runtime Data.M
[  OK  ] Finished Commit a transient machine-id on disk.M
[  OK  ] Finished Create Volatile Files and Directories.M
         Starting Network Time Synchronization...M
         Starting Update UTMP about System Boot/Shutdown...M
[  OK  ] Finished Update UTMP about System Boot/Shutdown.M
[  OK  ] Started Network Time Synchronization.M
[  OK  ] Reached target System Time Set.M
[  OK  ] Reached target System Time Synchronized.M
[  OK  ] Finished Load AppArmor profiles.M
         Starting Load AppArmor pro…managed internally by snapd...M
         Starting Initial cloud-init job (pre-networking)...M
[  OK  ] Finished Load AppArmor pro…s managed internally by snapd.M
[    5.894008] cloud-init[581]: Cloud-init v. 22.2-0ubuntu1~20.04.3 running 'init-local' at Wed, 21 Sep 2022 15:28:35 +0000. Up 5.69 seconds.M
[    5.896143] cloud-init[581]: 2022-09-21 15:28:35,359 - util.py[WARNING]: failed stage init-localM
[    5.903913] cloud-init[581]: failed run of stage init-localM
[    5.905422] cloud-init[581]: ------------------------------------------------------------M
[    5.906471] cloud-init[581]: Traceback (most recent call last):M
[    5.907310] cloud-init[581]:   File "/usr/lib/python3/dist-packages/cloudinit/cmd/main.py", line 740, in status_wrapperM
[    5.909029] cloud-init[581]:     ret = functor(name, args)M
[    5.909960] cloud-init[581]:   File "/usr/lib/python3/dist-packages/cloudinit/cmd/main.py", line 410, in main_initM
[    5.911394] cloud-init[581]:     init.apply_network_config(bring_up=bring_up_interfaces)M
[    5.912500] cloud-init[581]:   File "/usr/lib/python3/dist-packages/cloudinit/stages.py", line 937, in apply_network_configM
[    5.914275] cloud-init[581]:     return self.distro.apply_network_config(M
[    5.915263] cloud-init[581]:   File "/usr/lib/python3/dist-packages/cloudinit/distros/__init__.py", line 231, in apply_network_configM
[    5.917240] cloud-init[581]:     network_state = parse_net_config_data(netconfig)M
[    5.918276] cloud-init[581]:   File "/usr/lib/python3/dist-packages/cloudinit/net/network_state.py", line 1056, in parse_net_config_dataM
[    5.920264] cloud-init[581]:     nsi.parse_config(skip_broken=skip_broken)M
[    5.921246] cloud-init[581]:   File "/usr/lib/python3/dist-packages/cloudinit/net/network_state.py", line 278, in parse_configM
[    5.923113] cloud-init[581]:     self.parse_config_v2(skip_broken=skip_broken)M
[    5.924145] cloud-init[581]:   File "/usr/lib/python3/dist-packages/cloudinit/net/network_state.py", line 328, in parse_config_v2M
[    5.926022] cloud-init[581]:     self._v2_common(command)M
[    5.926872] cloud-init[581]:   File "/usr/lib/python3/dist-packages/cloudinit/net/network_state.py", line 782, in _v2_commonM
[    5.928616] cloud-init[581]:     self._handle_individual_nameserver(name_cmd, iface)M
[    5.929845] cloud-init[581]:   File "/usr/lib/python3/dist-packages/cloudinit/net/network_state.py", line 110, in decoratorM
[    5.931531] cloud-init[581]:     return func(self, command, *args, **kwargs)M
[    5.932559] cloud-init[581]:   File "/usr/lib/python3/dist-packages/cloudinit/net/network_state.py", line 570, in _handle_individual_nameserverM
[    5.934621] cloud-init[581]:     _iface[iface]["dns"] = {"nameservers": nameservers, "search": search}M
[    5.936065] cloud-init[581]: KeyError: 'eth0'M
[    5.936845] cloud-init[581]: ------------------------------------------------------------M
[FAILED] Failed to start Initial cloud-init job (pre-networking).M
See 'systemctl status cloud-init-local.service' for details.M
[  OK  ] Reached target Network (Pre).M
         Starting Network Service...M
[  OK  ] Started Network Service.M
         Starting Wait for Network to be Configured...M
         Starting Network Name Resolution...M
[  OK  ] Finished Wait for Network to be Configured.M
         Starting Initial cloud-ini… (metadata service crawler)...M
[  OK  ] Started Network Name Resolution.M
[  OK  ] Reached target Network.M
[  OK  ] Reached target Host and Network Name Lookups.M
[    6.651712] cloud-init[598]: Cloud-init v. 22.2-0ubuntu1~20.04.3 running 'init' at Wed, 21 Sep 2022 15:28:35 +0000. Up 6.42 seconds.M

These older images have the same issue:

• https://hub.grid.tf/linkmark.3bot/ubuntu-20-04-LTS-20220824.flist
• https://hub.grid.tf/linkmark.3bot/qubuntu-20-04-LTS-20220913.flist

[scottyeager]

Looks like an instance of this: https://bugs.launchpad.net/cloud-init/+bug/1979877

There's a fix included in cloud-init 22.3, but these Ubuntu images still ship with 22.2. Author of the issue above mentions a workaround:

As a workaround, you can give an explicit set-name statement, or change the key of the network config to match the physical device name.

Based on my read, our specific issue is that Zos is creating a cloud-init config with devices named eth0, eth1, ..., but these don't correspond to the actual device names given by systemd's "Predictable Interface Names".

I spun up a full Ubuntu VM:

# ip link
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN mode DEFAULT group default qlen 1000
    link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
2: ens3: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc fq_codel state UP mode DEFAULT group default qlen 1000
    link/ether ee:d5:aa:bd:6f:5b brd ff:ff:ff:ff:ff:ff
3: ens4: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc fq_codel state UP mode DEFAULT group default qlen 1000
    link/ether 82:37:7b:8c:17:fb brd ff:ff:ff:ff:ff:ff

Options:

1. Adding set-name and assigning to the Zos nic.ID
2. Predict the ens... names and use this as Name for cloud-init
3. Wait for Ubuntu images to include an updated cloud-init

[scottyeager]

The release status for cloud-init in Ubuntu can be seen here. Kinetic (22.10) is the only release to ship with cloud-init 22.3 currently. I tested the latest cloud image of 22.10 and it deploys without issue.

Another workaround for the earlier releases in the meantime is to activate the proposed archive and update cloud-init before uploading the image to the hub.

[muhamadazmy]

I tried the latest build of ubuntu 22.04 LTS and works fine. I have linked it in the official-vms. seems the older version had a broken cloud-init built