updating fails when sysext is enabled

[till]

Description

Tried to update a node from Flatcar 3510.2.2 to the latest stable and it failed. After some digging (thank you @t-lo, @jepio and @tormath1), it seems like enabling the python sysext (via enabled-sysext.conf) blocked the update process.

This is the journal when it failed:

Jun 26 17:26:08 node-001.node update_engine[883]: I0626 17:26:08.282311   883 subprocess.cc:43] Subprocess status: 1
Jun 26 17:26:08 node-001.node update_engine[883]: I0626 17:26:08.282341   883 subprocess.cc:46] Subprocess output:
Jun 26 17:26:08 node-001.node update_engine[883]: Args { output_dir: "/var/lib/update_engine/ue-rs/", target_filename: Some("oem-openstack.raw"), input_xml: Some("/var/lib/update_engine/prefs/full-response"), payload_url: None, pubkey_file: "/usr/share/update_engine/update-payload-key.pub.pem", image_match: ["*oem-openstack.gz"], take_first_match: true }
Jun 26 17:26:08 node-001.node update_engine[883]: writing to /var/lib/update_engine/ue-rs/.unverified/oem-openstack.gz
Jun 26 17:26:08 node-001.node update_engine[883]: Parsed and verified signature data from file "/var/lib/update_engine/ue-rs/.unverified/oem-openstack.gz"
Jun 26 17:26:08 node-001.node update_engine[883]: Trying to place /var/lib/update_engine/oem-openstack-4230.2.0.raw on OEM partition
Jun 26 17:26:08 node-001.node update_engine[883]: grep: /usr/share/flatcar/enabled-sysext.conf: No such file or directory
Jun 26 17:26:08 node-001.node update_engine[883]: Args { output_dir: "/var/lib/update_engine/ue-rs/", target_filename: Some("flatcar-python.raw"), input_xml: Some("/var/lib/update_engine/prefs/full-response"), payload_url: None, pubkey_file: "/usr/share/update_engine/update-payload-key.pub.pem", image_match: ["*flatcar-python.gz"], take_first_match: true }
Jun 26 17:26:08 node-001.node update_engine[883]: writing to /var/lib/update_engine/ue-rs/.unverified/flatcar-python.gz
Jun 26 17:26:08 node-001.node update_engine[883]: Parsed and verified signature data from file "/var/lib/update_engine/ue-rs/.unverified/flatcar-python.gz"
Jun 26 17:26:08 node-001.node update_engine[883]: mv: cannot move '/var/lib/update_engine/flatcar-python.raw' to '/etc/flatcar/sysext/flatcar-python-4230.2.0.raw': No such file or directory
Jun 26 17:26:08 node-001.node update_engine[883]: E0626 17:26:08.282346   883 postinstall_runner_action.cc:94] Postinst command failed with code: 1
Jun 26 17:26:08 node-001.node update_engine[883]: I0626 17:26:08.287457   883 action_processor.cc:68] ActionProcessor::ActionComplete: PostinstallRunnerAction action failed. Aborting processing.
Jun 26 17:26:08 node-001.node update_engine[883]: I0626 17:26:08.287475   883 action_processor.cc:73] ActionProcessor::ActionComplete: finished last action of type PostinstallRunnerAction
Jun 26 17:26:08 node-001.node update_engine[883]: I0626 17:26:08.287478   883 update_attempter.cc:302] Processing Done.
Jun 26 17:26:08 node-001.node update_engine[883]: E0626 17:26:08.287561   883 update_attempter.cc:615] Update failed.

Impact

The node is not updated.

Environment and steps to reproduce

• install e.g. 3510.2.2 (openstack)
• echo "python" > /etc/flatcar/enabled-sysext.conf
• update_engine_client -update

Expected behavior

Node is updated.

Additional information

It would be really nice, if the log/output from update_engine_client or flatcar-update elevated these problems. One previous problem was that I had changed the name of the sysext to flatcar-python (per current docs), but that also blocked the update because it is indeed called python.

[tormath1]

Thanks for the report and sorry for the issue here. So the main reason why it fails is that /etc/flatcar/sysext does not exist yet on those old Flatcar versions (3510.2.0.2) (created here during boot: https://github.com/flatcar/bootengine/pull/89) - manually creating the folder will fix the update.

I am wondering, why do you have at this point in time a file /etc/flatcar/enabled-sysext.conf on your instance? This feature only appears with the first sysext ZFS (or maybe a bit before) but not from 3510.2.0.2.

[till]

We import new flatcar releases on a irregular basis.

I had already updated butane to create the file as we are transitioning to the python included in Flatcar.

So you're right, the mix up is my picnic I just didn't expect this to block the entire process?

Maybe this is more an issue for (the new) update engine to elevate these problems?

[pothos]

Thanks for the report and the debugging, I've opened https://github.com/flatcar/update_engine/pull/50 to fix this

[pothos]

This will be part of the next Alpha but I didn't do backports, so it will take some time until nodes on Stable will update again if you didn't work around it already.

[till]

This will be part of the next Alpha but I didn't do backports, so it will take some time until nodes on Stable will update again if you didn't work around it already.

Roughly how long?

[pothos]

A couple of month :D We could create backport branches for a cherry pick if really needed, but would creating the folder in your Ignition config and for existing nodes, manually, also work for you?

[till]

Not a problem, where do I look?