Split generate/configure stages for sd-generator compliance

[slyon]

Description

Refactor Netplan's generate binary to be a proper systemd.generator, according to spec "FO165 – Netplan generator architecture".

New testing of a simulated systemd sandbox is implemented in tests/generator/test_sd-generator.py.

• Logic that writes systemd unit files (e.g. .service units or override.conf drop-in configs) is now generated in src/gen-*.c and run during daemon-reload.
• Writing of normal systemd-networkd/NetworkManager/wpa_supplicant/udev/Open vSwitch/SR-IOV config files remains in src/{networkd,nm,sriov,openvswitch}.c.
• New tests added, utilizing a systemd-run sandbox (TestSystemdGenerator.test_sandbox), that make sure the usr/libexec/netplan/generate binary does not write files outside the allowed scope for a systemd-generator.
• Adding a new --networkmanager-only parameter, to improve NM integration, by not touching any systemd[-networkd] files
• Introducing new _netplan_state_get/set_flags() API (internal, for now) to do "validation-only" runs over our NetDef data

Checklist

• [x] Runs make check successfully.
• [x] Retains code coverage (make check-coverage).
• [ ] New/changed keys in YAML format are documented.
• [ ] (Optional) Adds example YAML for new feature.
• [x] (Optional) Closes an open bug in Launchpad. LP#2071747

This should also fix https://bugs.launchpad.net/ubuntu/+source/network-manager/+bug/2083129 by not re-generating NM configuration on the fly. And https://bugs.launchpad.net/ubuntu/+source/netplan.io/+bug/2090848 by using the new /usr/libexec/netplan/configure --networkmanager-only flag.

[slyon]

Packaging TODOs

• re-generate config in .postinst using usr/libexec/netplan/configure binary as the (new) sd-generator does not touch any systemd-networkd configuration (or its permission).

  Setting up libnetplan1:amd64 (1.1.2-7+DEV) ...
  Setting up python3-netplan (1.1.2-7+DEV) ...
  Setting up netplan-generator (1.1.2-7+DEV) ...
  Removing 'diversion of /lib/systemd/system-generators/netplan to /lib/systemd/system-generators/netplan.usr-is-merged by netplan-generator'
  Bail out! ERROR:../src/gen-openvswitch.c:533:_netplan_state_finish_sd_ovs_write: assertion failed: (generator_dir != NULL)
  WARNING: Netplan could not re-generate network configuration. Please run 'netplan generate' to see details.
  Created symlink '/etc/systemd/system/sysinit.target.wants/netplan-configure.service' → '/usr/lib/systemd/system/netplan-configure.service'.
  Setting up netplan.io (1.1.2-7+DEV) ...
  

• Adopt NetworkManager capabilities, or avoid NM writing systemd-networkd configuration with root permissions, or rather call configure with the (new) --networkmanager-only parameter:

  /usr/libexec/netplan/configure --networkmanager-only
  
  printf "[Unit]\nCapabilityBoundingSet=CAP_CHOWN\n" | systemctl edit --stdin NetworkManager.service 2>/dev/null
  
  mkdir -p /etc/systemd/system/NetworkManager.service.d && echo "[Service]\nCapabilityBoundingSet=CAP_CHOWN\n" > /etc/systemd/system/NetworkManager.service.d/override.conf && systemctl daemon-reload
  

• Install systemd-resolved for cloud-init & autostart DEP-8 tests

[slyon]

The new CI failure in lxd-ubuntu-lts is unrelated to this PR. It's caused by a recent kernel update in Ubuntu, see LP: #2124257:

======================================================================
ERROR: test_tunnel_vxlan (__main__.TestNetworkd.test_tunnel_vxlan)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/tmp/autopkgtest.3y9ojK/build.aN1/real-tree/tests/integration/tunnels.py", line 281, in test_tunnel_vxlan
    json = self.iface_json('vx0')
           ^^^^^^^^^^^^^^^^^^^^^^
  File "/tmp/autopkgtest.3y9ojK/build.aN1/real-tree/tests/integration/base.py", line 326, in iface_json
    json_dict = json.loads(out)
                ^^^^^^^^^^^^^^^
  File "/usr/lib/python3.12/json/__init__.py", line 346, in loads
    return _default_decoder.decode(s)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/lib/python3.12/json/decoder.py", line 337, in decode
    obj, end = self.raw_decode(s, idx=_w(s, 0).end())
               ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/lib/python3.12/json/decoder.py", line 353, in raw_decode
    obj, end = self.scan_once(s, idx)
               ^^^^^^^^^^^^^^^^^^^^^^
json.decoder.JSONDecodeError: Expecting ',' delimiter: line 1 column 379 (char 378)

----------------------------------------------------------------------

[slyon]

rebased to resolve conflicts against https://github.com/canonical/netplan/pull/563