Storage: Add all custom storage vols to instance's backup config

[roosterfish]

Preceded by:

• https://github.com/canonical/lxd/pull/15775
• https://github.com/canonical/lxd/pull/15776
• https://github.com/canonical/lxd/pull/15777

Overview

This PR uses the new backup config metadata format introduced in https://github.com/canonical/lxd/pull/15129 to track all of an instance's custom storage volumes. In addition all of the storage volume's storage pools are now listed too.

Whenever a custom volume (or its snapshots) gets updated, LXD has to check whether or not this custom volume is attached to any instance. If it's attached, with the changes proposed in this PR, LXD rewrites the backup config file to propagate the latest state of the volume.

Detail

In order to pick up an instance's custom volume changes, the outcome of such a change has to be persisted in the parent instance's backup config. LXD currently only has support for GenerateCustomVolumeBackupConfig which produces a backup config file (using the standard format) to be included in custom volume exports.

This PR grows a new UpdateCustomVolumeBackupFile (same as UpdateInstanceBackupFile) which allows updates of a custom volume to be persisted to file. With the only difference that those updates are written to all the instances using the custom volume.

As multiple instances can use the same volume, looking up all the necessary information to rebuild an instance's backup config might be resource intensive. Therefore the instance grows another GenerateInstanceCustomVolumeBackupConfig function which allows generating the backup config for the instance's custom volumes by leveraging caching mechanisms. Also an already prepared backup config can now be passed down to GenerateInstanceBackupConfig and UpdateInstanceBackupFile if its already crafted from the cache.

Having the GenerateInstanceCustomVolumeBackupConfig exported on the Pool interface allows arbitrary callers to either generate the custom volume's config to be passed when creating an instance's backup config or simply provide nothing. This is helpful in the case of exporting instances as the received backup doesn't contain any information about custom volumes attached to the respective instance.

Fixes https://github.com/canonical/lxd/issues/11712

[tomponline]

Furthermore a general bug is fixed in LXD which caused an instance's backup file not to be updated when updating the storage pool of its root volume. This action is now performed for both an instance's root vol as well as all of its attached custom vols.

Fixes #11712

Please can we separate that into another pr ?

[roosterfish]

Once https://github.com/canonical/lxd/pull/15777 is merged I'll rebase this one and then it should be ready for another pass.

[roosterfish]

Implemented the suggested changes, ready for another review. The PR's description now also contains a summary of the changes including a more in depth explanation of the changes.

[roosterfish]

Can the "Update instance backup file if custom volume..." commits at the end of the PR be split off from this one?

Those are the actual commits that trigger the update of the instance's backup config in case any of their custom vols gets updated.

[roosterfish]

@tomponline ready for another pass.

[roosterfish]

Added one more commit to check explicitly the behavior of not having duplicate vols in the backup config in case the same vol is attached multiple times using different devices to the same instance.