lxd icon indicating copy to clipboard operation
lxd copied to clipboard

Check whether the Ceph RBD driver needs to have `RunningCopyFreeze=true` for block volumes

Open tomponline opened this issue 1 year ago • 0 comments

When the ceph RBD driver was added in https://github.com/canonical/lxd/commit/0a67c1fec52cc0f5aba26ff2fa0e567f3b0fa1d2 at the time the RunningSnapshotFreeze setting (later renamed to RunningCopyFreeze) was set to true without any comment as to why this was.

However there is some discussion around this where lvm had RunningCopyFreeze set to true in https://github.com/canonical/lxd/pull/9833 here:

https://github.com/canonical/lxd/issues/9789#issuecomment-1017944051 https://github.com/canonical/lxd/issues/9706#issuecomment-995595000

The decision for ceph and lvm seems to be related to the inability to take a temporary snapshot of the volume as a source for copying because for filesystem volumes using xfs and btrfs, they cannot be mounted at the same time as the main volume:

https://github.com/canonical/lxd/issues/9706#issuecomment-996020622

For the LVM and Ceph cases, snapshots are problematic as xfs and btrfs will not allow concurrent mounting of a snapshot and original volume unless their identifiers are changed. That rewrite operation, at least on btrfs is very costly. There's also the issue that we would take a snapshot while the instance is running, so writes wouldn't be flushed and the filesystem may be dirty.

But given that all block volumes are not crash consistent anyway (because we cannot ask the VM's OS to flush their writes currently) then it begs the question as to whether we need to freeze the VM when copying/snapshotting the block volumes when attached to a running the VM.

The answer is complicated because the answer depends on the volume type (and also for VMs there is the filesystem config volume to think about) and whether the Ceph RBD driver is using an optimized transfer mode with temporary snapshot or whether its using fallback rsync/dd mode.

tomponline avatar Mar 19 '24 10:03 tomponline