lxd icon indicating copy to clipboard operation
lxd copied to clipboard
verified publisher icon canonical

Permissions handling when moving storage volumes is broken

Open markylaing opened this issue 7 months ago • 1 comments

Please confirm

  • [x] I have searched existing issues to check if an issue already exists for the bug I encountered.

Distribution

Ubuntu

Distribution version

24.04

Output of "snap list --all lxd core20 core22 core24 snapd"

# snap list
Name    Version      Rev    Tracking       Publisher   Notes
core24  20250318     888    latest/stable  canonical✓  base
lxd     git-406f20a  33649  latest/edge    canonical✓  -
snapd   2.68.4       24505  latest/stable  canonical✓  snapd

Output of "lxc info" or system info if it fails

config:
  cluster.https_address: 10.166.156.73:8443
  core.https_address: 10.166.156.73:8443
api_extensions:
# Many many more here is the latest one.
- clustering_restore_skip_mode
api_status: stable
api_version: "1.0"
auth: trusted
public: false
auth_methods:
- tls
client_certificate: false
auth_user_name: root
auth_user_method: unix
environment:
  addresses:
  - 10.166.156.73:8443
  architectures:
  - x86_64
  - i686
  backup_metadata_version_range:
  - 1
  - 2
  certificate: |
    -----BEGIN CERTIFICATE-----
    MIIB6jCCAXCgAwIBAgIRAIghNyBGltXFmeZS36s6BYkwCgYIKoZIzj0EAwMwJjEM
    MAoGA1UEChMDTFhEMRYwFAYDVQQDDA1yb290QG1lbWJlcjAxMB4XDTI1MDUwODA5
    MzMyMloXDTM1MDUwNjA5MzMyMlowJjEMMAoGA1UEChMDTFhEMRYwFAYDVQQDDA1y
    b290QG1lbWJlcjAxMHYwEAYHKoZIzj0CAQYFK4EEACIDYgAECqsaORwelQW+4c/d
    SB+2dJR3Jjzc0YsKBPfR3eiLeJmdnrq0Hq/4tBVvhmnWQ1zejFNtvCfqZin7ix7z
    mNflhIro/FaHOOwIN7TAp/WKAfFc0l1puyq2Ninp6coVwo/eo2IwYDAOBgNVHQ8B
    Af8EBAMCBaAwEwYDVR0lBAwwCgYIKwYBBQUHAwEwDAYDVR0TAQH/BAIwADArBgNV
    HREEJDAigghtZW1iZXIwMYcEfwAAAYcQAAAAAAAAAAAAAAAAAAAAATAKBggqhkjO
    PQQDAwNoADBlAjBjgxFKKzN/cyufz6BNJBKsyxVi35wwUcdlA+6hMdGFgeaN5bnV
    I7acuYS/Lvp3AjoCMQCMV5oATder5/R7AlDPz5N+BSm+04qlI7OhCiMuJNr9Tz0O
    S5bA1FeAupVw4mHhHv8=
    -----END CERTIFICATE-----
  certificate_fingerprint: 78c9226739cf8b1b1cede8a8e4fe720708bb015dd05f1d584d2532951e19db3e
  driver: lxc
  driver_version: 6.0.0
  instance_types:
  - container
  firewall: nftables
  kernel: Linux
  kernel_architecture: x86_64
  kernel_features:
    bpf_token: "false"
    idmapped_mounts: "true"
    netnsid_getifaddrs: "true"
    seccomp_listener: "true"
    seccomp_listener_continue: "true"
    uevent_injection: "true"
    unpriv_binfmt: "true"
    unpriv_fscaps: "true"
  kernel_version: 6.8.0-59-generic
  lxc_features:
    cgroup2: "true"
    core_scheduling: "true"
    devpts_fd: "true"
    idmapped_mounts_v2: "true"
    mount_injection_file: "true"
    network_gateway_device_route: "true"
    network_ipvlan: "true"
    network_l2proxy: "true"
    network_phys_macvlan_mtu: "true"
    network_veth_router: "true"
    pidfd: "true"
    seccomp_allow_deny_syntax: "true"
    seccomp_notify: "true"
    seccomp_proxy_send_notify_fd: "true"
  os_name: Ubuntu
  os_version: "24.04"
  project: default
  server: lxd
  server_clustered: true
  server_event_mode: full-mesh
  server_name: member01
  server_pid: 1732
  server_version: "6.3"
  server_lts: false
  storage: dir
  storage_version: "1"
  storage_supported_drivers:
  - name: ceph
    version: 19.2.0
    remote: true
  - name: cephobject
    version: 19.2.0
    remote: true
  - name: dir
    version: "1"
    remote: false
  - name: powerflex
    version: 2.8 (nvme-cli)
    remote: true
  - name: pure
    version: 2.1.9 (iscsiadm) / 2.8 (nvme-cli)
    remote: true
  - name: btrfs
    version: 6.6.3
    remote: false
  - name: cephfs
    version: 19.2.0
    remote: true
  - name: lvm
    version: 2.03.16(2) (2022-05-18) / 1.02.185 (2022-05-18)
    remote: false
  - name: zfs
    version: 2.2.2-0ubuntu9.1
    remote: false

Issue description

Trying to move a storage volume between cluster members with minimum permissions.

Steps to reproduce

  1. Set up a LXD cluster:
root@member01:~# lxc cluster list
+----------+-----------------------------+-----------------+--------------+----------------+-------------+--------+-------------------+
|   NAME   |             URL             |      ROLES      | ARCHITECTURE | FAILURE DOMAIN | DESCRIPTION | STATE  |      MESSAGE      |
+----------+-----------------------------+-----------------+--------------+----------------+-------------+--------+-------------------+
| member01 | https://10.166.156.73:8443  | database-leader | x86_64       | default        |             | ONLINE | Fully operational |
|          |                             | database        |              |                |             |        |                   |
+----------+-----------------------------+-----------------+--------------+----------------+-------------+--------+-------------------+
| member02 | https://10.166.156.140:8443 | database        | x86_64       | default        |             | ONLINE | Fully operational |
+----------+-----------------------------+-----------------+--------------+----------------+-------------+--------+-------------------+
| member03 | https://10.166.156.238:8443 | database        | x86_64       | default        |             | ONLINE | Fully operational |
+----------+-----------------------------+-----------------+--------------+----------------+-------------+--------+-------------------+
  1. Create a storage volume
root@member01:~# lxc storage volume create default vol1
Storage volume vol1 created
root@member01:~# lxc storage volume list default
+--------+------+-------------+--------------+---------+----------+
|  TYPE  | NAME | DESCRIPTION | CONTENT-TYPE | USED BY | LOCATION |
+--------+------+-------------+--------------+---------+----------+
| custom | vol1 |             | filesystem   | 0       | member01 |
+--------+------+-------------+--------------+---------+----------+
  1. Moving it as admin (unix) works fine.
root@member01:~# lxc storage volume move default/vol1 default/vol1 --destination-target member02
Storage volume moved successfully!
  1. Create a fine-grained TLS identity:
root@member01:~# lxc auth identity create tls/test
TLS identity "tls/test" (0e5069ea-c62b-4b70-bca5-55e56edd4e16) pending identity token:
eyJjbGllbnRfbmFtZSI6InRlc3QiLCJmaW5nZXJwcmludCI6Ijc4YzkyMjY3MzljZjhiMWIxY2VkZThhOGU0ZmU3MjA3MDhiYjAxNWRkMDVmMWQ1ODRkMjUzMjk1MWUxOWRiM2UiLCJhZGRyZXNzZXMiOlsiMTAuMTY2LjE1Ni43Mzo4NDQzIl0sInNlY3JldCI6Ijg2YWUyYmJhNDc0ZDliN2Y4MTNjZGQ4OGU0MzViYWE1YjE3MGI3ZDc3NTc5NjE5NjY5ZmRjYmVhYWVlMGViYzkiLCJleHBpcmVzX2F0IjoiMDAwMS0wMS0wMVQwMDowMDowMFoiLCJ0eXBlIjoiQ2xpZW50IGNlcnRpZmljYXRlIn0=
root@member01:~# lxc remote add tls eyJjbGllbnRfbmFtZSI6InRlc3QiLCJmaW5nZXJwcmludCI6Ijc4YzkyMjY3MzljZjhiMWIxY2VkZThhOGU0ZmU3MjA3MDhiYjAxNWRkMDVmMWQ1ODRkMjUzMjk1MWUxOWRiM2UiLCJhZGRyZXNzZXMiOlsiMTAuMTY2LjE1Ni43Mzo4NDQzIl0sInNlY3JldCI6Ijg2YWUyYmJhNDc0ZDliN2Y4MTNjZGQ4OGU0MzViYWE1YjE3MGI3ZDc3NTc5NjE5NjY5ZmRjYmVhYWVlMGViYzkiLCJleHBpcmVzX2F0IjoiMDAwMS0wMS0wMVQwMDowMDowMFoiLCJ0eXBlIjoiQ2xpZW50IGNlcnRpZmljYXRlIn0=
Generating a client certificate. This may take a minute...
  1. Create a group with permission to view the volume, plus permission to create storage volumes in the project, and to view events in the project (needed for watching operations in our client still).
root@member01:~# lxc auth group create movers
Group movers created
root@member01:~# lxc auth identity group add tls/test movers
root@member01:~# lxc auth group permission add movers project default can_create_storage_volumes
root@member01:~# lxc auth group permission add movers project default can_view_events
root@member01:~# lxc auth group permission add movers storage_volume vol1 can_view project=default pool=default type=custom location=member02
root@member01:~# lxc storage volume list tls:default
+--------+------+-------------+--------------+---------+----------+
|  TYPE  | NAME | DESCRIPTION | CONTENT-TYPE | USED BY | LOCATION |
+--------+------+-------------+--------------+---------+----------+
| custom | vol1 |             | filesystem   | 0       | member02 |
+--------+------+-------------+--------------+---------+----------+
  1. Moving the volume fails because we don't have can_edit on it yet (expected)
root@member01:~# lxc storage volume move tls:default/vol1 tls:default/vol1 --destination-target=member01
Error: Forbidden
  1. Add edit permission:
root@member01:~# lxc auth group permission add movers storage_volume vol1 can_edit project=default pool=default type=custom location=member02
  1. When we move it now we get:
root@member01:~# lxc storage volume move tls:default/vol1 tls:default/vol1 --destination-target=member01
Error: Failed deleting source volume after copy: Forbidden
  1. After this, listing volumes as the restricted user we get:
root@member01:~# lxc storage volume list tls:default
+--------+------+-------------+--------------+---------+----------+
|  TYPE  | NAME | DESCRIPTION | CONTENT-TYPE | USED BY | LOCATION |
+--------+------+-------------+--------------+---------+----------+
| custom | vol1 |             | filesystem   | 0       | member02 |
+--------+------+-------------+--------------+---------+----------+

Which makes it look like we failed to move the volume and reverted, but we didn't, we actually copied the volume:

root@member01:~# lxc storage volume list default
+--------+------+-------------+--------------+---------+----------+
|  TYPE  | NAME | DESCRIPTION | CONTENT-TYPE | USED BY | LOCATION |
+--------+------+-------------+--------------+---------+----------+
| custom | vol1 |             | filesystem   | 0       | member02 |
+--------+------+-------------+--------------+---------+----------+
| custom | vol1 |             | filesystem   | 0       | member01 |
+--------+------+-------------+--------------+---------+----------+
  1. Delete the copy and add can_delete on the volume for the group (this is already more than should be required)
root@member01:~# lxc auth group permission add movers storage_volume vol1 can_delete project=default pool=default type=custom location=member02
root@member01:~# lxc storage volume delete default vol1 --target member01
Storage volume vol1 deleted
  1. As the restricted user, try to move the volume again - it works!
root@member01:~# lxc storage volume move tls:default/vol1 tls:default/vol1 --destination-target=member01
Storage volume moved successfully!

But now we can't view the volume anymore:

root@member01:~# lxc storage volume list tls:default
+------+------+-------------+--------------+---------+----------+
| TYPE | NAME | DESCRIPTION | CONTENT-TYPE | USED BY | LOCATION |
+------+------+-------------+--------------+---------+----------+
root@member01:~# lxc storage volume list default
+--------+------+-------------+--------------+---------+----------+
|  TYPE  | NAME | DESCRIPTION | CONTENT-TYPE | USED BY | LOCATION |
+--------+------+-------------+--------------+---------+----------+
| custom | vol1 |             | filesystem   | 0       | member01 |
+--------+------+-------------+--------------+---------+----------+

Nor can we move it back:

root@member01:~# lxc storage volume move tls:default/vol1 tls:default/vol1 --destination-target=member02
Error: Not Found

Information to attach

  • [ ] Any relevant kernel output (dmesg)
  • [ ] Instance log (lxc info NAME --show-log)
  • [ ] Instance configuration (lxc config show NAME --expanded)
  • [ ] Main daemon log (at /var/log/lxd/lxd.log or /var/snap/lxd/common/lxd/logs/lxd.log)
  • [ ] Output of the client with --debug
  • [ ] Output of the daemon with --debug (or use lxc monitor while reproducing the issue)

markylaing avatar May 09 '25 10:05 markylaing

General initial comments:

  • #12386 did not implement server side move, only server side copy. CLI deletes the old volume after copying it.
  • The ID of the storage volume is not maintained throughout the copy, so permissions are lost.

The solution to this will likely be to:

  1. Clarify that the cluster_internal_custom_volume_copy extension only implements server side copy and does not implement server side move.
  2. Implement server side move, and add another API extension for it. If this extension is present, the CLI will not need to delete the old volume.
  3. When performing server side move, copy the permissions associated with the old volume to the new volume once created, before deleting the old volume (as in #15534)

markylaing avatar May 09 '25 11:05 markylaing