lxd icon indicating copy to clipboard operation
lxd copied to clipboard
verified publisher icon canonical

snapshot of Instance(VM) ceph-rbd based return -1 size

Open lccnblack opened this issue 1 year ago • 8 comments

Required information

  • Distribution: 22.04.4 LTS

  • Distribution version: lxd 6.1-c14927a 29551 6.1/stable canonical** in-cohort lxd 6.1-0d4d89b 29469 6.1/stable canonical** disabled,in-cohort microceph 0+git.4a608fc 793 quincy/stable canonical** in-cohort microcloud 1.1-04a1c49 734 latest/stable canonical** in-cohort

  • The output of "snap list --all lxd core20 core22 core24 snapd":

  • The output of "lxc info" or if that fails:

    • Kernel version: 5.15.0-116-generic
    • LXC version:
    • LXD version:
    • Storage backend in use: microceph

Issue description

The customer has a vm that is ceph rbd based. lxc config show / shows that the size is -1. However, the rbd du /.block returns normally. What's interesting is that only the first snapshot returns the normal size, and all the other snapshots return size -1. It seems that the other snapshots are taken via the schedule.

lxc info ConvertVM

Name: ConvertVM
Status: RUNNING
Type: virtual-machine
Architecture: x86_64
Location: node2
PID: 71917
Created: 2024/06/14 02:28 UTC
Last Used: 2024/08/28 09:08 UTC
 
Resources:
  Processes: 23
  CPU usage:
    CPU usage (in seconds): 3041
  Memory usage:
    Memory (current): 1.85GiB
  Network usage:
    enp5s0:
      Type: broadcast
      State: UP
      Host interface: tap716b3cff
      MAC address: 00:16:3e:4d:24:d0
      MTU: 1450
      Bytes received: 12.42MB
      Bytes sent: 907.53kB
      Packets received: 12904
      Packets sent: 9896
      IP addresses:
        inet:  240.242.0.55/8 (global)
        inet6: fe80::216:3eff:fe4d:24d0/64 (link)
    enp6s0:
      Type: broadcast
      State: DOWN
      Host interface: mac4d62bf00
      MAC address: 00:16:3e:d2:2a:22
      MTU: 1500
      Bytes received: 0B
      Bytes sent: 0B
      Packets received: 0
      Packets sent: 0
      IP addresses:
    lo:
      Type: loopback
      State: UP
      MTU: 65536
      Bytes received: 89.40MB
      Bytes sent: 89.40MB
      Packets received: 1164630
      Packets sent: 1164630
      IP addresses:
        inet:  127.0.0.1/8 (local)
        inet6: ::1/128 (local)
 
Snapshots:
+--------+----------------------+----------------------+----------+
|  NAME  |       TAKEN AT       |      EXPIRES AT      | STATEFUL |
+--------+----------------------+----------------------+----------+
| snap1  | 2024/07/31 03:49 UTC |                      | NO       |
+--------+----------------------+----------------------+----------+
| snap2  | 2024/07/31 03:56 UTC |                      | NO       |
+--------+----------------------+----------------------+----------+
| snap23 | 2024/08/23 11:59 UTC | 2024/08/30 11:59 UTC | NO       |
+--------+----------------------+----------------------+----------+
| snap24 | 2024/08/24 11:59 UTC | 2024/08/31 11:59 UTC | NO       |
+--------+----------------------+----------------------+----------+
| snap25 | 2024/08/25 11:59 UTC | 2024/09/01 11:59 UTC | NO       |
+--------+----------------------+----------------------+----------+
| snap26 | 2024/08/26 12:00 UTC | 2024/09/02 12:00 UTC | NO       |
+--------+----------------------+----------------------+----------+
| snap27 | 2024/08/28 11:59 UTC | 2024/09/04 11:59 UTC | NO       |
+--------+----------------------+----------------------+----------+
| snap28 | 2024/08/29 11:59 UTC | 2024/09/05 11:59 UTC | NO       |
+--------+----------------------+----------------------+----------+

lxc config show ConvertVM/snap22

architecture: x86_64
config:
  limits.cpu: "4"
  limits.memory: 16GiB
  migration.stateful: "true"
  snapshots.expiry: 7d
  snapshots.schedule: 0 12 * * *
  volatile.cloud-init.instance-id: 2e8109dd-0685-470d-9ba9-ab04833c4eae
  volatile.eth0.host_name: tapf70fb09c
  volatile.eth0.hwaddr: 00:16:3e:4d:24:d0
  volatile.eth1.host_name: mac32d52912
  volatile.eth1.hwaddr: 00:16:3e:d2:2a:22
  volatile.eth1.last_state.created: "false"
  volatile.last_state.power: RUNNING
  volatile.last_state.ready: "false"
  volatile.uuid: 6107e0ed-a4c2-46dc-a7c0-6bd208c327d9
  volatile.uuid.generation: 6107e0ed-a4c2-46dc-a7c0-6bd208c327d9
  volatile.vsock_id: "1292222863"
created_at: 2024-08-22T11:59:56.735294543Z
expires_at: 2024-08-29T11:59:56.733439655Z
devices:
  root:
    path: /
    pool: remote
    size: 210GiB
    type: disk
ephemeral: false
expanded_config:
  limits.cpu: "4"
  limits.memory: 16GiB
  migration.stateful: "true"
  snapshots.expiry: 7d
  snapshots.schedule: 0 12 * * *
  volatile.cloud-init.instance-id: 2e8109dd-0685-470d-9ba9-ab04833c4eae
  volatile.eth0.host_name: tapf70fb09c
  volatile.eth0.hwaddr: 00:16:3e:4d:24:d0
  volatile.eth1.host_name: mac32d52912
  volatile.eth1.hwaddr: 00:16:3e:d2:2a:22
  volatile.eth1.last_state.created: "false"
  volatile.last_state.power: RUNNING
  volatile.last_state.ready: "false"
  volatile.uuid: 6107e0ed-a4c2-46dc-a7c0-6bd208c327d9
  volatile.uuid.generation: 6107e0ed-a4c2-46dc-a7c0-6bd208c327d9
  volatile.vsock_id: "1292222863"
expanded_devices:
  eth0:
    name: eth0
    network: lxdfan0
    type: nic
  eth1:
    nictype: macvlan
    parent: eno1
    type: nic
  root:
    path: /
    pool: remote
    size: 210GiB
    type: disk
last_used_at: 0001-01-01T00:00:00Z
name: snap22
profiles:
- default
- macvlan
stateful: false
size: -1

lxc config show ConvertVM/snap1

architecture: x86_64
config:
  limits.cpu: "4"
  limits.memory: 16GiB
  volatile.cloud-init.instance-id: 2e8109dd-0685-470d-9ba9-ab04833c4eae
  volatile.eth0.host_name: tap019c0ee9
  volatile.eth0.hwaddr: 00:16:3e:4d:24:d0
  volatile.eth1.host_name: mac626955d9
  volatile.eth1.hwaddr: 00:16:3e:d2:2a:22
  volatile.eth1.last_state.created: "false"
  volatile.last_state.power: RUNNING
  volatile.last_state.ready: "false"
  volatile.uuid: 6107e0ed-a4c2-46dc-a7c0-6bd208c327d9
  volatile.uuid.generation: 6107e0ed-a4c2-46dc-a7c0-6bd208c327d9
  volatile.vsock_id: "1292222863"
created_at: 2024-07-31T03:49:50.041802922Z
expires_at: 0001-01-01T00:00:00Z
devices:
  root:
    path: /
    pool: remote
    size: 210GiB
    type: disk
ephemeral: false
expanded_config:
  limits.cpu: "4"
  limits.memory: 16GiB
  volatile.cloud-init.instance-id: 2e8109dd-0685-470d-9ba9-ab04833c4eae
  volatile.eth0.host_name: tap019c0ee9
  volatile.eth0.hwaddr: 00:16:3e:4d:24:d0
  volatile.eth1.host_name: mac626955d9
  volatile.eth1.hwaddr: 00:16:3e:d2:2a:22
  volatile.eth1.last_state.created: "false"
  volatile.last_state.power: RUNNING
  volatile.last_state.ready: "false"
  volatile.uuid: 6107e0ed-a4c2-46dc-a7c0-6bd208c327d9
  volatile.uuid.generation: 6107e0ed-a4c2-46dc-a7c0-6bd208c327d9
  volatile.vsock_id: "1292222863"
expanded_devices:
  eth0:
    name: eth0
    network: lxdfan0
    type: nic
  eth1:
    nictype: macvlan
    parent: eno1
    type: nic
  root:
    path: /
    pool: remote
    size: 210GiB
    type: disk
last_used_at: 0001-01-01T00:00:00Z
name: snap1
profiles:
- default
- macvlan
stateful: false
size: 94795464704

rbd du remote/virtual-machine_ConvertVM.block

NAME                                             PROVISIONED  USED
virtual-machine_ConvertVM.block@snapshot_snap1       210 GiB   88 GiB
virtual-machine_ConvertVM.block@snapshot_snap2       210 GiB   92 MiB
virtual-machine_ConvertVM.block@snapshot_snap20      210 GiB   12 GiB
virtual-machine_ConvertVM.block@snapshot_snap21      210 GiB  117 GiB
virtual-machine_ConvertVM.block@snapshot_snap22      210 GiB  1.6 GiB
virtual-machine_ConvertVM.block@snapshot_snap23      210 GiB   58 GiB
virtual-machine_ConvertVM.block@snapshot_snap24      210 GiB  1.1 GiB
virtual-machine_ConvertVM.block@snapshot_snap25      210 GiB  1.1 GiB
virtual-machine_ConvertVM.block@snapshot_snap26      210 GiB   15 GiB
virtual-machine_ConvertVM.block                      210 GiB  1.3 GiB
<TOTAL>                                              210 GiB  296 GiB

lccnblack avatar Aug 30 '24 09:08 lccnblack

Did you have lxc storage set <pool> ceph.rbd.du=true enabled?

tomponline avatar Aug 30 '24 10:08 tomponline

Thank you for the reply! I will ask customer to try it. However, in my lab, even though I didn't set this parameter, I still can't see the problem of -1. It seems that this parameter is for stopped instance. https://documentation.ubuntu.com/lxd/en/latest/reference/storage_ceph/#storage-pool-configuration This option specifies whether to use RBD du to obtain disk usage data for stopped instances.

lccnblack avatar Sep 02 '24 04:09 lccnblack 

root@node4:~# lxc storage set remote ceph.rbd.du=true
root@node4:~# lxc config show ConvertVM/snap40
architecture: x86_64
config:
  limits.cpu: "4"
  limits.memory: 16GiB
  migration.stateful: "true"
  snapshots.expiry: 7d
  snapshots.schedule: 0 12 * * *
  volatile.cloud-init.instance-id: 2e8109dd-0685-470d-9ba9-ab04833c4eae
  volatile.eth0.host_name: tape7b3ae59
  volatile.eth0.hwaddr: 00:16:3e:4d:24:d0
  volatile.eth1.host_name: macc762736b
  volatile.eth1.hwaddr: 00:16:3e:d2:2a:22
  volatile.eth1.last_state.created: "false"
  volatile.last_state.power: RUNNING
  volatile.uuid: 6107e0ed-a4c2-46dc-a7c0-6bd208c327d9
  volatile.uuid.generation: 6107e0ed-a4c2-46dc-a7c0-6bd208c327d9
  volatile.vsock_id: "1292222863"
created_at: 2024-09-12T11:59:55.432874933Z
expires_at: 2024-09-19T11:59:55.430551817Z
devices:
  root:
    path: /
    pool: remote
    size: 210GiB
    type: disk
ephemeral: false
expanded_config:
  limits.cpu: "4"
  limits.memory: 16GiB
  migration.stateful: "true"
  snapshots.expiry: 7d
  snapshots.schedule: 0 12 * * *
  volatile.cloud-init.instance-id: 2e8109dd-0685-470d-9ba9-ab04833c4eae
  volatile.eth0.host_name: tape7b3ae59
  volatile.eth0.hwaddr: 00:16:3e:4d:24:d0
  volatile.eth1.host_name: macc762736b
  volatile.eth1.hwaddr: 00:16:3e:d2:2a:22
  volatile.eth1.last_state.created: "false"
  volatile.last_state.power: RUNNING
  volatile.uuid: 6107e0ed-a4c2-46dc-a7c0-6bd208c327d9
  volatile.uuid.generation: 6107e0ed-a4c2-46dc-a7c0-6bd208c327d9
  volatile.vsock_id: "1292222863"
expanded_devices:
  eth0:
    name: eth0
    network: lxdfan0
    type: nic
  eth1:
    nictype: macvlan
    parent: eno1
    type: nic
  root:
    path: /
    pool: remote
    size: 210GiB
    type: disk
last_used_at: 0001-01-01T00:00:00Z
name: snap40
profiles:
- default
- macvlan
stateful: false
size: -1

Do we need to take a new snapshot after setting ceph.rbd.du=true?

lccnblack avatar Sep 13 '24 07:09 lccnblack

Do we need to take a new snapshot after setting ceph.rbd.du=true?

Can you please try running lxc storage set <pool> ceph.rbd.du=true and then take another snapshot?

kadinsayani avatar Oct 10 '24 20:10 kadinsayani

Can you please try running lxc storage set <pool> ceph.rbd.du=true and then take another snapshot?

Does this affect existing snapshots?

tomponline avatar Oct 15 '24 07:10 tomponline

Does this affect existing snapshots?

Yes - after changing the setting and running lxc config show <vm>/<snap>, the expected size is returned.

kadinsayani avatar Oct 18 '24 20:10 kadinsayani

Here are my test results using a VM with a ceph backend (single node cluster) 

root@v1:~# lxc storage set pool1 ceph.rbd.du=false
root@v1:~# lxc snapshot v1 snap1
root@v1:~# lxc config show v1/snap1
architecture: x86_64
config:
  image.architecture: amd64
  image.description: ubuntu 24.04 LTS amd64 (release) (20241004)
  image.label: release
  image.os: ubuntu
  image.release: noble
  image.serial: "20241004"
  image.type: disk1.img
  image.version: "24.04"
  volatile.base_image: 427eb975d1b2c0561d59b0a6092e1c9046585c6c6718319edfdcfdc8a5ec3dc1
  volatile.cloud-init.instance-id: cfb5d25b-470f-49a6-b384-789700dccf85
  volatile.eth0.host_name: tapb0d39ad9
  volatile.eth0.hwaddr: 00:16:3e:27:e9:dd
  volatile.last_state.power: RUNNING
  volatile.uuid: 830d6b08-e882-4924-9c81-cc62a632b451
  volatile.uuid.generation: 830d6b08-e882-4924-9c81-cc62a632b451
  volatile.vsock_id: "1155209746"
created_at: 2024-10-18T20:55:57.744013777Z
expires_at: 0001-01-01T00:00:00Z
devices:
  root:
    path: /
    pool: pool1
    type: disk
ephemeral: false
expanded_config:
  image.architecture: amd64
  image.description: ubuntu 24.04 LTS amd64 (release) (20241004)
  image.label: release
  image.os: ubuntu
  image.release: noble
  image.serial: "20241004"
  image.type: disk1.img
  image.version: "24.04"
  volatile.base_image: 427eb975d1b2c0561d59b0a6092e1c9046585c6c6718319edfdcfdc8a5ec3dc1
  volatile.cloud-init.instance-id: cfb5d25b-470f-49a6-b384-789700dccf85
  volatile.eth0.host_name: tapb0d39ad9
  volatile.eth0.hwaddr: 00:16:3e:27:e9:dd
  volatile.last_state.power: RUNNING
  volatile.uuid: 830d6b08-e882-4924-9c81-cc62a632b451
  volatile.uuid.generation: 830d6b08-e882-4924-9c81-cc62a632b451
  volatile.vsock_id: "1155209746"
expanded_devices:
  eth0:
    name: eth0
    network: lxdbr0
    type: nic
  root:
    path: /
    pool: pool1
    type: disk
last_used_at: 0001-01-01T00:00:00Z
name: snap1
profiles:
- default
stateful: false
size: -1
root@v1:~# lxc storage set pool1 ceph.rbd.du=true
root@v1:~# lxc config show v1/snap1
architecture: x86_64
config:
  image.architecture: amd64
  image.description: ubuntu 24.04 LTS amd64 (release) (20241004)
  image.label: release
  image.os: ubuntu
  image.release: noble
  image.serial: "20241004"
  image.type: disk1.img
  image.version: "24.04"
  volatile.base_image: 427eb975d1b2c0561d59b0a6092e1c9046585c6c6718319edfdcfdc8a5ec3dc1
  volatile.cloud-init.instance-id: cfb5d25b-470f-49a6-b384-789700dccf85
  volatile.eth0.host_name: tapb0d39ad9
  volatile.eth0.hwaddr: 00:16:3e:27:e9:dd
  volatile.last_state.power: RUNNING
  volatile.uuid: 830d6b08-e882-4924-9c81-cc62a632b451
  volatile.uuid.generation: 830d6b08-e882-4924-9c81-cc62a632b451
  volatile.vsock_id: "1155209746"
created_at: 2024-10-18T20:55:57.744013777Z
expires_at: 0001-01-01T00:00:00Z
devices:
  root:
    path: /
    pool: pool1
    type: disk
ephemeral: false
expanded_config:
  image.architecture: amd64
  image.description: ubuntu 24.04 LTS amd64 (release) (20241004)
  image.label: release
  image.os: ubuntu
  image.release: noble
  image.serial: "20241004"
  image.type: disk1.img
  image.version: "24.04"
  volatile.base_image: 427eb975d1b2c0561d59b0a6092e1c9046585c6c6718319edfdcfdc8a5ec3dc1
  volatile.cloud-init.instance-id: cfb5d25b-470f-49a6-b384-789700dccf85
  volatile.eth0.host_name: tapb0d39ad9
  volatile.eth0.hwaddr: 00:16:3e:27:e9:dd
  volatile.last_state.power: RUNNING
  volatile.uuid: 830d6b08-e882-4924-9c81-cc62a632b451
  volatile.uuid.generation: 830d6b08-e882-4924-9c81-cc62a632b451
  volatile.vsock_id: "1155209746"
expanded_devices:
  eth0:
    name: eth0
    network: lxdbr0
    type: nic
  root:
    path: /
    pool: pool1
    type: disk
last_used_at: 0001-01-01T00:00:00Z
name: snap1
profiles:
- default
stateful: false
size: 138412032

kadinsayani avatar Oct 18 '24 20:10 kadinsayani

@lccnblack happy to close this now?

tomponline avatar Oct 21 '24 07:10 tomponline

Closing this issue but if you have any concerns or run into any more issues please feel free to reopen this or reach out! Thanks!

kadinsayani avatar Oct 29 '24 11:10 kadinsayani