vdsm icon indicating copy to clipboard operation
vdsm copied to clipboard

Optimize Volume.refresh flow

Open nirs opened this issue 2 years ago • 0 comments

When finishing extend flow we refresh the volume using storage API: Volume.refresh.

Refreshing takes 25% of total extend time, and creates lots of I/O since lvm must read entire vg metadata twice.

Volume.refresh does:

  • produce a storage domain, possibly blocking for long time if the storage domain cache is in the middle of a rescan, possibly because of unrelated storage domain.
  • produce a volume, validating that the volume exists by calling lvm.getLV()
  • call volume.refreshVolume()
  • call lvm.refreshLVs(), running "lvchange --refresh vg-name/lv-name"

In the logs we see that for every call we run "lvs" - reading entire vg metadata from storage. The lvs command always succeeds with a warning:

WARNING: Cannot find matching striped segment for bafd0f16-9aba-4f9f-ba90-46d3b8a29157/b91914e5-8446-4ff8-a143-a3f37cf23188.

The warning is likely expected and can be suppressed, but we really should not run lvs for every call.

All this flow is pointless since call this from running VM context. We know that the logical volume exist and active, so we should really call only:

lvm.refreshLVs(self.sdUUID, [self.volUUID])

Volume.refresh() does nothing for file volume, so callers should not call this API with file volumes.

Example log:

2022-05-10 00:43:32,953+0300 INFO  (mailbox-hsm/2) [vdsm.api] START refreshVolume(sdUUID='bafd0f16-9aba-4f9f-ba90-46d3b8a29157', spUUID='fa20ea36-c317-49a6-af6d-afaa5b1561df', imgUUID='a710d32b-eb03-4f19-8049-9803be251f85', volUUID='b91914e5-8446-4ff8-a143-a3f37cf23188') from=internal, task_id=62cbdc9d-9ecb-4607-b620-5e7c564465aa (api:48)

2022-05-10 00:43:33,034+0300 WARN  (mailbox-hsm/2) [storage.lvm] Command ['/usr/sbin/lvm', 'lvs', '--devices', '/dev/mapper/0QEMU_QEMU_HARDDISK_data-fc-01,/dev/mapper/0QEMU_QEMU_HARDDISK_data-fc-02,/dev/mapper/0QEMU_QEMU_HARDDISK_data-fc-03', '--config', 'devices {  preferred_names=["^/dev/mapper/"]  ignore_suspended_devices=1  write_cache_state=0  disable_after_error_count=3    hints="none"  obtain_device_list_from_udev=0 } global {  prioritise_write_locks=1  wait_for_locks=1  use_lvmpolld=1 } backup {  retain_min=50  retain_days=0 }', '--noheadings', '--units', 'b', '--nosuffix', '--separator', '|', '--ignoreskippedcluster', '-o', 'uuid,name,vg_name,attr,size,seg_start_pe,devices,tags', 'bafd0f16-9aba-4f9f-ba90-46d3b8a29157'] succeeded with warnings: ['  WARNING: Cannot find matching striped segment for bafd0f16-9aba-4f9f-ba90-46d3b8a29157/b91914e5-8446-4ff8-a143-a3f37cf23188.'] (lvm:358)

2022-05-10 00:43:33,035+0300 INFO  (mailbox-hsm/2) [storage.lvm] Refreshing LVs (vg=bafd0f16-9aba-4f9f-ba90-46d3b8a29157, lvs=('b91914e5-8446-4ff8-a143-a3f37cf23188',)) (lvm:1735)

2022-05-10 00:43:33,136+0300 INFO  (mailbox-hsm/2) [vdsm.api] FINISH refreshVolume return=None from=internal, task_id=62cbdc9d-9ecb-4607-b620-5e7c564465aa (api:54)

Fix:

  • Call lvm directly bypassing storage layer

nirs avatar May 09 '22 23:05 nirs