"lvextend" command freezes my Debian 11(.3) server

[tigerblue77]

Hello everyone,

Still working on my project to make VDO work on my Debian 11(.3), I'm testing on my production machine now and encountered a problem while extending a VDOPOOL and VDOdataLV, everything is detailed here. I have not found a fix for the moment.

My problem description is a bit short because I don't really know which logs or screenshots I could show you, don't hesitate to ask. Thanks per advance !

[bjohnsto]

Hi Tigerblue77. My name is Bruce Johnston and I work on the VDO team. I'm looking into your issue but had a couple questions. First it appears, according to the logs you posted, that the VDO volume is 'dirty' as it is trying to rebuild the volume. I was wondering if you could give me some details as to what you might have been doing with the volume before trying to extend? Was this a brand new VDO LVM volume?

I'd also like to get some general info on how big the volume was before and after resizing, and what parameters you used to create the volume, if possible.

Also I was wondering if you had tried lvresize vs lvextend and seen if there were any differences?

Thanks in advance for your help. Bruce

[tigerblue77]

Hello Bruce, nice to meet you. At this rate, I'll end up knowing the whole team! Or at least be known by your whole team as a pain in the a** who opens github issues all the time haha!

Thanks for taking a look at my issue. I see that you're hungry for informations and I will feed you as much as I can. Do not hesitate to ask any other question you would have or give me advice(s) if you see anything wrong. In my logs for example, I'm a bit worried by warnings like tools version not matching but I suppose that it's because of my LVM version ?

First it appears, according to the logs you posted, that the VDO volume is 'dirty' as it is trying to rebuild the volume.

It was dirty because I brutally restarted from iDrac as it was frozen. I just tested a normal reboot and the "dirty" message is not present in logs

I was wondering if you could give me some details as to what you might have been doing with the volume before trying to extend? Was this a brand new VDO LVM volume?

Yes it's a brand new LVM VDO volume created with the following LVM metadata profile :

# Custom configuration for VDO using compression/deduplication and more CPUs (based on local vCPU count)
# Custom parameters based on: https://is.muni.cz/th/rq7e2/petrovic_diploma_thesis.pdf

allocation {
  vdo_use_compression=1
  vdo_use_deduplication=1
  vdo_use_metadata_hints=1
  vdo_minimum_io_size=4096
  vdo_block_map_cache_size_mb=16384
  vdo_block_map_period=16380
  vdo_check_point_frequency=0
  vdo_use_sparse_index=1
  vdo_index_memory_size_mb=3072
  vdo_slab_size_mb=8192
  vdo_ack_threads=$((VCPU_COUNT/4))
  vdo_bio_threads=$VCPU_COUNT
  vdo_bio_rotation=64
  vdo_cpu_threads=$((VCPU_COUNT/2))
  vdo_hash_zone_threads=$((VCPU_COUNT/4))
  vdo_logical_threads=$((VCPU_COUNT/4))
  vdo_physical_threads=1
  vdo_write_policy="async"
  vdo_max_discard=1
}

Except this metadata profile, no particular parameter was used. I just gave --size --virtualsize, VDO pool name, LVM VG name and VDOdataLV name. I first wrote a BASH script, tested on a test VM, then I converted it to an Ansible playbook, tested on the test VM and now I'm testing on my future "production" bare-metal server ("" because it's a homelab). Don't hesitate to ask if you need more details.

About resizing, the volume was initially 12GB then 1TB and then I added TB until now 12,1TB pool and 13,1TB VDOdataLV. FYI my VG is 21,8TB. I think the problem is not about the size and I would be able to reproduce it with any size. I think the problem is about IO, as discussed in Reddit thread.

I didn't test lvresize instead of lvextend, do you want me to try ? Once again, I'm no longer sure about the problem root cause, maybe we should try to fix the IO problem first ?

Thanks for you help ! Tiger

[bjohnsto]

Thanks for taking a look at my issue. I see that you're hungry for informations and I will feed you as much as I can. Do not hesitate to ask any other question you would have or give me advice(s) if you see anything wrong. In my logs for example, I'm a bit worried by warnings like tools version not matching but I suppose that it's because of my LVM version ?

I wouldn't worry about that at the moment. This issue should be solved with a future LVM update. I'm not sure if it is out yet, but should be soon.

It was dirty because I brutally restarted from iDrac as it was frozen. I just tested a normal reboot and the "dirty" message is not present in logs

Would you have any logs from before the crash? During the actual extend time and then the hang?

About resizing, the volume was initially 12GB then 1TB and then I added TB until now 12,1TB pool and 13,1TB VDOdataLV. FYI my VG is 21,8TB. I think the problem is not about the size and I would be able to reproduce it with any size. I think the problem is about IO, as discussed in Reddit thread.

It could be. It certainly seems like you have some leads there. I'd be happy to wait on your testing there to see if that fixes issues. Let us know if you want to revisit this.

Thanks Bruce

[tigerblue77]

Unable to reproduce problem with my new installation (KVDO v8.2.1.3 and VDO v8.2.0.2). Thanks for your help.