Unbalanced RAW NVME disks usage

[ksqrtr]

Hello!

We have 4 NVME disks and noticed recently that their usage is some kind of unbalanced:

# for nvme in 0 1 2 3; do echo "nvme${nvme}:"; smartctl -x /dev/nvme"${nvme}"n1 | grep 'Data Units' | sed 's/^/    /' ; done
nvme0:
    Data Units Read:                    127 608 511 556 [65,3 PB]
    Data Units Written:                 293 649 334 [150 TB]
nvme1:
    Data Units Read:                    129 983 891 432 [66,5 PB]
    Data Units Written:                 384 164 362 [196 TB]
nvme2:
    Data Units Read:                    562 268 [287 GB]
    Data Units Written:                 8 382 370 [4,29 TB]
nvme3:
    Data Units Read:                    551 991 [282 GB]
    Data Units Written:                 8 387 551 [4,29 TB]

What might be the cause of such usage unbalance and is there any options in ATS we can tune?

Our storage.config is pretty straightforward:

# cat storage.config 
/dev/crypt-nvme-luks-S2UDNX0J602140
/dev/crypt-nvme-luks-S2UDNX0J402480
/dev/crypt-nvme-luks-S2UDNX0J602013
/dev/crypt-nvme-luks-S2UDNX0J602462

We are using ATS 9.2.8.

[cmcfarlen]

There could be a few reasons for this. Could you share your cache config and volume config along with some sample urls? If the same urls are updated frequently then they would all write to the same disk.