glusterfs Error in count inode glusterfs

GlusterFS inodes count show 97% IUse% but all physical discs in this volume, inodes count show only 5% IUse%

df -i Filesystem Inodes IUsed ...... /dev/nvme13n1 1500292096 69419375 1430872721 /dev/nvme8n1 1500292096 67343419 1432948677 /dev/nvme22n1 1500292096 66027751 1434264345 /dev/nvme6n1 1500292096 69619855 1430672241 /dev/nvme18n1 1500292096 68580148 1431711948 /dev/nvme23n1 1500292096 66027751 1434264345 /dev/nvme14n1 1500292096 69419375 1430872721 /dev/nvme11n1 1500292096 67343419 1432948677 /dev/nvme16n1 1500292096 68580148 1431711948 /dev/nvme1n1 1500292096 72355100 1427936996 /dev/nvme5n1 1500292096 69619855 1430672241 /dev/nvme19n1 1500292096 68580148 1431711948 /dev/nvme15n1 1500292096 69419375 1430872721 /dev/nvme12n1 1500292096 69419375 1430872721 /dev/nvme2n1 1500292096 72355100 1427936996 /dev/nvme3n1 1500292096 72355100 1427936996 /dev/nvme21n1 1500292096 66027751 1434264345 /dev/nvme20n1 1500292096 66027751 1434264345 /dev/nvme0n1 1500292096 72355100 1427936996 /dev/nvme9n1 1500292096 67343419 1432948677 /dev/nvme4n1 1500292096 69619855 1430672241 /dev/nvme17n1 1500292096 68580148 1431711948 /dev/nvme7n1 1500292096 69619855 1430672241 /dev/nvme10n1 1500292096 67343419 1432948677 ...... localhost:/nfs_volume 9001752576 8643902291 357850285 ...... IFree IUse% Mounted on 5% /data/brick_nvme13n1 5% /data/brick_nvme8n1 5% /data/brick_nvme22n1 5% /data/brick_nvme6n1 5% /data/brick_nvme18n1 5% /data/brick_nvme23n1 5% /data/brick_nvme14n1 5% /data/brick_nvme11n1 5% /data/brick_nvme16n1 5% /data/brick_nvme1n1 5% /data/brick_nvme5n1 5% /data/brick_nvme19n1 5% /data/brick_nvme15n1 5% /data/brick_nvme12n1 5% /data/brick_nvme2n1 5% /data/brick_nvme3n1 5% /data/brick_nvme21n1 5% /data/brick_nvme20n1 5% /data/brick_nvme0n1 5% /data/brick_nvme9n1 5% /data/brick_nvme4n1 5% /data/brick_nvme17n1 5% /data/brick_nvme7n1 5% /data/brick_nvme10n1 97% /mnt/nfs_volume

df -HT Filesystem Type Size Used Avail Use% Mounted on ....... /dev/nvme13n1 xfs 16T 470G 15T 4% /data/brick_nvme13n1 /dev/nvme8n1 xfs 16T 472G 15T 4% /data/brick_nvme8n1 /dev/nvme22n1 xfs 16T 491G 15T 4% /data/brick_nvme22n1 /dev/nvme6n1 xfs 16T 528G 15T 4% /data/brick_nvme6n1 /dev/nvme18n1 xfs 16T 465G 15T 4% /data/brick_nvme18n1 /dev/nvme23n1 xfs 16T 491G 15T 4% /data/brick_nvme23n1 /dev/nvme14n1 xfs 16T 470G 15T 4% /data/brick_nvme14n1 /dev/nvme11n1 xfs 16T 472G 15T 4% /data/brick_nvme11n1 /dev/nvme16n1 xfs 16T 465G 15T 4% /data/brick_nvme16n1 /dev/nvme1n1 xfs 16T 618G 15T 5% /data/brick_nvme1n1 /dev/nvme5n1 xfs 16T 528G 15T 4% /data/brick_nvme5n1 /dev/nvme19n1 xfs 16T 465G 15T 4% /data/brick_nvme19n1 /dev/nvme15n1 xfs 16T 470G 15T 4% /data/brick_nvme15n1 /dev/nvme12n1 xfs 16T 470G 15T 4% /data/brick_nvme12n1 /dev/nvme2n1 xfs 16T 618G 15T 5% /data/brick_nvme2n1 /dev/nvme3n1 xfs 16T 618G 15T 5% /data/brick_nvme3n1 /dev/nvme21n1 xfs 16T 491G 15T 4% /data/brick_nvme21n1 /dev/nvme20n1 xfs 16T 491G 15T 4% /data/brick_nvme20n1 /dev/nvme0n1 xfs 16T 618G 15T 5% /data/brick_nvme0n1 /dev/nvme9n1 xfs 16T 472G 15T 4% /data/brick_nvme9n1 /dev/nvme4n1 xfs 16T 528G 15T 4% /data/brick_nvme4n1 /dev/nvme17n1 xfs 16T 465G 15T 4% /data/brick_nvme17n1 /dev/nvme7n1 xfs 16T 528G 15T 4% /data/brick_nvme7n1 /dev/nvme10n1 xfs 16T 472G 15T 4% /data/brick_nvme10n1 ........ localhost:/nfs_volume fuse.glusterfs 31T 1.4T 30T 5% /mnt/nfs_volume .......

Expected result:

IUsed of glusterfs = 6xIUsed of disc (where 6 - number of data disc in subvolume) IFree of glusterfs = 6xIFree of disc (where 6 - number of data disc in subvolume) IUse% of glusterfs = 6xIUse% of disc (where 6 - number of data disc in subvolume)

Size of glusterfs = 6xSize of disc (where 6 - number of data disc in subvolume) Used of glusterfs = 6xUsed of disc (where 6 - number of data disc in subvolume) Avail of glusterfs = 6Avail of disc (where 6 - number of data disc in subvolume)

*** Mandatory information ***

gluster volume info nfs_volume

Volume Name: nfs_volume Type: Distributed-Disperse Volume ID: 9c039b2e-ebd7-4159-b807-6b2e56f64db7 Status: Started Snapshot Count: 0 Number of Bricks: 6 x (8 + 4) = 72 Transport-type: tcp Bricks: Brick1: p1-nx-fsr0001-be.mydomain.com:/data/brick_nvme0n1 Brick2: p1-nx-fsr0001-be.mydomain.com:/data/brick_nvme1n1 Brick3: p1-nx-fsr0001-be.mydomain.com:/data/brick_nvme2n1 Brick4: p1-nx-fsr0001-be.mydomain.com:/data/brick_nvme3n1 Brick5: p1-nx-fsr0002-be.mydomain.com:/data/brick_nvme0n1 Brick6: p1-nx-fsr0002-be.mydomain.com:/data/brick_nvme1n1 Brick7: p1-nx-fsr0002-be.mydomain.com:/data/brick_nvme2n1 Brick8: p1-nx-fsr0002-be.mydomain.com:/data/brick_nvme3n1 Brick9: p1-nx-fsr0003-be.mydomain.com:/data/brick_nvme0n1 Brick10: p1-nx-fsr0003-be.mydomain.com:/data/brick_nvme1n1 Brick11: p1-nx-fsr0003-be.mydomain.com:/data/brick_nvme2n1 Brick12: p1-nx-fsr0003-be.mydomain.com:/data/brick_nvme3n1 Brick13: p1-nx-fsr0001-be.mydomain.com:/data/brick_nvme4n1 Brick14: p1-nx-fsr0001-be.mydomain.com:/data/brick_nvme5n1 Brick15: p1-nx-fsr0001-be.mydomain.com:/data/brick_nvme6n1 Brick16: p1-nx-fsr0001-be.mydomain.com:/data/brick_nvme7n1 Brick17: p1-nx-fsr0002-be.mydomain.com:/data/brick_nvme4n1 Brick18: p1-nx-fsr0002-be.mydomain.com:/data/brick_nvme5n1 Brick19: p1-nx-fsr0002-be.mydomain.com:/data/brick_nvme6n1 Brick20: p1-nx-fsr0002-be.mydomain.com:/data/brick_nvme7n1 Brick21: p1-nx-fsr0003-be.mydomain.com:/data/brick_nvme4n1 Brick22: p1-nx-fsr0003-be.mydomain.com:/data/brick_nvme5n1 Brick23: p1-nx-fsr0003-be.mydomain.com:/data/brick_nvme6n1 Brick24: p1-nx-fsr0003-be.mydomain.com:/data/brick_nvme7n1 Brick25: p1-nx-fsr0001-be.mydomain.com:/data/brick_nvme8n1 Brick26: p1-nx-fsr0001-be.mydomain.com:/data/brick_nvme9n1 Brick27: p1-nx-fsr0001-be.mydomain.com:/data/brick_nvme10n1 Brick28: p1-nx-fsr0001-be.mydomain.com:/data/brick_nvme11n1 Brick29: p1-nx-fsr0002-be.mydomain.com:/data/brick_nvme8n1 Brick30: p1-nx-fsr0002-be.mydomain.com:/data/brick_nvme9n1 Brick31: p1-nx-fsr0002-be.mydomain.com:/data/brick_nvme10n1 Brick32: p1-nx-fsr0002-be.mydomain.com:/data/brick_nvme11n1 Brick33: p1-nx-fsr0003-be.mydomain.com:/data/brick_nvme8n1 Brick34: p1-nx-fsr0003-be.mydomain.com:/data/brick_nvme9n1 Brick35: p1-nx-fsr0003-be.mydomain.com:/data/brick_nvme10n1 Brick36: p1-nx-fsr0003-be.mydomain.com:/data/brick_nvme11n1 Brick37: p1-nx-fsr0001-be.mydomain.com:/data/brick_nvme12n1 Brick38: p1-nx-fsr0001-be.mydomain.com:/data/brick_nvme13n1 Brick39: p1-nx-fsr0001-be.mydomain.com:/data/brick_nvme14n1 Brick40: p1-nx-fsr0001-be.mydomain.com:/data/brick_nvme15n1 Brick41: p1-nx-fsr0002-be.mydomain.com:/data/brick_nvme12n1 Brick42: p1-nx-fsr0002-be.mydomain.com:/data/brick_nvme13n1 Brick43: p1-nx-fsr0002-be.mydomain.com:/data/brick_nvme14n1 Brick44: p1-nx-fsr0002-be.mydomain.com:/data/brick_nvme15n1 Brick45: p1-nx-fsr0003-be.mydomain.com:/data/brick_nvme12n1 Brick46: p1-nx-fsr0003-be.mydomain.com:/data/brick_nvme13n1 Brick47: p1-nx-fsr0003-be.mydomain.com:/data/brick_nvme14n1 Brick48: p1-nx-fsr0003-be.mydomain.com:/data/brick_nvme15n1 Brick49: p1-nx-fsr0001-be.mydomain.com:/data/brick_nvme16n1 Brick50: p1-nx-fsr0001-be.mydomain.com:/data/brick_nvme17n1 Brick51: p1-nx-fsr0001-be.mydomain.com:/data/brick_nvme18n1 Brick52: p1-nx-fsr0001-be.mydomain.com:/data/brick_nvme19n1 Brick53: p1-nx-fsr0002-be.mydomain.com:/data/brick_nvme16n1 Brick54: p1-nx-fsr0002-be.mydomain.com:/data/brick_nvme17n1 Brick55: p1-nx-fsr0002-be.mydomain.com:/data/brick_nvme18n1 Brick56: p1-nx-fsr0002-be.mydomain.com:/data/brick_nvme19n1 Brick57: p1-nx-fsr0003-be.mydomain.com:/data/brick_nvme16n1 Brick58: p1-nx-fsr0003-be.mydomain.com:/data/brick_nvme17n1 Brick59: p1-nx-fsr0003-be.mydomain.com:/data/brick_nvme18n1 Brick60: p1-nx-fsr0003-be.mydomain.com:/data/brick_nvme19n1 Brick61: p1-nx-fsr0001-be.mydomain.com:/data/brick_nvme20n1 Brick62: p1-nx-fsr0001-be.mydomain.com:/data/brick_nvme21n1 Brick63: p1-nx-fsr0001-be.mydomain.com:/data/brick_nvme22n1 Brick64: p1-nx-fsr0001-be.mydomain.com:/data/brick_nvme23n1 Brick65: p1-nx-fsr0002-be.mydomain.com:/data/brick_nvme20n1 Brick66: p1-nx-fsr0002-be.mydomain.com:/data/brick_nvme21n1 Brick67: p1-nx-fsr0002-be.mydomain.com:/data/brick_nvme22n1 Brick68: p1-nx-fsr0002-be.mydomain.com:/data/brick_nvme23n1 Brick69: p1-nx-fsr0003-be.mydomain.com:/data/brick_nvme20n1 Brick70: p1-nx-fsr0003-be.mydomain.com:/data/brick_nvme21n1 Brick71: p1-nx-fsr0003-be.mydomain.com:/data/brick_nvme22n1 Brick72: p1-nx-fsr0003-be.mydomain.com:/data/brick_nvme23n1 Options Reconfigured: performance.parallel-readdir: on ctime.noatime: on performance.stat-prefetch: on network.inode-lru-limit: 400000 features.cache-invalidation: on features.cache-invalidation-timeout: 600 performance.cache-invalidation: on performance.md-cache-timeout: 600 cluster.readdir-optimize: on performance.readdir-ahead: on performance.io-cache: on performance.cache-size: 1GB client.event-threads: 4 server.event-threads: 4 features.quota-deem-statfs: on features.inode-quota: on features.quota: on storage.fips-mode-rchecksum: on transport.address-family: inet nfs.disable: on cluster.enable-shared-storage: enable nfs-ganesha: enable

gluster volume status nfs_volume

Status of volume: nfs_volume Gluster process TCP Port RDMA Port Online Pid

Brick p1-nx-fsr0001-be.mydomain.com:/data/brick_nv me0n1 49153 0 Y 8396 Brick p1-nx-fsr0001-be.mydomain.com:/data/brick_nv me1n1 49154 0 Y 8407 Brick p1-nx-fsr0001-be.mydomain.com:/data/brick_nv me2n1 49155 0 Y 8422 Brick p1-nx-fsr0001-be.mydomain.com:/data/brick_nv me3n1 49156 0 Y 8432 Brick p1-nx-fsr0002-be.mydomain.com:/data/brick_nv me0n1 49153 0 Y 4171 Brick p1-nx-fsr0002-be.mydomain.com:/data/brick_nv me1n1 49154 0 Y 4179 Brick p1-nx-fsr0002-be.mydomain.com:/data/brick_nv me2n1 49155 0 Y 4187 Brick p1-nx-fsr0002-be.mydomain.com:/data/brick_nv me3n1 49156 0 Y 4197 Brick p1-nx-fsr0003-be.mydomain.com:/data/brick_nv me0n1 49153 0 Y 4360 Brick p1-nx-fsr0003-be.mydomain.com:/data/brick_nv me1n1 49154 0 Y 4381 Brick p1-nx-fsr0003-be.mydomain.com:/data/brick_nv me2n1 49155 0 Y 4379 Brick p1-nx-fsr0003-be.mydomain.com:/data/brick_nv me3n1 49156 0 Y 4380 Brick p1-nx-fsr0001-be.mydomain.com:/data/brick_nv me4n1 49157 0 Y 8451 Brick p1-nx-fsr0001-be.mydomain.com:/data/brick_nv me5n1 49158 0 Y 8468 Brick p1-nx-fsr0001-be.mydomain.com:/data/brick_nv me6n1 49159 0 Y 8473 Brick p1-nx-fsr0001-be.mydomain.com:/data/brick_nv me7n1 49160 0 Y 8485 Brick p1-nx-fsr0002-be.mydomain.com:/data/brick_nv me4n1 49157 0 Y 4207 Brick p1-nx-fsr0002-be.mydomain.com:/data/brick_nv me5n1 49158 0 Y 4222 Brick p1-nx-fsr0002-be.mydomain.com:/data/brick_nv me6n1 49159 0 Y 4230 Brick p1-nx-fsr0002-be.mydomain.com:/data/brick_nv me7n1 49160 0 Y 4237 Brick p1-nx-fsr0003-be.mydomain.com:/data/brick_nv me4n1 49157 0 Y 4404 Brick p1-nx-fsr0003-be.mydomain.com:/data/brick_nv me5n1 49158 0 Y 4411 Brick p1-nx-fsr0003-be.mydomain.com:/data/brick_nv me6n1 49159 0 Y 4430 Brick p1-nx-fsr0003-be.mydomain.com:/data/brick_nv me7n1 49160 0 Y 4443 Brick p1-nx-fsr0001-be.mydomain.com:/data/brick_nv me8n1 49161 0 Y 8505 Brick p1-nx-fsr0001-be.mydomain.com:/data/brick_nv me9n1 49162 0 Y 8515 Brick p1-nx-fsr0001-be.mydomain.com:/data/brick_nv me10n1 49163 0 Y 8527 Brick p1-nx-fsr0001-be.mydomain.com:/data/brick_nv me11n1 49164 0 Y 8548 Brick p1-nx-fsr0002-be.mydomain.com:/data/brick_nv me8n1 49161 0 Y 4249 Brick p1-nx-fsr0002-be.mydomain.com:/data/brick_nv me9n1 49162 0 Y 4261 Brick p1-nx-fsr0002-be.mydomain.com:/data/brick_nv me10n1 49163 0 Y 4274 Brick p1-nx-fsr0002-be.mydomain.com:/data/brick_nv me11n1 49164 0 Y 4286 Brick p1-nx-fsr0003-be.mydomain.com:/data/brick_nv me8n1 49161 0 Y 4459 Brick p1-nx-fsr0003-be.mydomain.com:/data/brick_nv me9n1 49162 0 Y 4458 Brick p1-nx-fsr0003-be.mydomain.com:/data/brick_nv me10n1 49163 0 Y 4482 Brick p1-nx-fsr0003-be.mydomain.com:/data/brick_nv me11n1 49164 0 Y 4497 Brick p1-nx-fsr0001-be.mydomain.com:/data/brick_nv me12n1 49165 0 Y 8557 Brick p1-nx-fsr0001-be.mydomain.com:/data/brick_nv me13n1 49166 0 Y 8569 Brick p1-nx-fsr0001-be.mydomain.com:/data/brick_nv me14n1 49167 0 Y 8584 Brick p1-nx-fsr0001-be.mydomain.com:/data/brick_nv me15n1 49168 0 Y 8593 Brick p1-nx-fsr0002-be.mydomain.com:/data/brick_nv me12n1 49165 0 Y 4293 Brick p1-nx-fsr0002-be.mydomain.com:/data/brick_nv me13n1 49166 0 Y 4305 Brick p1-nx-fsr0002-be.mydomain.com:/data/brick_nv me14n1 49167 0 Y 4324 Brick p1-nx-fsr0002-be.mydomain.com:/data/brick_nv me15n1 49168 0 Y 4336 Brick p1-nx-fsr0003-be.mydomain.com:/data/brick_nv me12n1 49165 0 Y 4511 Brick p1-nx-fsr0003-be.mydomain.com:/data/brick_nv me13n1 49166 0 Y 4522 Brick p1-nx-fsr0003-be.mydomain.com:/data/brick_nv me14n1 49167 0 Y 4532 Brick p1-nx-fsr0003-be.mydomain.com:/data/brick_nv me15n1 49168 0 Y 4549 Brick p1-nx-fsr0001-be.mydomain.com:/data/brick_nv me16n1 49169 0 Y 8607 Brick p1-nx-fsr0001-be.mydomain.com:/data/brick_nv me17n1 49170 0 Y 8617 Brick p1-nx-fsr0001-be.mydomain.com:/data/brick_nv me18n1 49171 0 Y 8644 Brick p1-nx-fsr0001-be.mydomain.com:/data/brick_nv me19n1 49172 0 Y 8652 Brick p1-nx-fsr0002-be.mydomain.com:/data/brick_nv me16n1 49169 0 Y 4344 Brick p1-nx-fsr0002-be.mydomain.com:/data/brick_nv me17n1 49170 0 Y 4360 Brick p1-nx-fsr0002-be.mydomain.com:/data/brick_nv me18n1 49171 0 Y 4372 Brick p1-nx-fsr0002-be.mydomain.com:/data/brick_nv me19n1 49172 0 Y 4383 Brick p1-nx-fsr0003-be.mydomain.com:/data/brick_nv me16n1 49169 0 Y 4566 Brick p1-nx-fsr0003-be.mydomain.com:/data/brick_nv me17n1 49170 0 Y 4572 Brick p1-nx-fsr0003-be.mydomain.com:/data/brick_nv me18n1 49171 0 Y 4589 Brick p1-nx-fsr0003-be.mydomain.com:/data/brick_nv me19n1 49172 0 Y 4602 Brick p1-nx-fsr0001-be.mydomain.com:/data/brick_nv me20n1 49173 0 Y 8663 Brick p1-nx-fsr0001-be.mydomain.com:/data/brick_nv me21n1 49174 0 Y 8692 Brick p1-nx-fsr0001-be.mydomain.com:/data/brick_nv me22n1 49175 0 Y 8694 Brick p1-nx-fsr0001-be.mydomain.com:/data/brick_nv me23n1 49176 0 Y 8716 Brick p1-nx-fsr0002-be.mydomain.com:/data/brick_nv me20n1 49173 0 Y 4404 Brick p1-nx-fsr0002-be.mydomain.com:/data/brick_nv me21n1 49174 0 Y 4407 Brick p1-nx-fsr0002-be.mydomain.com:/data/brick_nv me22n1 49175 0 Y 4427 Brick p1-nx-fsr0002-be.mydomain.com:/data/brick_nv me23n1 49176 0 Y 4444 Brick p1-nx-fsr0003-be.mydomain.com:/data/brick_nv me20n1 49173 0 Y 4610 Brick p1-nx-fsr0003-be.mydomain.com:/data/brick_nv me21n1 49174 0 Y 4624 Brick p1-nx-fsr0003-be.mydomain.com:/data/brick_nv me22n1 49175 0 Y 4633 Brick p1-nx-fsr0003-be.mydomain.com:/data/brick_nv me23n1 49176 0 Y 4643 Self-heal Daemon on localhost N/A N/A Y 4535 Quota Daemon on localhost N/A N/A Y 4834 Self-heal Daemon on p1-nx-fsr0003-be.mydomain.com N/A N/A Y 4707 Quota Daemon on p1-nx-fsr0003-be.mydomain.com N/A N/A Y 4887 Self-heal Daemon on p1-nx-fsr0001-be.mydomain.com N/A N/A Y 9000 Quota Daemon on p1-nx-fsr0001-be.mydomain.com N/A N/A Y 9513

Task Status of Volume nfs_volume

There are no active volume tasks

gluster volume heal nfs_volume info Brick p1-nx-fsr0001-be.mydomain.com:/data/brick_nvme0n1 Status: Connected Number of entries: 0

Brick p1-nx-fsr0001-be.mydomain.com:/data/brick_nvme1n1 Status: Connected Number of entries: 0

Brick p1-nx-fsr0001-be.mydomain.com:/data/brick_nvme2n1 Status: Connected Number of entries: 0

Brick p1-nx-fsr0001-be.mydomain.com:/data/brick_nvme3n1 Status: Connected Number of entries: 0

Brick p1-nx-fsr0002-be.mydomain.com:/data/brick_nvme0n1 Status: Connected Number of entries: 0

Brick p1-nx-fsr0002-be.mydomain.com:/data/brick_nvme1n1 Status: Connected Number of entries: 0

Brick p1-nx-fsr0002-be.mydomain.com:/data/brick_nvme2n1 Status: Connected Number of entries: 0

Brick p1-nx-fsr0002-be.mydomain.com:/data/brick_nvme3n1 Status: Connected Number of entries: 0

Brick p1-nx-fsr0003-be.mydomain.com:/data/brick_nvme0n1 Status: Connected Number of entries: 0

Brick p1-nx-fsr0003-be.mydomain.com:/data/brick_nvme1n1 Status: Connected Number of entries: 0

Brick p1-nx-fsr0003-be.mydomain.com:/data/brick_nvme2n1 Status: Connected Number of entries: 0

Brick p1-nx-fsr0003-be.mydomain.com:/data/brick_nvme3n1 Status: Connected Number of entries: 0

Brick p1-nx-fsr0001-be.mydomain.com:/data/brick_nvme4n1 Status: Connected Number of entries: 0

Brick p1-nx-fsr0001-be.mydomain.com:/data/brick_nvme5n1 Status: Connected Number of entries: 0

Brick p1-nx-fsr0001-be.mydomain.com:/data/brick_nvme6n1 Status: Connected Number of entries: 0

Brick p1-nx-fsr0001-be.mydomain.com:/data/brick_nvme7n1 Status: Connected Number of entries: 0

Brick p1-nx-fsr0002-be.mydomain.com:/data/brick_nvme4n1 Status: Connected Number of entries: 0

Brick p1-nx-fsr0002-be.mydomain.com:/data/brick_nvme5n1 Status: Connected Number of entries: 0

Brick p1-nx-fsr0002-be.mydomain.com:/data/brick_nvme6n1 Status: Connected Number of entries: 0

Brick p1-nx-fsr0002-be.mydomain.com:/data/brick_nvme7n1 Status: Connected Number of entries: 0

Brick p1-nx-fsr0003-be.mydomain.com:/data/brick_nvme4n1 Status: Connected Number of entries: 0

Brick p1-nx-fsr0003-be.mydomain.com:/data/brick_nvme5n1 Status: Connected Number of entries: 0

Brick p1-nx-fsr0003-be.mydomain.com:/data/brick_nvme6n1 Status: Connected Number of entries: 0

Brick p1-nx-fsr0003-be.mydomain.com:/data/brick_nvme7n1 Status: Connected Number of entries: 0

Brick p1-nx-fsr0001-be.mydomain.com:/data/brick_nvme8n1 Status: Connected Number of entries: 0

Brick p1-nx-fsr0001-be.mydomain.com:/data/brick_nvme9n1 Status: Connected Number of entries: 0

Brick p1-nx-fsr0001-be.mydomain.com:/data/brick_nvme10n1 Status: Connected Number of entries: 0

Brick p1-nx-fsr0001-be.mydomain.com:/data/brick_nvme11n1 Status: Connected Number of entries: 0

Brick p1-nx-fsr0002-be.mydomain.com:/data/brick_nvme8n1 Status: Connected Number of entries: 0

Brick p1-nx-fsr0002-be.mydomain.com:/data/brick_nvme9n1 Status: Connected Number of entries: 0

Brick p1-nx-fsr0002-be.mydomain.com:/data/brick_nvme10n1 Status: Connected Number of entries: 0

Brick p1-nx-fsr0002-be.mydomain.com:/data/brick_nvme11n1 Status: Connected Number of entries: 0

Brick p1-nx-fsr0003-be.mydomain.com:/data/brick_nvme8n1 Status: Connected Number of entries: 0

Brick p1-nx-fsr0003-be.mydomain.com:/data/brick_nvme9n1 Status: Connected Number of entries: 0

Brick p1-nx-fsr0003-be.mydomain.com:/data/brick_nvme10n1 Status: Connected Number of entries: 0

Brick p1-nx-fsr0003-be.mydomain.com:/data/brick_nvme11n1 Status: Connected Number of entries: 0

Brick p1-nx-fsr0001-be.mydomain.com:/data/brick_nvme12n1 Status: Connected Number of entries: 0

Brick p1-nx-fsr0001-be.mydomain.com:/data/brick_nvme13n1 Status: Connected Number of entries: 0

Brick p1-nx-fsr0001-be.mydomain.com:/data/brick_nvme14n1 Status: Connected Number of entries: 0

Brick p1-nx-fsr0001-be.mydomain.com:/data/brick_nvme15n1 Status: Connected Number of entries: 0

Brick p1-nx-fsr0002-be.mydomain.com:/data/brick_nvme12n1 Status: Connected Number of entries: 0

Brick p1-nx-fsr0002-be.mydomain.com:/data/brick_nvme13n1 Status: Connected Number of entries: 0

Brick p1-nx-fsr0002-be.mydomain.com:/data/brick_nvme14n1 Status: Connected Number of entries: 0

Brick p1-nx-fsr0002-be.mydomain.com:/data/brick_nvme15n1 Status: Connected Number of entries: 0

Brick p1-nx-fsr0003-be.mydomain.com:/data/brick_nvme12n1 Status: Connected Number of entries: 0

Brick p1-nx-fsr0003-be.mydomain.com:/data/brick_nvme13n1 Status: Connected Number of entries: 0

Brick p1-nx-fsr0003-be.mydomain.com:/data/brick_nvme14n1 Status: Connected Number of entries: 0

Brick p1-nx-fsr0003-be.mydomain.com:/data/brick_nvme15n1 Status: Connected Number of entries: 0

Brick p1-nx-fsr0001-be.mydomain.com:/data/brick_nvme16n1 Status: Connected Number of entries: 0

Brick p1-nx-fsr0001-be.mydomain.com:/data/brick_nvme17n1 Status: Connected Number of entries: 0

Brick p1-nx-fsr0001-be.mydomain.com:/data/brick_nvme18n1 Status: Connected Number of entries: 0

Brick p1-nx-fsr0001-be.mydomain.com:/data/brick_nvme19n1 Status: Connected Number of entries: 0

Brick p1-nx-fsr0002-be.mydomain.com:/data/brick_nvme16n1 Status: Connected Number of entries: 0

Brick p1-nx-fsr0002-be.mydomain.com:/data/brick_nvme17n1 Status: Connected Number of entries: 0

Brick p1-nx-fsr0002-be.mydomain.com:/data/brick_nvme18n1 Status: Connected Number of entries: 0

Brick p1-nx-fsr0002-be.mydomain.com:/data/brick_nvme19n1 Status: Connected Number of entries: 0

Brick p1-nx-fsr0003-be.mydomain.com:/data/brick_nvme16n1 Status: Connected Number of entries: 0

Brick p1-nx-fsr0003-be.mydomain.com:/data/brick_nvme17n1 Status: Connected Number of entries: 0

Brick p1-nx-fsr0003-be.mydomain.com:/data/brick_nvme18n1 Status: Connected Number of entries: 0

Brick p1-nx-fsr0003-be.mydomain.com:/data/brick_nvme19n1 Status: Connected Number of entries: 0

Brick p1-nx-fsr0001-be.mydomain.com:/data/brick_nvme20n1 Status: Connected Number of entries: 0

Brick p1-nx-fsr0001-be.mydomain.com:/data/brick_nvme21n1 Status: Connected Number of entries: 0

Brick p1-nx-fsr0001-be.mydomain.com:/data/brick_nvme22n1 Status: Connected Number of entries: 0

Brick p1-nx-fsr0001-be.mydomain.com:/data/brick_nvme23n1 Status: Connected Number of entries: 0

Brick p1-nx-fsr0002-be.mydomain.com:/data/brick_nvme20n1 Status: Connected Number of entries: 0

Brick p1-nx-fsr0002-be.mydomain.com:/data/brick_nvme21n1 Status: Connected Number of entries: 0

Brick p1-nx-fsr0002-be.mydomain.com:/data/brick_nvme22n1 Status: Connected Number of entries: 0

Brick p1-nx-fsr0002-be.mydomain.com:/data/brick_nvme23n1 Status: Connected Number of entries: 0

Brick p1-nx-fsr0003-be.mydomain.com:/data/brick_nvme20n1 Status: Connected Number of entries: 0

Brick p1-nx-fsr0003-be.mydomain.com:/data/brick_nvme21n1 Status: Connected Number of entries: 0

Brick p1-nx-fsr0003-be.mydomain.com:/data/brick_nvme22n1 Status: Connected Number of entries: 0

Brick p1-nx-fsr0003-be.mydomain.com:/data/brick_nvme23n1 Status: Connected Number of entries: 0

grep '] E [' /var/log/glusterfs/glusterd.log [2024-05-03 10:35:59.973676] E [MSGID: 106332] [glusterd-utils.c:13437:glusterd_get_value_for_vme_entry] 0-management: Failed to get option for transport.socket.own-thread key [2024-05-03 10:35:59.975461] E [MSGID: 101097] [xlator.c:218:xlator_volopt_dynload] 0-xlator: dlsym(xlator_api) missing: /usr/lib64/glusterfs/6.0/rpc-transport/socket.so: undefined symbol: xlator_api [2024-05-03 10:35:59.975477] E [MSGID: 106434] [glusterd-utils.c:13426:glusterd_get_value_for_vme_entry] 0-management: xlator_volopt_dynload error (-1) [2024-05-03 10:35:59.975485] E [MSGID: 101097] [xlator.c:218:xlator_volopt_dynload] 0-xlator: dlsym(xlator_api) missing: /usr/lib64/glusterfs/6.0/rpc-transport/socket.so: undefined symbol: xlator_api [2024-05-03 10:35:59.975488] E [MSGID: 106434] [glusterd-utils.c:13426:glusterd_get_value_for_vme_entry] 0-management: xlator_volopt_dynload error (-1) [2024-05-03 10:35:59.975493] E [MSGID: 101097] [xlator.c:218:xlator_volopt_dynload] 0-xlator: dlsym(xlator_api) missing: /usr/lib64/glusterfs/6.0/rpc-transport/socket.so: undefined symbol: xlator_api [2024-05-03 10:35:59.975495] E [MSGID: 106434] [glusterd-utils.c:13426:glusterd_get_value_for_vme_entry] 0-management: xlator_volopt_dynload error (-1) [2024-05-03 10:35:59.975500] E [MSGID: 101097] [xlator.c:218:xlator_volopt_dynload] 0-xlator: dlsym(xlator_api) missing: /usr/lib64/glusterfs/6.0/rpc-transport/socket.so: undefined symbol: xlator_api [2024-05-03 10:35:59.975503] E [MSGID: 106434] [glusterd-utils.c:13426:glusterd_get_value_for_vme_entry] 0-management: xlator_volopt_dynload error (-1) [2024-05-03 10:35:59.975507] E [MSGID: 101097] [xlator.c:218:xlator_volopt_dynload] 0-xlator: dlsym(xlator_api) missing: /usr/lib64/glusterfs/6.0/rpc-transport/socket.so: undefined symbol: xlator_api [2024-05-03 10:35:59.975510] E [MSGID: 106434] [glusterd-utils.c:13426:glusterd_get_value_for_vme_entry] 0-management: xlator_volopt_dynload error (-1) [2024-05-03 10:35:59.975515] E [MSGID: 101097] [xlator.c:218:xlator_volopt_dynload] 0-xlator: dlsym(xlator_api) missing: /usr/lib64/glusterfs/6.0/rpc-transport/socket.so: undefined symbol: xlator_api [2024-05-03 10:35:59.975517] E [MSGID: 106434] [glusterd-utils.c:13426:glusterd_get_value_for_vme_entry] 0-management: xlator_volopt_dynload error (-1) [2024-05-03 10:35:59.975522] E [MSGID: 101097] [xlator.c:218:xlator_volopt_dynload] 0-xlator: dlsym(xlator_api) missing: /usr/lib64/glusterfs/6.0/rpc-transport/socket.so: undefined symbol: xlator_api [2024-05-03 10:35:59.975524] E [MSGID: 106434] [glusterd-utils.c:13426:glusterd_get_value_for_vme_entry] 0-management: xlator_volopt_dynload error (-1) [2024-05-03 10:35:59.975529] E [MSGID: 101097] [xlator.c:218:xlator_volopt_dynload] 0-xlator: dlsym(xlator_api) missing: /usr/lib64/glusterfs/6.0/rpc-transport/socket.so: undefined symbol: xlator_api [2024-05-03 10:35:59.975532] E [MSGID: 106434] [glusterd-utils.c:13426:glusterd_get_value_for_vme_entry] 0-management: xlator_volopt_dynload error (-1) [2024-05-03 10:35:59.976201] E [MSGID: 106332] [glusterd-utils.c:13437:glusterd_get_value_for_vme_entry] 0-management: Failed to get option for limit-set key [2024-05-03 10:35:59.980767] E [MSGID: 106332] [glusterd-utils.c:13437:glusterd_get_value_for_vme_entry] 0-management: Failed to get option for nfs.transport-type key [2024-05-03 10:35:59.983032] E [MSGID: 106332] [glusterd-utils.c:13437:glusterd_get_value_for_vme_entry] 0-management: Failed to get option for nfs.log-level key [2024-05-03 10:35:59.986613] E [MSGID: 106332] [glusterd-utils.c:13437:glusterd_get_value_for_vme_entry] 0-management: Failed to get option for xattr-user-namespace-mode key [2024-05-03 10:35:59.989246] E [MSGID: 106434] [glusterd-utils.c:13426:glusterd_get_value_for_vme_entry] 0-management: xlator_volopt_dynload error (-1) [2024-05-03 10:35:59.989260] E [MSGID: 106332] [glusterd-utils.c:13437:glusterd_get_value_for_vme_entry] 0-management: Failed to get option for config key [2024-05-03 10:35:59.989268] E [MSGID: 106332] [glusterd-utils.c:13437:glusterd_get_value_for_vme_entry] 0-management: Failed to get option for config key [2024-05-03 10:35:59.989880] E [MSGID: 106332] [glusterd-utils.c:13437:glusterd_get_value_for_vme_entry] 0-management: Failed to get option for op-version key [2024-05-03 10:35:59.989901] E [MSGID: 106332] [glusterd-utils.c:13437:glusterd_get_value_for_vme_entry] 0-management: Failed to get option for max-op-version key [2024-05-03 10:35:59.991201] E [MSGID: 106434] [glusterd-utils.c:13426:glusterd_get_value_for_vme_entry] 0-management: xlator_volopt_dynload error (-1) [2024-05-03 10:35:59.991212] E [MSGID: 106434] [glusterd-utils.c:13426:glusterd_get_value_for_vme_entry] 0-management: xlator_volopt_dynload error (-1) [2024-05-03 10:35:59.995471] E [MSGID: 106332] [glusterd-utils.c:13437:glusterd_get_value_for_vme_entry] 0-management: Failed to get option for s3plugin-seckey key [2024-05-03 10:35:59.995543] E [MSGID: 106332] [glusterd-utils.c:13437:glusterd_get_value_for_vme_entry] 0-management: Failed to get option for s3plugin-keyid key [2024-05-03 10:35:59.995601] E [MSGID: 106332] [glusterd-utils.c:13437:glusterd_get_value_for_vme_entry] 0-management: Failed to get option for s3plugin-bucketid key [2024-05-03 10:35:59.995659] E [MSGID: 106332] [glusterd-utils.c:13437:glusterd_get_value_for_vme_entry] 0-management: Failed to get option for s3plugin-hostname key [2024-05-03 10:36:22.885397] E [MSGID: 106332] [glusterd-utils.c:13437:glusterd_get_value_for_vme_entry] 0-management: Failed to get option for nufa key [2024-05-03 10:36:22.886475] E [MSGID: 106332] [glusterd-utils.c:13437:glusterd_get_value_for_vme_entry] 0-management: Failed to get option for switch key [2024-05-03 10:36:22.890405] E [MSGID: 106332] [glusterd-utils.c:13437:glusterd_get_value_for_vme_entry] 0-management: Failed to get option for strict-readdir key [2024-05-03 10:36:22.893165] E [MSGID: 106332] [glusterd-utils.c:13437:glusterd_get_value_for_vme_entry] 0-management: Failed to get option for readdir-failover key [2024-05-03 10:36:22.897031] E [MSGID: 106332] [glusterd-utils.c:13437:glusterd_get_value_for_vme_entry] 0-management: Failed to get option for least-rate-limit key [2024-05-03 10:36:22.899258] E [MSGID: 106434] [glusterd-utils.c:13426:glusterd_get_value_for_vme_entry] 0-management: xlator_volopt_dynload error (-1) [2024-05-03 10:36:22.899269] E [MSGID: 106434] [glusterd-utils.c:13426:glusterd_get_value_for_vme_entry] 0-management: xlator_volopt_dynload error (-1) [2024-05-03 10:36:22.899278] E [MSGID: 106434] [glusterd-utils.c:13426:glusterd_get_value_for_vme_entry] 0-management: xlator_volopt_dynload error (-1) [2024-05-03 10:36:22.899803] E [MSGID: 106332] [glusterd-utils.c:13437:glusterd_get_value_for_vme_entry] 0-management: Failed to get option for transport.socket.own-thread key [2024-05-03 10:36:22.901527] E [MSGID: 101097] [xlator.c:218:xlator_volopt_dynload] 0-xlator: dlsym(xlator_api) missing: /usr/lib64/glusterfs/6.0/rpc-transport/socket.so: undefined symbol: xlator_api [2024-05-03 10:36:22.901536] E [MSGID: 106434] [glusterd-utils.c:13426:glusterd_get_value_for_vme_entry] 0-management: xlator_volopt_dynload error (-1) [2024-05-03 10:36:22.901546] E [MSGID: 101097] [xlator.c:218:xlator_volopt_dynload] 0-xlator: dlsym(xlator_api) missing: /usr/lib64/glusterfs/6.0/rpc-transport/socket.so: undefined symbol: xlator_api [2024-05-03 10:36:22.901549] E [MSGID: 106434] [glusterd-utils.c:13426:glusterd_get_value_for_vme_entry] 0-management: xlator_volopt_dynload error (-1) [2024-05-03 10:36:22.901554] E [MSGID: 101097] [xlator.c:218:xlator_volopt_dynload] 0-xlator: dlsym(xlator_api) missing: /usr/lib64/glusterfs/6.0/rpc-transport/socket.so: undefined symbol: xlator_api [2024-05-03 10:36:22.901557] E [MSGID: 106434] [glusterd-utils.c:13426:glusterd_get_value_for_vme_entry] 0-management: xlator_volopt_dynload error (-1) [2024-05-03 10:36:22.901561] E [MSGID: 101097] [xlator.c:218:xlator_volopt_dynload] 0-xlator: dlsym(xlator_api) missing: /usr/lib64/glusterfs/6.0/rpc-transport/socket.so: undefined symbol: xlator_api [2024-05-03 10:36:22.901564] E [MSGID: 106434] [glusterd-utils.c:13426:glusterd_get_value_for_vme_entry] 0-management: xlator_volopt_dynload error (-1) [2024-05-03 10:36:22.901569] E [MSGID: 101097] [xlator.c:218:xlator_volopt_dynload] 0-xlator: dlsym(xlator_api) missing: /usr/lib64/glusterfs/6.0/rpc-transport/socket.so: undefined symbol: xlator_api [2024-05-03 10:36:22.901571] E [MSGID: 106434] [glusterd-utils.c:13426:glusterd_get_value_for_vme_entry] 0-management: xlator_volopt_dynload error (-1) [2024-05-03 10:36:22.901576] E [MSGID: 101097] [xlator.c:218:xlator_volopt_dynload] 0-xlator: dlsym(xlator_api) missing: /usr/lib64/glusterfs/6.0/rpc-transport/socket.so: undefined symbol: xlator_api [2024-05-03 10:36:22.901579] E [MSGID: 106434] [glusterd-utils.c:13426:glusterd_get_value_for_vme_entry] 0-management: xlator_volopt_dynload error (-1) [2024-05-03 10:36:22.901583] E [MSGID: 101097] [xlator.c:218:xlator_volopt_dynload] 0-xlator: dlsym(xlator_api) missing: /usr/lib64/glusterfs/6.0/rpc-transport/socket.so: undefined symbol: xlator_api [2024-05-03 10:36:22.901586] E [MSGID: 106434] [glusterd-utils.c:13426:glusterd_get_value_for_vme_entry] 0-management: xlator_volopt_dynload error (-1) [2024-05-03 10:36:22.901591] E [MSGID: 101097] [xlator.c:218:xlator_volopt_dynload] 0-xlator: dlsym(xlator_api) missing: /usr/lib64/glusterfs/6.0/rpc-transport/socket.so: undefined symbol: xlator_api [2024-05-03 10:36:22.901593] E [MSGID: 106434] [glusterd-utils.c:13426:glusterd_get_value_for_vme_entry] 0-management: xlator_volopt_dynload error (-1) [2024-05-03 10:36:22.902237] E [MSGID: 106332] [glusterd-utils.c:13437:glusterd_get_value_for_vme_entry] 0-management: Failed to get option for limit-set key [2024-05-03 10:36:22.906725] E [MSGID: 106332] [glusterd-utils.c:13437:glusterd_get_value_for_vme_entry] 0-management: Failed to get option for nfs.transport-type key [2024-05-03 10:36:22.909475] E [MSGID: 106332] [glusterd-utils.c:13437:glusterd_get_value_for_vme_entry] 0-management: Failed to get option for nfs.log-level key [2024-05-03 10:36:22.913172] E [MSGID: 106332] [glusterd-utils.c:13437:glusterd_get_value_for_vme_entry] 0-management: Failed to get option for xattr-user-namespace-mode key [2024-05-03 10:36:22.915801] E [MSGID: 106434] [glusterd-utils.c:13426:glusterd_get_value_for_vme_entry] 0-management: xlator_volopt_dynload error (-1) [2024-05-03 10:36:22.915822] E [MSGID: 106332] [glusterd-utils.c:13437:glusterd_get_value_for_vme_entry] 0-management: Failed to get option for config key [2024-05-03 10:36:22.915831] E [MSGID: 106332] [glusterd-utils.c:13437:glusterd_get_value_for_vme_entry] 0-management: Failed to get option for config key [2024-05-03 10:36:22.916452] E [MSGID: 106332] [glusterd-utils.c:13437:glusterd_get_value_for_vme_entry] 0-management: Failed to get option for op-version key [2024-05-03 10:36:22.916466] E [MSGID: 106332] [glusterd-utils.c:13437:glusterd_get_value_for_vme_entry] 0-management: Failed to get option for max-op-version key [2024-05-03 10:36:22.917824] E [MSGID: 106434] [glusterd-utils.c:13426:glusterd_get_value_for_vme_entry] 0-management: xlator_volopt_dynload error (-1) [2024-05-03 10:36:22.917838] E [MSGID: 106434] [glusterd-utils.c:13426:glusterd_get_value_for_vme_entry] 0-management: xlator_volopt_dynload error (-1) [2024-05-03 10:36:22.922104] E [MSGID: 106332] [glusterd-utils.c:13437:glusterd_get_value_for_vme_entry] 0-management: Failed to get option for s3plugin-seckey key [2024-05-03 10:36:22.922176] E [MSGID: 106332] [glusterd-utils.c:13437:glusterd_get_value_for_vme_entry] 0-management: Failed to get option for s3plugin-keyid key [2024-05-03 10:36:22.922235] E [MSGID: 106332] [glusterd-utils.c:13437:glusterd_get_value_for_vme_entry] 0-management: Failed to get option for s3plugin-bucketid key [2024-05-03 10:36:22.922292] E [MSGID: 106332] [glusterd-utils.c:13437:glusterd_get_value_for_vme_entry] 0-management: Failed to get option for s3plugin-hostname key [2024-05-03 11:20:15.635887] E [MSGID: 106332] [glusterd-utils.c:13437:glusterd_get_value_for_vme_entry] 0-management: Failed to get option for nufa key [2024-05-03 11:20:15.636789] E [MSGID: 106332] [glusterd-utils.c:13437:glusterd_get_value_for_vme_entry] 0-management: Failed to get option for switch key [2024-05-03 11:20:15.640027] E [MSGID: 106332] [glusterd-utils.c:13437:glusterd_get_value_for_vme_entry] 0-management: Failed to get option for strict-readdir key [2024-05-03 11:20:15.643036] E [MSGID: 106332] [glusterd-utils.c:13437:glusterd_get_value_for_vme_entry] 0-management: Failed to get option for readdir-failover key [2024-05-03 11:20:15.646432] E [MSGID: 106332] [glusterd-utils.c:13437:glusterd_get_value_for_vme_entry] 0-management: Failed to get option for least-rate-limit key [2024-05-03 11:20:15.648956] E [MSGID: 106434] [glusterd-utils.c:13426:glusterd_get_value_for_vme_entry] 0-management: xlator_volopt_dynload error (-1) [2024-05-03 11:20:15.648970] E [MSGID: 106434] [glusterd-utils.c:13426:glusterd_get_value_for_vme_entry] 0-management: xlator_volopt_dynload error (-1) [2024-05-03 11:20:15.648983] E [MSGID: 106434] [glusterd-utils.c:13426:glusterd_get_value_for_vme_entry] 0-management: xlator_volopt_dynload error (-1) [2024-05-03 11:20:15.649665] E [MSGID: 106332] [glusterd-utils.c:13437:glusterd_get_value_for_vme_entry] 0-management: Failed to get option for transport.socket.own-thread key [2024-05-03 11:20:15.651461] E [MSGID: 101097] [xlator.c:218:xlator_volopt_dynload] 0-xlator: dlsym(xlator_api) missing: /usr/lib64/glusterfs/6.0/rpc-transport/socket.so: undefined symbol: xlator_api [2024-05-03 11:20:15.651471] E [MSGID: 106434] [glusterd-utils.c:13426:glusterd_get_value_for_vme_entry] 0-management: xlator_volopt_dynload error (-1) [2024-05-03 11:20:15.651477] E [MSGID: 101097] [xlator.c:218:xlator_volopt_dynload] 0-xlator: dlsym(xlator_api) missing: /usr/lib64/glusterfs/6.0/rpc-transport/socket.so: undefined symbol: xlator_api [2024-05-03 11:20:15.651480] E [MSGID: 106434] [glusterd-utils.c:13426:glusterd_get_value_for_vme_entry] 0-management: xlator_volopt_dynload error (-1) [2024-05-03 11:20:15.651485] E [MSGID: 101097] [xlator.c:218:xlator_volopt_dynload] 0-xlator: dlsym(xlator_api) missing: /usr/lib64/glusterfs/6.0/rpc-transport/socket.so: undefined symbol: xlator_api [2024-05-03 11:20:15.651488] E [MSGID: 106434] [glusterd-utils.c:13426:glusterd_get_value_for_vme_entry] 0-management: xlator_volopt_dynload error (-1) [2024-05-03 11:20:15.651492] E [MSGID: 101097] [xlator.c:218:xlator_volopt_dynload] 0-xlator: dlsym(xlator_api) missing: /usr/lib64/glusterfs/6.0/rpc-transport/socket.so: undefined symbol: xlator_api [2024-05-03 11:20:15.651495] E [MSGID: 106434] [glusterd-utils.c:13426:glusterd_get_value_for_vme_entry] 0-management: xlator_volopt_dynload error (-1) [2024-05-03 11:20:15.651500] E [MSGID: 101097] [xlator.c:218:xlator_volopt_dynload] 0-xlator: dlsym(xlator_api) missing: /usr/lib64/glusterfs/6.0/rpc-transport/socket.so: undefined symbol: xlator_api [2024-05-03 11:20:15.651502] E [MSGID: 106434] [glusterd-utils.c:13426:glusterd_get_value_for_vme_entry] 0-management: xlator_volopt_dynload error (-1) [2024-05-03 11:20:15.651507] E [MSGID: 101097] [xlator.c:218:xlator_volopt_dynload] 0-xlator: dlsym(xlator_api) missing: /usr/lib64/glusterfs/6.0/rpc-transport/socket.so: undefined symbol: xlator_api [2024-05-03 11:20:15.651510] E [MSGID: 106434] [glusterd-utils.c:13426:glusterd_get_value_for_vme_entry] 0-management: xlator_volopt_dynload error (-1) [2024-05-03 11:20:15.651514] E [MSGID: 101097] [xlator.c:218:xlator_volopt_dynload] 0-xlator: dlsym(xlator_api) missing: /usr/lib64/glusterfs/6.0/rpc-transport/socket.so: undefined symbol: xlator_api [2024-05-03 11:20:15.651517] E [MSGID: 106434] [glusterd-utils.c:13426:glusterd_get_value_for_vme_entry] 0-management: xlator_volopt_dynload error (-1) [2024-05-03 11:20:15.651525] E [MSGID: 101097] [xlator.c:218:xlator_volopt_dynload] 0-xlator: dlsym(xlator_api) missing: /usr/lib64/glusterfs/6.0/rpc-transport/socket.so: undefined symbol: xlator_api [2024-05-03 11:20:15.651527] E [MSGID: 106434] [glusterd-utils.c:13426:glusterd_get_value_for_vme_entry] 0-management: xlator_volopt_dynload error (-1) [2024-05-03 11:20:15.652129] E [MSGID: 106332] [glusterd-utils.c:13437:glusterd_get_value_for_vme_entry] 0-management: Failed to get option for limit-set key [2024-05-03 11:20:15.656638] E [MSGID: 106332] [glusterd-utils.c:13437:glusterd_get_value_for_vme_entry] 0-management: Failed to get option for nfs.transport-type key [2024-05-03 11:20:15.658884] E [MSGID: 106332] [glusterd-utils.c:13437:glusterd_get_value_for_vme_entry] 0-management: Failed to get option for nfs.log-level key [2024-05-03 11:20:15.662529] E [MSGID: 106332] [glusterd-utils.c:13437:glusterd_get_value_for_vme_entry] 0-management: Failed to get option for xattr-user-namespace-mode key [2024-05-03 11:20:15.665164] E [MSGID: 106434] [glusterd-utils.c:13426:glusterd_get_value_for_vme_entry] 0-management: xlator_volopt_dynload error (-1) [2024-05-03 11:20:15.665183] E [MSGID: 106332] [glusterd-utils.c:13437:glusterd_get_value_for_vme_entry] 0-management: Failed to get option for config key [2024-05-03 11:20:15.665190] E [MSGID: 106332] [glusterd-utils.c:13437:glusterd_get_value_for_vme_entry] 0-management: Failed to get option for config key [2024-05-03 11:20:15.665802] E [MSGID: 106332] [glusterd-utils.c:13437:glusterd_get_value_for_vme_entry] 0-management: Failed to get option for op-version key [2024-05-03 11:20:15.665817] E [MSGID: 106332] [glusterd-utils.c:13437:glusterd_get_value_for_vme_entry] 0-management: Failed to get option for max-op-version key [2024-05-03 11:20:15.667074] E [MSGID: 106434] [glusterd-utils.c:13426:glusterd_get_value_for_vme_entry] 0-management: xlator_volopt_dynload error (-1) [2024-05-03 11:20:15.667087] E [MSGID: 106434] [glusterd-utils.c:13426:glusterd_get_value_for_vme_entry] 0-management: xlator_volopt_dynload error (-1) [2024-05-03 11:20:15.671348] E [MSGID: 106332] [glusterd-utils.c:13437:glusterd_get_value_for_vme_entry] 0-management: Failed to get option for s3plugin-seckey key [2024-05-03 11:20:15.671422] E [MSGID: 106332] [glusterd-utils.c:13437:glusterd_get_value_for_vme_entry] 0-management: Failed to get option for s3plugin-keyid key [2024-05-03 11:20:15.671480] E [MSGID: 106332] [glusterd-utils.c:13437:glusterd_get_value_for_vme_entry] 0-management: Failed to get option for s3plugin-bucketid key [2024-05-03 11:20:15.671537] E [MSGID: 106332] [glusterd-utils.c:13437:glusterd_get_value_for_vme_entry] 0-management: Failed to get option for s3plugin-hostname key [2024-05-03 12:00:23.329796] E [MSGID: 106301] [glusterd-syncop.c:1322:gd_stage_op_phase] 0-management: Staging of operation 'Volume Heal' failed on localhost : This command is supported for only volume of replicated type. Volume nfs_volume is not of type replicate

*** Additional information ***

cat /etc/os-release NAME="Red Hat Enterprise Linux" VERSION="8.8 (Ootpa)"

rpm -qa | grep gluster glusterfs-selinux-1.0-5.el8rhgs.noarch glusterfs-libs-6.0-63.el8rhgs.x86_64 glusterfs-cli-6.0-63.el8rhgs.x86_64 glusterfs-server-6.0-63.el8rhgs.x86_64 glusterfs-ganesha-6.0-63.el8rhgs.x86_64 glusterfs-client-xlators-6.0-63.el8rhgs.x86_64 glusterfs-geo-replication-6.0-63.el8rhgs.x86_64 python3-gluster-6.0-63.el8rhgs.x86_64 glusterfs-api-6.0-63.el8rhgs.x86_64 glusterfs-fuse-6.0-63.el8rhgs.x86_64 glusterfs-6.0-63.el8rhgs.x86_64 nfs-ganesha-gluster-3.4-10.el8rhgs.x86_64

May 03 '24 12:05 alvic2000

FYI, rghs will be EOL December of this year. I would consider migrating to the latest stable maintained version of 10.x or 11.x. I am not sure if the version you currently use (6.0-63) has any of the current bug fixes backported to it.

Similarly related issue: https://github.com/gluster/glusterfs/issues/4157

May 03 '24 17:05 anon314159

@alvic2000 Are you still facing this issue? I saw this issue on zfs when a replace-brick happens. But once replace-brick completes, things come back to normalcy. Could you post the output of all the 72 disks df -i output? In one disperse/ec group, max(available nodes) is taken as total inodes and min(free inodes) is taken as free inodes It makes more sense to do min in both cases as per my understanding.

Also want to fix if I understand the issue you are seeing on this

Feb 10 '25 10:02 pranithk