index: Optimize index xlator code to improve file creation

[mohit84]

The afr/ec use xattrop internal fop to maintain consistency for data transactions. As index xlator got a xattrop wind request it takes action to add/delete gfid path under indices directory specific to gfid. In case of indices add, index xlator call sys_link and in case of delete it call sys_rename and sys_unlink based on the index type.

Solution: The index xlator uses directory xattrop,entry-changes,dirty under indices(.glusterfs/indices) to link/unlink gfids so keep fd open for the index parent directories and use at based system calls(linkat,renameat,fstatat,unlinkat,openat) to improve the performance.

Note: For data please refer the link https://github.com/gluster/glusterfs/issues/3606#issuecomment-1161730428

Fixes: #3606 Change-Id: Iac13fbcde84744f4981ed94b6ecddddd51219586 Signed-off-by: Mohit Agrawal [email protected]

[mohit84]

/run regression

[xhernandez]

@mohit84 do you know what's the major contributor to the performance improvement ? at functions or the removal of the lock ?

[mohit84]

@mohit84 do you know what's the major contributor to the performance improvement ? at functions or the removal of the lock ?

at based call are major improvement, i am trying to change other code path in the index xlator. I will upload a new patch.

[mohit84]

/run regression

[mohit84]

/run regression

[mohit84]

/run regression

[gluster-ant]

1 test(s) failed ./tests/basic/afr/add-brick-self-heal-non-granular.t

0 test(s) generated core

7 test(s) needed retry ./tests/000-flaky/basic_afr_split-brain-favorite-child-policy.t ./tests/000-flaky/basic_ec_ec-quorum-count-partial-failure.t ./tests/000-flaky/basic_mount-nfs-auth.t ./tests/000-flaky/bugs_nfs_bug-1116503.t ./tests/000-flaky/glusterd-restart-shd-mux.t ./tests/00-geo-rep/01-georep-glusterd-tests.t ./tests/basic/afr/add-brick-self-heal-non-granular.t

5 flaky test(s) marked as success even though they failed ./tests/000-flaky/basic_afr_split-brain-favorite-child-policy.t ./tests/000-flaky/basic_ec_ec-quorum-count-partial-failure.t ./tests/000-flaky/basic_mount-nfs-auth.t ./tests/000-flaky/bugs_nfs_bug-1116503.t ./tests/000-flaky/glusterd-restart-shd-mux.t https://build.gluster.org/job/gh_centos7-regression/2589/

[mohit84]

/run full regression

[gluster-ant]

99 test(s) failed ./tests/basic/afr/add-brick-self-heal-non-granular.t ./tests/basic/afr/add-brick-self-heal.t ./tests/basic/afr/afr-anon-inode.t ./tests/basic/afr/arbiter-add-brick.t ./tests/basic/afr/bug-1130892-non-granular.t ./tests/basic/afr/bug-1493415-gfid-heal-non-granular.t ./tests/basic/afr/bug-1722507-type-mismatch-error-handling-non-granular.t ./tests/basic/afr/bug-1749322-entry-heal-not-happening-non-granular.t ./tests/basic/afr/client-side-heal.t ./tests/basic/afr/data-self-heal.t ./tests/basic/afr/durability-off.t ./tests/basic/afr/entry-self-heal-anon-dir-off.t ./tests/basic/afr/entry-self-heal.t ./tests/basic/afr/gfid-mismatch-resolution-with-cli.t ./tests/basic/afr/gfid-mismatch-resolution-with-fav-child-policy.t ./tests/basic/afr/granular-esh/add-brick.t ./tests/basic/afr/granular-esh/cli.t ./tests/basic/afr/granular-esh/conservative-merge.t ./tests/basic/afr/granular-esh/granular-esh.t ./tests/basic/afr/granular-esh/granular-indices-but-non-granular-heal.t ./tests/basic/afr/granular-esh/replace-brick.t ./tests/basic/afr/metadata-self-heal.t ./tests/basic/afr/replace-brick-self-heal-non-granular.t ./tests/basic/afr/replace-brick-self-heal.t ./tests/basic/afr/self-heald.t ./tests/basic/afr/self-heal.t ./tests/basic/afr/sparse-file-self-heal.t ./tests/basic/afr/split-brain-favorite-child-policy-client-side-healing.t ./tests/basic/afr/split-brain-heal-info.t ./tests/basic/afr/split-brain-healing-ctime.t ./tests/basic/afr/split-brain-healing.t ./tests/basic/afr/ta-shd.t ./tests/basic/ctime/ctime-ec-heal.t ./tests/basic/ctime/ctime-rep-heal.t ./tests/basic/ec/ec-1468261.t ./tests/basic/ec/ec-background-heals.t ./tests/basic/ec/ec-data-heal.t ./tests/basic/ec/ec-new-entry.t ./tests/basic/ec/ec-optimistic-changelog.t ./tests/basic/ec/ec-quorum-count.t ./tests/basic/ec/ec-reset-brick.t ./tests/basic/ec/ec.t ./tests/basic/ec/gfapi-ec-open-truncate.t ./tests/basic/ec/self-heal-read-write-fail.t ./tests/basic/shd-mux-afr.t ./tests/basic/shd-mux-ec.t ./tests/bugs/changelog/bug-1321955.t ./tests/bugs/cli/bug-1320388.t ./tests/bugs/core/bug-1402841.t-mt-dir-scan-race.t ./tests/bugs/ec/bug-1251446.t ./tests/bugs/ec/bug-1547662.t ./tests/bugs/gfapi/bug-1032894.t ./tests/bugs/glusterfs/bug-861015-index.t ./tests/bugs/glusterfs/bug-866459.t ./tests/bugs/glusterfs/bug-906646.t ./tests/bugs/index/bug-1559004-EMLINK-handling.t ./tests/bugs/quota/afr-quota-xattr-mdata-heal.t ./tests/bugs/quota/bug-1178130.t ./tests/bugs/replicate/bug-1101647.t ./tests/bugs/replicate/bug-1130892.t ./tests/bugs/replicate/bug-1134691-afr-lookup-metadata-heal.t ./tests/bugs/replicate/bug-1139230.t ./tests/bugs/replicate/bug-1180545.t ./tests/bugs/replicate/bug-1190069-afr-stale-index-entries.t ./tests/bugs/replicate/bug-1238508-self-heal.t ./tests/bugs/replicate/bug-1266876-allow-reset-brick-for-same-path.t ./tests/bugs/replicate/bug-1297695.t ./tests/bugs/replicate/bug-1341650.t ./tests/bugs/replicate/bug-1363721.t ./tests/bugs/replicate/bug-1365455.t ./tests/bugs/replicate/bug-1402730.t ./tests/bugs/replicate/bug-1408712.t ./tests/bugs/replicate/bug-1477169-entry-selfheal-rename.t ./tests/bugs/replicate/bug-1493415-gfid-heal.t ./tests/bugs/replicate/bug-1539358-split-brain-detection.t ./tests/bugs/replicate/bug-1591193-assign-gfid-and-heal.t ./tests/bugs/replicate/bug-1626994-info-split-brain.t ./tests/bugs/replicate/bug-1637802-arbiter-stale-data-heal-lock.t ./tests/bugs/replicate/bug-1655050-dir-sbrain-size-policy.t ./tests/bugs/replicate/bug-1655052-sbrain-policy-same-size.t ./tests/bugs/replicate/bug-1655854-support-dist-to-rep3-arb-conversion.t ./tests/bugs/replicate/bug-1686568-send-truncate-on-arbiter-from-shd.t ./tests/bugs/replicate/bug-1717819-metadata-split-brain-detection.t ./tests/bugs/replicate/bug-1722507-type-mismatch-error-handling.t ./tests/bugs/replicate/bug-1744548-heal-timeout.t ./tests/bugs/replicate/bug-1749322-entry-heal-not-happening.t ./tests/bugs/replicate/bug-1756938-replica-3-sbrain-cli.t ./tests/bugs/replicate/bug-1761531-metadata-heal-restore-time.t ./tests/bugs/replicate/bug-1801624-entry-heal.t ./tests/bugs/replicate/bug-957877.t ./tests/bugs/replicate/bug-977797.t ./tests/bugs/replicate/bug-986905.t ./tests/bugs/replicate/do-not-reopen-fd.t ./tests/bugs/replicate/issue-3288-dir-gfid-mismatch-heal-option.t ./tests/bugs/replicate/mdata-heal-no-xattrs.t ./tests/bugs/shard/bug-1248887.t ./tests/bugs/unclassified/bug-874498.t ./tests/features/index/index-link-count-lifecycle.t ./tests/features/worm_sh.t

0 test(s) generated core

106 test(s) needed retry ./tests/000-flaky/basic_afr_split-brain-favorite-child-policy.t ./tests/000-flaky/basic_ec_ec-quorum-count-partial-failure.t ./tests/000-flaky/basic_mount-nfs-auth.t ./tests/000-flaky/bugs_nfs_bug-1116503.t ./tests/000-flaky/glusterd-restart-shd-mux.t ./tests/basic/afr/add-brick-self-heal-non-granular.t ./tests/basic/afr/add-brick-self-heal.t ./tests/basic/afr/afr-anon-inode.t ./tests/basic/afr/arbiter-add-brick.t ./tests/basic/afr/bug-1130892-non-granular.t ./tests/basic/afr/bug-1493415-gfid-heal-non-granular.t ./tests/basic/afr/bug-1722507-type-mismatch-error-handling-non-granular.t ./tests/basic/afr/bug-1749322-entry-heal-not-happening-non-granular.t ./tests/basic/afr/client-side-heal.t ./tests/basic/afr/data-self-heal.t ./tests/basic/afr/durability-off.t ./tests/basic/afr/entry-self-heal-anon-dir-off.t ./tests/basic/afr/entry-self-heal.t ./tests/basic/afr/gfid-mismatch-resolution-with-cli.t ./tests/basic/afr/gfid-mismatch-resolution-with-fav-child-policy.t ./tests/basic/afr/granular-esh/add-brick.t ./tests/basic/afr/granular-esh/cli.t ./tests/basic/afr/granular-esh/conservative-merge.t ./tests/basic/afr/granular-esh/granular-esh.t ./tests/basic/afr/granular-esh/granular-indices-but-non-granular-heal.t ./tests/basic/afr/granular-esh/replace-brick.t ./tests/basic/afr/metadata-self-heal.t ./tests/basic/afr/replace-brick-self-heal-non-granular.t ./tests/basic/afr/replace-brick-self-heal.t ./tests/basic/afr/self-heald.t ./tests/basic/afr/self-heal.t ./tests/basic/afr/sparse-file-self-heal.t ./tests/basic/afr/split-brain-favorite-child-policy-client-side-healing.t ./tests/basic/afr/split-brain-heal-info.t ./tests/basic/afr/split-brain-healing-ctime.t ./tests/basic/afr/split-brain-healing.t ./tests/basic/afr/ta-shd.t ./tests/basic/ctime/ctime-ec-heal.t ./tests/basic/ctime/ctime-rep-heal.t ./tests/basic/ec/ec-1468261.t ./tests/basic/ec/ec-background-heals.t ./tests/basic/ec/ec-data-heal.t ./tests/basic/ec/ec-new-entry.t ./tests/basic/ec/ec-optimistic-changelog.t ./tests/basic/ec/ec-quorum-count.t ./tests/basic/ec/ec-reset-brick.t ./tests/basic/ec/ec.t ./tests/basic/ec/gfapi-ec-open-truncate.t ./tests/basic/ec/self-heal-read-write-fail.t ./tests/basic/shd-mux-afr.t ./tests/basic/shd-mux-ec.t ./tests/bugs/bitrot/bug-1227996.t ./tests/bugs/changelog/bug-1321955.t ./tests/bugs/cli/bug-1320388.t ./tests/bugs/core/bug-1402841.t-mt-dir-scan-race.t ./tests/bugs/ec/bug-1251446.t ./tests/bugs/ec/bug-1547662.t ./tests/bugs/gfapi/bug-1032894.t ./tests/bugs/glusterfs/bug-861015-index.t ./tests/bugs/glusterfs/bug-866459.t ./tests/bugs/glusterfs/bug-906646.t ./tests/bugs/index/bug-1559004-EMLINK-handling.t ./tests/bugs/quota/afr-quota-xattr-mdata-heal.t ./tests/bugs/quota/bug-1178130.t ./tests/bugs/replicate/bug-1101647.t ./tests/bugs/replicate/bug-1130892.t ./tests/bugs/replicate/bug-1134691-afr-lookup-metadata-heal.t ./tests/bugs/replicate/bug-1139230.t ./tests/bugs/replicate/bug-1180545.t ./tests/bugs/replicate/bug-1190069-afr-stale-index-entries.t ./tests/bugs/replicate/bug-1238508-self-heal.t ./tests/bugs/replicate/bug-1266876-allow-reset-brick-for-same-path.t ./tests/bugs/replicate/bug-1297695.t ./tests/bugs/replicate/bug-1341650.t ./tests/bugs/replicate/bug-1363721.t ./tests/bugs/replicate/bug-1365455.t ./tests/bugs/replicate/bug-1402730.t ./tests/bugs/replicate/bug-1408712.t ./tests/bugs/replicate/bug-1477169-entry-selfheal-rename.t ./tests/bugs/replicate/bug-1493415-gfid-heal.t ./tests/bugs/replicate/bug-1539358-split-brain-detection.t ./tests/bugs/replicate/bug-1591193-assign-gfid-and-heal.t ./tests/bugs/replicate/bug-1626994-info-split-brain.t ./tests/bugs/replicate/bug-1637802-arbiter-stale-data-heal-lock.t ./tests/bugs/replicate/bug-1655050-dir-sbrain-size-policy.t ./tests/bugs/replicate/bug-1655052-sbrain-policy-same-size.t ./tests/bugs/replicate/bug-1655854-support-dist-to-rep3-arb-conversion.t ./tests/bugs/replicate/bug-1686568-send-truncate-on-arbiter-from-shd.t ./tests/bugs/replicate/bug-1717819-metadata-split-brain-detection.t ./tests/bugs/replicate/bug-1722507-type-mismatch-error-handling.t ./tests/bugs/replicate/bug-1734370-entry-heal-restore-time.t ./tests/bugs/replicate/bug-1744548-heal-timeout.t ./tests/bugs/replicate/bug-1749322-entry-heal-not-happening.t ./tests/bugs/replicate/bug-1756938-replica-3-sbrain-cli.t ./tests/bugs/replicate/bug-1761531-metadata-heal-restore-time.t ./tests/bugs/replicate/bug-1801624-entry-heal.t ./tests/bugs/replicate/bug-957877.t ./tests/bugs/replicate/bug-977797.t ./tests/bugs/replicate/bug-986905.t ./tests/bugs/replicate/do-not-reopen-fd.t ./tests/bugs/replicate/issue-3288-dir-gfid-mismatch-heal-option.t ./tests/bugs/replicate/mdata-heal-no-xattrs.t ./tests/bugs/shard/bug-1248887.t ./tests/bugs/unclassified/bug-874498.t ./tests/features/index/index-link-count-lifecycle.t ./tests/features/worm_sh.t

5 flaky test(s) marked as success even though they failed ./tests/000-flaky/basic_afr_split-brain-favorite-child-policy.t ./tests/000-flaky/basic_ec_ec-quorum-count-partial-failure.t ./tests/000-flaky/basic_mount-nfs-auth.t ./tests/000-flaky/bugs_nfs_bug-1116503.t ./tests/000-flaky/glusterd-restart-shd-mux.t

[mohit84]

/run regression

[mohit84]

/run regression

[mohit84]

/run regression

[gluster-ant]

1 test(s) failed ./tests/basic/afr/entry-self-heal-anon-dir-off.t

0 test(s) generated core

7 test(s) needed retry ./tests/000-flaky/basic_afr_split-brain-favorite-child-policy.t ./tests/000-flaky/basic_ec_ec-quorum-count-partial-failure.t ./tests/000-flaky/basic_mount-nfs-auth.t ./tests/000-flaky/bugs_core_multiplex-limit-issue-151.t ./tests/000-flaky/bugs_nfs_bug-1116503.t ./tests/000-flaky/glusterd-restart-shd-mux.t ./tests/basic/afr/entry-self-heal-anon-dir-off.t

2 flaky test(s) marked as success even though they failed ./tests/000-flaky/basic_mount-nfs-auth.t ./tests/000-flaky/bugs_nfs_bug-1116503.t https://build.gluster.org/job/gh_centos7-regression/2603/

[mohit84]

/run full regression

[gluster-ant]

10 test(s) failed ./tests/basic/afr/entry-self-heal-anon-dir-off.t ./tests/basic/afr/entry-self-heal.t ./tests/bugs/core/bug-1650403.t ./tests/bugs/index/bug-1559004-EMLINK-handling.t ./tests/bugs/replicate/bug-1101647.t ./tests/bugs/replicate/bug-1539358-split-brain-detection.t ./tests/bugs/replicate/bug-1626994-info-split-brain.t ./tests/bugs/replicate/bug-1717819-metadata-split-brain-detection.t ./tests/bugs/replicate/mdata-heal-no-xattrs.t ./tests/features/index/index-link-count-lifecycle.t

0 test(s) generated core

13 test(s) needed retry ./tests/000-flaky/basic_mount-nfs-auth.t ./tests/000-flaky/bugs_nfs_bug-1116503.t ./tests/basic/afr/entry-self-heal-anon-dir-off.t ./tests/basic/afr/entry-self-heal.t ./tests/bugs/core/bug-1650403.t ./tests/bugs/glusterd/brick-order-check-add-brick.t ./tests/bugs/index/bug-1559004-EMLINK-handling.t ./tests/bugs/replicate/bug-1101647.t ./tests/bugs/replicate/bug-1539358-split-brain-detection.t ./tests/bugs/replicate/bug-1626994-info-split-brain.t ./tests/bugs/replicate/bug-1717819-metadata-split-brain-detection.t ./tests/bugs/replicate/mdata-heal-no-xattrs.t ./tests/features/index/index-link-count-lifecycle.t

2 flaky test(s) marked as success even though they failed ./tests/000-flaky/basic_mount-nfs-auth.t ./tests/000-flaky/bugs_nfs_bug-1116503.t

[stale[bot]]

Thank you for your contributions. Noticed that this issue is not having any activity in last ~6 months! We are marking this issue as stale because it has not had recent activity. It will be closed in 2 weeks if no one responds with a comment here.

[stale[bot]]

Closing this issue as there was no update since my last update on issue. If this is an issue which is still valid, feel free to open it.

[stale[bot]]

Closing this issue as there was no update since my last update on issue. If this is an issue which is still valid, feel free to open it.