DAOS-14908 vos: Reduce aggregation conflicts

[jolivier23]

Rather than blocking vos_obj_discard entirely when discard or aggregation are running, let's block it only when there is an actual conflict on the object being discarded.

Features: rebuild

Required-githooks: true

Change-Id: I110dd2e37e299df25c002bba63776559d689b1cf

Before requesting gatekeeper:

• [ ] Two review approvals and any prior change requests have been resolved.
• [ ] Testing is complete and all tests passed or there is a reason documented in the PR why it should be force landed and forced-landing tag is set.
• [ ] Features: (or Test-tag*) commit pragma was used or there is a reason documented that there are no appropriate tags for this PR.
• [ ] Commit messages follows the guidelines outlined here.
• [ ] Any tests skipped by the ticket being addressed have been run and passed in the PR.

Gatekeeper:

• [ ] You are the appropriate gatekeeper to be landing the patch.
• [ ] The PR has 2 reviews by people familiar with the code, including appropriate owners.
• [ ] Githooks were used. If not, request that user install them and check copyright dates.
• [ ] Checkpatch issues are resolved. Pay particular attention to ones that will show up on future PRs.
• [ ] All builds have passed. Check non-required builds for any new compiler warnings.
• [ ] Sufficient testing is done. Check feature pragmas and test tags and that tests skipped for the ticket are run and now pass with the changes.
• [ ] If applicable, the PR has addressed any potential version compatibility issues.
• [ ] Check the target branch. If it is master branch, should the PR go to a feature branch? If it is a release branch, does it have merge approval in the JIRA ticket.
• [ ] Extra checks if forced landing is requested
  • [ ] Review comments are sufficiently resolved, particularly by prior reviewers that requested changes.
  • [ ] No new NLT or valgrind warnings. Check the classic view.
  • [ ] Quick-build or Quick-functional is not used.
• [ ] Fix the commit message upon landing. Check the standard here. Edit it to create a single commit. If necessary, ask submitter for a new summary.

[github-actions[bot]]

Ticket title is 'soak - online harassers with 2.4.1-2 - rebuild took over 30 minutes to complete' Status is 'In Review' Labels: '2.4.2rc1,2.6.0,2.6.0tb1,scrubbed,soak,triaged' https://daosio.atlassian.net/browse/DAOS-14908

[daosbuild1]

Test stage Functional on EL 8.8 completed with status FAILURE. https://build.hpdd.intel.com//job/daos-stack/job/daos/view/change-requests/job/PR-14143/1/execution/node/1198/log

[daosbuild1]

Test stage Functional on EL 8.8 completed with status FAILURE. https://build.hpdd.intel.com//job/daos-stack/job/daos/view/change-requests/job/PR-14143/2/execution/node/1198/log

[jolivier23]

It looks to me that "obj discard" is only used in two cases today:

1. Discard old data on the target to be reintegrated before full reintegration. I think it's not quite correct, and Di has a pending PR tries to use vos container destroy instead of "obj discard".
2. Once the target is reintegrated, delete unused data on source target to reclaim space. I don't quite see why 'obj discard' is used in this case, can't we just delete the whole obj tree to reclaim space?

I think we review the rebuild/reintegration design first to see if "obj discard" is still relevant, what do you think?

Discussed in our last call but just to summarize the discussion.

For reintegration, container destroy/recreate precludes the need for vos_obj_discard.

For reclaim, I think it's still needed but can potentially be simplified to do as you say and delete the whole object tree. We can forego visibility checks. We still need to avoid conflict with aggregation...

Random thought, would it make any sense to get rid of vos_obj_discard and instead add an upcall in aggregation (should_remove(oid)) ?

[daosbuild1]

Test stage Functional on EL 8.8 completed with status FAILURE. https://build.hpdd.intel.com//job/daos-stack/job/daos/view/change-requests/job/PR-14143/3/execution/node/1174/log

[daosbuild1]

Test stage Functional Hardware Medium completed with status FAILURE. https://build.hpdd.intel.com//job/daos-stack/job/daos/view/change-requests/job/PR-14143/3/execution/node/1415/log

[daosbuild1]

Test stage Functional Hardware Medium Verbs Provider completed with status FAILURE. https://build.hpdd.intel.com//job/daos-stack/job/daos/view/change-requests/job/PR-14143/3/execution/node/1508/log

[daosbuild1]

Test stage Functional Hardware Large completed with status FAILURE. https://build.hpdd.intel.com//job/daos-stack/job/daos/view/change-requests/job/PR-14143/3/execution/node/1601/log

[NiuYawei]

It looks to me that "obj discard" is only used in two cases today:

1. Discard old data on the target to be reintegrated before full reintegration. I think it's not quite correct, and Di has a pending PR tries to use vos container destroy instead of "obj discard".
2. Once the target is reintegrated, delete unused data on source target to reclaim space. I don't quite see why 'obj discard' is used in this case, can't we just delete the whole obj tree to reclaim space?

I think we review the rebuild/reintegration design first to see if "obj discard" is still relevant, what do you think?

Discussed in our last call but just to summarize the discussion.

For reintegration, container destroy/recreate precludes the need for vos_obj_discard.

For reclaim, I think it's still needed but can potentially be simplified to do as you say and delete the whole object tree. We can forego visibility checks. We still need to avoid conflict with aggregation...

Random thought, would it make any sense to get rid of vos_obj_discard and instead add an upcall in aggregation (should_remove(oid)) ?

I asked Di about "obj discard" use cases, looks so far it's used in three cases:

1. Discard old data on the target being integrated before reintegration, this could be removed since Di has pushed a PR tries to discard old data by VOS container destroy.
2. Reclaim space by discarding the old data on source targets after reintegration, this obj discard requires only discard epoch [A, MAX), I think could be replaced by some more efficient new API to delete the whole oid tree.
3. When rebuild fail then retry on second time, discard the already rebuilt data from the first round rebuild, this obj discard requires discarding epoch [A, B), the purpose of this discard is to avoid overwrite on second rebuild. Given that VOS now can support overwrite, we might be able to skip obj discard in this case now.

The conclusion is that obj discard is till required in current rebuild/reintegration, but we may consider to drop it in the future.

[jolivier23]

I'm wondering should we move further on this direction?

Looks such fix won't solve the long rebuild reclaim time issue when the system has only few giant objects. If the "obj discard" is only used for reclaiming space by deleting object after reintegration, why don't we just mark the object as "obsoleted" (which ignores snapshots) in ilog and rely on aggregation to put it for GC?

Not a bad idea. Let me look at what that would entail.

[daosbuild1]

Test stage Unit Test with memcheck on EL 8.8 completed with status FAILURE. https://build.hpdd.intel.com/job/daos-stack/job/daos/job/PR-14143/8/display/redirect

[daosbuild1]

Test stage Build on EL 8 completed with status FAILURE. https://build.hpdd.intel.com//job/daos-stack/job/daos/view/change-requests/job/PR-14143/9/execution/node/306/log

[daosbuild1]

Test stage Build on Leap 15.5 with Intel-C and TARGET_PREFIX completed with status FAILURE. https://build.hpdd.intel.com//job/daos-stack/job/daos/view/change-requests/job/PR-14143/9/execution/node/394/log

[daosbuild1]

Test stage Build RPM on EL 9 completed with status FAILURE. https://build.hpdd.intel.com//job/daos-stack/job/daos/view/change-requests/job/PR-14143/9/execution/node/379/log

[daosbuild1]

Test stage Build RPM on EL 8 completed with status FAILURE. https://build.hpdd.intel.com//job/daos-stack/job/daos/view/change-requests/job/PR-14143/9/execution/node/378/log

[daosbuild1]

Test stage Build RPM on Leap 15.5 completed with status FAILURE. https://build.hpdd.intel.com//job/daos-stack/job/daos/view/change-requests/job/PR-14143/9/execution/node/383/log

[daosbuild1]

Test stage Build DEB on Ubuntu 20.04 completed with status FAILURE. https://build.hpdd.intel.com//job/daos-stack/job/daos/view/change-requests/job/PR-14143/9/execution/node/282/log

[daosbuild1]

Test stage Functional on EL 8.8 completed with status FAILURE. https://build.hpdd.intel.com//job/daos-stack/job/daos/view/change-requests/job/PR-14143/10/execution/node/1176/log

[daosbuild1]

Test stage Unit Test with memcheck on EL 8.8 completed with status FAILURE. https://build.hpdd.intel.com/job/daos-stack/job/daos/job/PR-14143/11/display/redirect

[jolivier23]

Note to @daos-stack/daos-gatekeeper this one is on top of #14382 so that should land first

[jolivier23]

sorry for the re-review. I tried an experiment and it failed. I fixed another issue in another PR and based this one on it. That preserved the reviews but after landing that one, it removed them. This should be identical to before.