DAOS-16477 mgmt: return suspect engines for pool healthy query

[wangshilong]

After significant failures, the system may leave behind some suspect engines that were marked as DEAD by the SWIM protocol, but were not excluded from the system to prevent data loss. An administrator can bring these ranks back online by restarting them.

This PR aims to provide an administrative interface for querying suspect engines following a massive failure. These suspect engines can be retrieved using the daos/dmg --health-only command.

An example of output of dmg pool query --health-only:

Pool 6f450a68-8c7d-4da9-8900-02691650f6a2, ntarget=8, disabled=2, leader=3, version=4, state=Degraded Pool health info:

• Disabled ranks: 1
• Suspect ranks: 2
• Rebuild busy, 0 objs, 0 recs

Required-githooks: true

Before requesting gatekeeper:

• [ ] Two review approvals and any prior change requests have been resolved.
• [ ] Testing is complete and all tests passed or there is a reason documented in the PR why it should be force landed and forced-landing tag is set.
• [ ] Features: (or Test-tag*) commit pragma was used or there is a reason documented that there are no appropriate tags for this PR.
• [ ] Commit messages follows the guidelines outlined here.
• [ ] Any tests skipped by the ticket being addressed have been run and passed in the PR.

Gatekeeper:

• [ ] You are the appropriate gatekeeper to be landing the patch.
• [ ] The PR has 2 reviews by people familiar with the code, including appropriate owners.
• [ ] Githooks were used. If not, request that user install them and check copyright dates.
• [ ] Checkpatch issues are resolved. Pay particular attention to ones that will show up on future PRs.
• [ ] All builds have passed. Check non-required builds for any new compiler warnings.
• [ ] Sufficient testing is done. Check feature pragmas and test tags and that tests skipped for the ticket are run and now pass with the changes.
• [ ] If applicable, the PR has addressed any potential version compatibility issues.
• [ ] Check the target branch. If it is master branch, should the PR go to a feature branch? If it is a release branch, does it have merge approval in the JIRA ticket.
• [ ] Extra checks if forced landing is requested
  • [ ] Review comments are sufficiently resolved, particularly by prior reviewers that requested changes.
  • [ ] No new NLT or valgrind warnings. Check the classic view.
  • [ ] Quick-build or Quick-functional is not used.
• [ ] Fix the commit message upon landing. Check the standard here. Edit it to create a single commit. If necessary, ask submitter for a new summary.

[github-actions[bot]]

Ticket title is 'Provide admin interface to query hanging engines after massive failure' Status is 'In Review' https://daosio.atlassian.net/browse/DAOS-16477

[johannlombardi]

@wangshilong reviewing the terminology at the TSC working group and we are suggesting to replace "hanging" in the output of the tool with "suspect".

[wangshilong]

@wangshilong reviewing the terminology at the TSC working group and we are suggesting to replace "hanging" in the output of the tool with "suspect".

@johannlombardi Sure

[wangshilong]

Agreed with @tanabarr that documentation is needed here to tell users what "suspect" ranks are, and understanding what this means for their system. From reading the code, I get the idea that they are ranks we think may be down, but aren't sure yet for some reason (is this correct?).

I added documents in the following commits.

I noticed that the suspect ranks were added to the default flags. Do we really want to show these "maybe up, maybe down" ranks by default, if this state is probably transient?

Yes, it is transient. Notice this is only added to health query default flags.

Also, at minimum I think all the Go changes should have matching unit test cases demonstrating the added logic.

Done

[wangshilong]

Changes look okay but we need some updates to documentation to explain what the new parameter means in pool query output and how and why the administrator should use this information.

I added example and documents in the "pool operation"

Also please provide some control-plane unit tests and preferably update ftest for dmg pool query coverage.

Added, control-plane unit tests, I am trying to update ftests which might cause more time(I might test commits in the PR or in a separate PR) Thanks.

[daosbuild1]

Test stage Functional on EL 8.8 completed with status FAILURE. https://build.hpdd.intel.com//job/daos-stack/job/daos/view/change-requests/job/PR-15196/7/execution/node/1173/log

[daosbuild1]

Test stage Functional on EL 8.8 completed with status FAILURE. https://build.hpdd.intel.com//job/daos-stack/job/daos/view/change-requests/job/PR-15196/8/execution/node/1202/log

[daosbuild1]

Test stage NLT on EL 8.8 completed with status UNSTABLE. https://build.hpdd.intel.com/job/daos-stack/job/daos//view/change-requests/job/PR-15196/9/testReport/

[daosbuild1]

Test stage Functional on EL 8.8 completed with status FAILURE. https://build.hpdd.intel.com//job/daos-stack/job/daos/view/change-requests/job/PR-15196/9/execution/node/1197/log

[daosbuild1]

Test stage Functional Hardware Medium completed with status FAILURE. https://build.hpdd.intel.com//job/daos-stack/job/daos/view/change-requests/job/PR-15196/16/execution/node/1351/log

[daosbuild1]

Test stage Functional Hardware Medium Verbs Provider completed with status FAILURE. https://build.hpdd.intel.com//job/daos-stack/job/daos/view/change-requests/job/PR-15196/16/execution/node/1397/log

[wangshilong]

ping reviewers..