daos icon indicating copy to clipboard operation
daos copied to clipboard
verified publisher icon daos-stack

DAOS-14657: To add a functional test for fio in different modes

Open knard38 opened this issue 1 year ago • 34 comments

Description

This PR adds a new functional test running fio with the following io engines and compare their bandwidths:

  • ioengine psync with Dfuse mount point and the libpil4dfs interception library.
  • ioengine dfs. Different block size are tested.

For running fio with a relevant number of processes and binding, a new util class has been added. This last one returns miscellaneous information about client CPU processing resources of a given set of client nodes.

Finally, to allow the local validation of this new functional test, the following modifications have been done on the ansible ftest deployment scripts:

  • support of fio.
  • support of shared cluster and verbs provider.
  • remove useless MPI configuration file.
  • fix RPMs and python dependencies.

Validation

The functional tests are limited with the pragma Test-tag: Pil4dfsFio,test_pil4dfs_vs_dfs as this PR is only introducing the new functional test test_pil4dfs_vs_dfs and some ansible scripts update not validated by the CI.

The new functional tests have been run successfully 10 times. More details could be found at: https://build.hpdd.intel.com/job/daos-stack/job/daos/job/PR-13797/15/testReport/FTEST_dfuse/Pil4dfsFio/

Before requesting gatekeeper:

  • [ ] Two review approvals and any prior change requests have been resolved.
  • [ ] Testing is complete and all tests passed or there is a reason documented in the PR why it should be force landed and forced-landing tag is set.
  • [ ] Features: (or Test-tag*) commit pragma was used or there is a reason documented that there are no appropriate tags for this PR.
  • [ ] Commit messages follows the guidelines outlined here.
  • [ ] Any tests skipped by the ticket being addressed have been run and passed in the PR.

Gatekeeper:

  • [ ] You are the appropriate gatekeeper to be landing the patch.
  • [ ] The PR has 2 reviews by people familiar with the code, including appropriate owners.
  • [ ] Githooks were used. If not, request that user install them and check copyright dates.
  • [ ] Checkpatch issues are resolved. Pay particular attention to ones that will show up on future PRs.
  • [ ] All builds have passed. Check non-required builds for any new compiler warnings.
  • [ ] Sufficient testing is done. Check feature pragmas and test tags and that tests skipped for the ticket are run and now pass with the changes.
  • [ ] If applicable, the PR has addressed any potential version compatibility issues.
  • [ ] Check the target branch. If it is master branch, should the PR go to a feature branch? If it is a release branch, does it have merge approval in the JIRA ticket.
  • [ ] Extra checks if forced landing is requested
    • [ ] Review comments are sufficiently resolved, particularly by prior reviewers that requested changes.
    • [ ] No new NLT or valgrind warnings. Check the classic view.
    • [ ] Quick-build or Quick-functional is not used.
  • [ ] Fix the commit message upon landing. Check the standard here. Edit it to create a single commit. If necessary, ask submitter for a new summary.

knard38 avatar Feb 13 '24 08:02 knard38

Bug-tracker data: Errors are component not formatted correctly,Ticket number prefix incorrect,PR title is malformatted. See https://daosio.atlassian.net/wiki/spaces/DC/pages/11133911069/Commit+Comments,Unable to load ticket data https://daosio.atlassian.net/browse/Ckochhof/dev/master/daos

github-actions[bot] avatar Feb 13 '24 08:02 github-actions[bot]

Test stage Python Bandit check completed with status FAILURE. https://build.hpdd.intel.com//job/daos-stack/job/daos/view/change-requests/job/PR-13797/1/execution/node/139/log

daosbuild1 avatar Feb 13 '24 08:02 daosbuild1

Test stage Build DEB on Ubuntu 20.04 completed with status FAILURE. https://build.hpdd.intel.com//job/daos-stack/job/daos/view/change-requests/job/PR-13797/1/execution/node/406/log

daosbuild1 avatar Feb 13 '24 08:02 daosbuild1

Test stage Build RPM on EL 9 completed with status FAILURE. https://build.hpdd.intel.com//job/daos-stack/job/daos/view/change-requests/job/PR-13797/1/execution/node/344/log

daosbuild1 avatar Feb 13 '24 08:02 daosbuild1

Test stage Build RPM on Leap 15.5 completed with status FAILURE. https://build.hpdd.intel.com//job/daos-stack/job/daos/view/change-requests/job/PR-13797/1/execution/node/331/log

daosbuild1 avatar Feb 13 '24 08:02 daosbuild1

Test stage Build RPM on EL 8 completed with status FAILURE. https://build.hpdd.intel.com//job/daos-stack/job/daos/view/change-requests/job/PR-13797/1/execution/node/378/log

daosbuild1 avatar Feb 13 '24 08:02 daosbuild1

Test stage Build DEB on Ubuntu 20.04 completed with status FAILURE. https://build.hpdd.intel.com//job/daos-stack/job/daos/view/change-requests/job/PR-13797/2/execution/node/313/log

daosbuild1 avatar Feb 13 '24 09:02 daosbuild1

Test stage Build RPM on EL 9 completed with status FAILURE. https://build.hpdd.intel.com//job/daos-stack/job/daos/view/change-requests/job/PR-13797/2/execution/node/397/log

daosbuild1 avatar Feb 13 '24 09:02 daosbuild1

Test stage Build RPM on EL 8 completed with status FAILURE. https://build.hpdd.intel.com//job/daos-stack/job/daos/view/change-requests/job/PR-13797/2/execution/node/394/log

daosbuild1 avatar Feb 13 '24 09:02 daosbuild1

Test stage Build RPM on Leap 15.5 completed with status FAILURE. https://build.hpdd.intel.com//job/daos-stack/job/daos/view/change-requests/job/PR-13797/2/execution/node/318/log

daosbuild1 avatar Feb 13 '24 09:02 daosbuild1

Test stage Build DEB on Ubuntu 20.04 completed with status FAILURE. https://build.hpdd.intel.com//job/daos-stack/job/daos/view/change-requests/job/PR-13797/3/execution/node/356/log

daosbuild1 avatar Feb 13 '24 09:02 daosbuild1

Test stage Build RPM on EL 9 completed with status FAILURE. https://build.hpdd.intel.com//job/daos-stack/job/daos/view/change-requests/job/PR-13797/3/execution/node/362/log

daosbuild1 avatar Feb 13 '24 09:02 daosbuild1

Test stage Build RPM on EL 8 completed with status FAILURE. https://build.hpdd.intel.com//job/daos-stack/job/daos/view/change-requests/job/PR-13797/3/execution/node/385/log

daosbuild1 avatar Feb 13 '24 09:02 daosbuild1

Test stage Build RPM on Leap 15.5 completed with status FAILURE. https://build.hpdd.intel.com//job/daos-stack/job/daos/view/change-requests/job/PR-13797/3/execution/node/353/log

daosbuild1 avatar Feb 13 '24 09:02 daosbuild1

Test stage Build DEB on Ubuntu 20.04 completed with status FAILURE. https://build.hpdd.intel.com//job/daos-stack/job/daos/view/change-requests/job/PR-13797/4/execution/node/381/log

daosbuild1 avatar Feb 13 '24 09:02 daosbuild1

Test stage Build RPM on EL 9 completed with status FAILURE. https://build.hpdd.intel.com//job/daos-stack/job/daos/view/change-requests/job/PR-13797/4/execution/node/417/log

daosbuild1 avatar Feb 13 '24 09:02 daosbuild1

Test stage Build RPM on EL 8 completed with status FAILURE. https://build.hpdd.intel.com//job/daos-stack/job/daos/view/change-requests/job/PR-13797/4/execution/node/390/log

daosbuild1 avatar Feb 13 '24 09:02 daosbuild1

Test stage Build RPM on Leap 15.5 completed with status FAILURE. https://build.hpdd.intel.com//job/daos-stack/job/daos/view/change-requests/job/PR-13797/4/execution/node/303/log

daosbuild1 avatar Feb 13 '24 09:02 daosbuild1

Test stage Build DEB on Ubuntu 20.04 completed with status FAILURE. https://build.hpdd.intel.com//job/daos-stack/job/daos/view/change-requests/job/PR-13797/5/execution/node/287/log

daosbuild1 avatar Feb 13 '24 13:02 daosbuild1

Test stage Build RPM on EL 9 completed with status FAILURE. https://build.hpdd.intel.com//job/daos-stack/job/daos/view/change-requests/job/PR-13797/5/execution/node/288/log

daosbuild1 avatar Feb 13 '24 13:02 daosbuild1

Test stage Build RPM on EL 8 completed with status FAILURE. https://build.hpdd.intel.com//job/daos-stack/job/daos/view/change-requests/job/PR-13797/5/execution/node/291/log

daosbuild1 avatar Feb 13 '24 13:02 daosbuild1

Test stage Build RPM on Leap 15.5 completed with status FAILURE. https://build.hpdd.intel.com//job/daos-stack/job/daos/view/change-requests/job/PR-13797/5/execution/node/281/log

daosbuild1 avatar Feb 13 '24 13:02 daosbuild1

Test stage Functional Hardware Medium completed with status FAILURE. https://build.hpdd.intel.com//job/daos-stack/job/daos/view/change-requests/job/PR-13797/6/execution/node/715/log

daosbuild1 avatar Feb 13 '24 13:02 daosbuild1

Test stage Functional Hardware Medium completed with status FAILURE. https://build.hpdd.intel.com//job/daos-stack/job/daos/view/change-requests/job/PR-13797/7/execution/node/715/log

daosbuild1 avatar Feb 13 '24 15:02 daosbuild1

Test stage Functional Hardware Medium completed with status UNSTABLE. https://build.hpdd.intel.com/job/daos-stack/job/daos//view/change-requests/job/PR-13797/8/testReport/

daosbuild1 avatar Feb 13 '24 20:02 daosbuild1

Test stage Functional Hardware Medium completed with status UNSTABLE. https://build.hpdd.intel.com/job/daos-stack/job/daos//view/change-requests/job/PR-13797/10/testReport/

daosbuild1 avatar Feb 15 '24 00:02 daosbuild1

Bug-tracker data: Errors are component not formatted correctly,Ticket number prefix incorrect,PR title is malformatted. See https://daosio.atlassian.net/wiki/spaces/DC/pages/11133911069/Commit+Comments,Unable to load ticket data https://daosio.atlassian.net/browse/Ckochhof/dev/master/daos

github-actions[bot] avatar Feb 16 '24 10:02 github-actions[bot]

Test stage Functional Hardware Medium completed with status UNSTABLE. https://build.hpdd.intel.com/job/daos-stack/job/daos//view/change-requests/job/PR-13797/11/testReport/

daosbuild1 avatar Feb 16 '24 14:02 daosbuild1

Bug-tracker data: Errors are component not formatted correctly,Ticket number prefix incorrect,PR title is malformatted. See https://daosio.atlassian.net/wiki/spaces/DC/pages/11133911069/Commit+Comments,Unable to load ticket data https://daosio.atlassian.net/browse/Ckochhof/dev/master/daos

github-actions[bot] avatar Feb 21 '24 16:02 github-actions[bot]

Test stage Functional Hardware Medium completed with status UNSTABLE. https://build.hpdd.intel.com/job/daos-stack/job/daos//view/change-requests/job/PR-13797/12/testReport/

daosbuild1 avatar Feb 22 '24 01:02 daosbuild1