daos icon indicating copy to clipboard operation
daos copied to clipboard
verified publisher icon daos-stack

DAOS-16027 test: Adding daos_test REBUILD31-34 subtests

Open phender opened this issue 1 year ago • 3 comments

Adding rebuild tests currently not executed by doas_test/rebuild.py.

Skip-unit-tests: true Skip-fault-injection-test: true Skip-func-hw-test-medium-md-on-ssd: false Test-tag: test_rebuild_31 test_rebuild_32 test_rebuild_33 test_rebuild_34

Required-githooks: true

Before requesting gatekeeper:

  • [ ] Two review approvals and any prior change requests have been resolved.
  • [ ] Testing is complete and all tests passed or there is a reason documented in the PR why it should be force landed and forced-landing tag is set.
  • [ ] Features: (or Test-tag*) commit pragma was used or there is a reason documented that there are no appropriate tags for this PR.
  • [ ] Commit messages follows the guidelines outlined here.
  • [ ] Any tests skipped by the ticket being addressed have been run and passed in the PR.

Gatekeeper:

  • [ ] You are the appropriate gatekeeper to be landing the patch.
  • [ ] The PR has 2 reviews by people familiar with the code, including appropriate owners.
  • [ ] Githooks were used. If not, request that user install them and check copyright dates.
  • [ ] Checkpatch issues are resolved. Pay particular attention to ones that will show up on future PRs.
  • [ ] All builds have passed. Check non-required builds for any new compiler warnings.
  • [ ] Sufficient testing is done. Check feature pragmas and test tags and that tests skipped for the ticket are run and now pass with the changes.
  • [ ] If applicable, the PR has addressed any potential version compatibility issues.
  • [ ] Check the target branch. If it is master branch, should the PR go to a feature branch? If it is a release branch, does it have merge approval in the JIRA ticket.
  • [ ] Extra checks if forced landing is requested
    • [ ] Review comments are sufficiently resolved, particularly by prior reviewers that requested changes.
    • [ ] No new NLT or valgrind warnings. Check the classic view.
    • [ ] Quick-build or Quick-functional is not used.
  • [ ] Fix the commit message upon landing. Check the standard here. Edit it to create a single commit. If necessary, ask submitter for a new summary.

phender avatar Jun 14 '24 18:06 phender

Ticket title is 'daos_test -r cases REBUILD31-REBUILD35 are not tested in functional testing' Status is 'In Review' Labels: 'scrubbed_2.8,triaged,testp2' https://daosio.atlassian.net/browse/DAOS-16027

github-actions[bot] avatar Jun 14 '24 18:06 github-actions[bot]

Test stage Functional Hardware Medium MD on SSD completed with status FAILURE. https://build.hpdd.intel.com//job/daos-stack/job/daos/view/change-requests/job/PR-14584/1/execution/node/916/log

daosbuild1 avatar Jun 18 '24 12:06 daosbuild1

Test stage Functional Hardware Medium completed with status FAILURE. https://build.hpdd.intel.com//job/daos-stack/job/daos/view/change-requests/job/PR-14584/1/execution/node/962/log

daosbuild1 avatar Jun 18 '24 12:06 daosbuild1

Test stage Functional Hardware Medium MD on SSD completed with status FAILURE. https://build.hpdd.intel.com//job/daos-stack/job/daos/view/change-requests/job/PR-14584/2/execution/node/916/log

daosbuild1 avatar Aug 09 '24 17:08 daosbuild1

Test stage Functional Hardware Medium completed with status FAILURE. https://build.hpdd.intel.com//job/daos-stack/job/daos/view/change-requests/job/PR-14584/2/execution/node/1014/log

daosbuild1 avatar Aug 09 '24 18:08 daosbuild1

Test stage Functional Hardware Medium MD on SSD completed with status FAILURE. https://build.hpdd.intel.com//job/daos-stack/job/daos/view/change-requests/job/PR-14584/3/execution/node/916/log

daosbuild1 avatar Aug 30 '24 06:08 daosbuild1

Test stage Functional Hardware Medium completed with status FAILURE. https://build.hpdd.intel.com//job/daos-stack/job/daos/view/change-requests/job/PR-14584/3/execution/node/1014/log

daosbuild1 avatar Aug 30 '24 07:08 daosbuild1

Skip list entries added for the new tests:

[['DAOS-16575', 'test_method_name', 'test_rebuild_31']]
[['DAOS-16576', 'test_method_name', 'test_rebuild_32']]

phender avatar Sep 16 '24 19:09 phender

Can we put these in daily instead of pr? this adds 4 tests * 800s each. Which is almost 1 hour for every pr

@kccain thoughts?

phender avatar Sep 17 '24 15:09 phender

Rebuild 31-32 skipped and Rebuild 33-34 passed in https://build.hpdd.intel.com/job/daos-stack/job/daos/job/PR-14584/4/testReport/FTEST_daos_test/DaosCoreTestRebuild/ taking ~5.5 minutes per stage:

2024-09-16 20:32:28,369 test             L0530 INFO | START 1-./daos_test/rebuild.py:DaosCoreTestRebuild.test_rebuild_31;run-agent_config-transport_config-daos_tests-args-daos_test-pools_created-stopped_ranks-test_name-dmg-hosts-pool-server_config-engines-0-storage-0-1-timeouts-4b1a
2024-09-16 20:32:28,378 test             L0471 INFO | ==> Step 1: tearDown(): Called after test completion (test timeout: 1590s, elapsed: 0.01s, remaining: 1589.99s) [elapsed since last step: 0.01s]
2024-09-16 20:32:28,754 test             L0530 INFO | START 2-./daos_test/rebuild.py:DaosCoreTestRebuild.test_rebuild_32;run-agent_config-transport_config-daos_tests-args-daos_test-pools_created-stopped_ranks-test_name-dmg-hosts-pool-server_config-engines-0-storage-0-1-timeouts-4b1a
2024-09-16 20:32:28,762 test             L0471 INFO | ==> Step 1: tearDown(): Called after test completion (test timeout: 1590s, elapsed: 0.01s, remaining: 1589.99s) [elapsed since last step: 0.01s]
2024-09-16 20:32:29,139 test             L0530 INFO | START 3-./daos_test/rebuild.py:DaosCoreTestRebuild.test_rebuild_33;run-agent_config-transport_config-daos_tests-args-daos_test-pools_created-stopped_ranks-test_name-dmg-hosts-pool-server_config-engines-0-storage-0-1-timeouts-4b1a
2024-09-16 20:32:29,563 test             L0471 INFO | ==> Step 1: setUp(): Starting servers [elapsed since last step: 0.43s]
2024-09-16 20:33:20,315 test             L0471 INFO | ==> Step 2: setUp(): Starting agents [elapsed since last step: 50.75s]
2024-09-16 20:33:25,183 test             L0471 INFO | ==> Step 3: setUp(): Setup complete [elapsed since last step: 4.87s]
2024-09-16 20:34:54,204 test             L0471 INFO | ==> Step 4: tearDown(): Called after test completion (test timeout: 290s, elapsed: 145.06s, remaining: 144.94s) [elapsed since last step: 89.02s]
2024-09-16 20:34:57,237 test             L0939 INFO | PASS 3-./daos_test/rebuild.py:DaosCoreTestRebuild.test_rebuild_33;run-agent_config-transport_config-daos_tests-args-daos_test-pools_created-stopped_ranks-test_name-dmg-hosts-pool-server_config-engines-0-storage-0-1-timeouts-4b1a
2024-09-16 20:34:57,406 test             L0530 INFO | START 4-./daos_test/rebuild.py:DaosCoreTestRebuild.test_rebuild_34;run-agent_config-transport_config-daos_tests-args-daos_test-pools_created-stopped_ranks-test_name-dmg-hosts-pool-server_config-engines-0-storage-0-1-timeouts-4b1a
2024-09-16 20:34:57,841 test             L0471 INFO | ==> Step 1: setUp(): Starting servers [elapsed since last step: 0.44s]
2024-09-16 20:35:01,320 test             L0471 INFO | ==> Step 2: setUp(): Starting agents [elapsed since last step: 3.48s]
2024-09-16 20:35:01,806 test             L0471 INFO | ==> Step 3: setUp(): Destroying any existing pools before the test [elapsed since last step: 0.49s]
2024-09-16 20:35:03,002 test             L0471 INFO | ==> Step 4: setUp(): Setup complete [elapsed since last step: 1.20s]
2024-09-16 20:37:48,141 test             L0471 INFO | ==> Step 5: tearDown(): Called after test completion (test timeout: 1590s, elapsed: 170.73s, remaining: 1419.27s) [elapsed since last step: 165.14s]
2024-09-16 20:37:51,175 test             L0939 INFO | PASS 4-./daos_test/rebuild.py:DaosCoreTestRebuild.test_rebuild_34;run-agent_config-transport_config-daos_tests-args-daos_test-pools_created-stopped_ranks-test_name-dmg-hosts-pool-server_config-engines-0-storage-0-1-timeouts-4b1a

phender avatar Sep 17 '24 15:09 phender

Tests passing/skipping as expected in https://build.hpdd.intel.com/job/daos-stack/job/daos/job/PR-14584/5/testReport/FTEST_daos_test/DaosCoreTestRebuild/

phender avatar Sep 17 '24 20:09 phender