daos icon indicating copy to clipboard operation
daos copied to clipboard
verified publisher icon daos-stack

DAOS-16008 cart: cart_ctl fixes

Open frostedcmos opened this issue 1 year ago • 3 comments

  • Change cart_ctl to no longer fail if pre-ping fails in daos env
  • Set individual cmd pings to be 3 seconds
  • Change cart_ctl invocation to use 'no sync' option, avoiding pinging all ranks, including possibly dead ones

Required-githooks: true

Before requesting gatekeeper:

  • [ ] Two review approvals and any prior change requests have been resolved.
  • [ ] Testing is complete and all tests passed or there is a reason documented in the PR why it should be force landed and forced-landing tag is set.
  • [ ] Features: (or Test-tag*) commit pragma was used or there is a reason documented that there are no appropriate tags for this PR.
  • [ ] Commit messages follows the guidelines outlined here.
  • [ ] Any tests skipped by the ticket being addressed have been run and passed in the PR.

Gatekeeper:

  • [ ] You are the appropriate gatekeeper to be landing the patch.
  • [ ] The PR has 2 reviews by people familiar with the code, including appropriate owners.
  • [ ] Githooks were used. If not, request that user install them and check copyright dates.
  • [ ] Checkpatch issues are resolved. Pay particular attention to ones that will show up on future PRs.
  • [ ] All builds have passed. Check non-required builds for any new compiler warnings.
  • [ ] Sufficient testing is done. Check feature pragmas and test tags and that tests skipped for the ticket are run and now pass with the changes.
  • [ ] If applicable, the PR has addressed any potential version compatibility issues.
  • [ ] Check the target branch. If it is master branch, should the PR go to a feature branch? If it is a release branch, does it have merge approval in the JIRA ticket.
  • [ ] Extra checks if forced landing is requested
    • [ ] Review comments are sufficiently resolved, particularly by prior reviewers that requested changes.
    • [ ] No new NLT or valgrind warnings. Check the classic view.
    • [ ] Quick-build or Quick-functional is not used.
  • [ ] Fix the commit message upon landing. Check the standard here. Edit it to create a single commit. If necessary, ask submitter for a new summary.

frostedcmos avatar Jul 02 '24 20:07 frostedcmos

Ticket title is 'daos_test/suite.py:DaosCoreTest - a variants failing with cart_ctl Connection timeout' Status is 'In Progress' Labels: 'ci_impact,pr_test,scrubbed_2.8,ucx_provider' https://daosio.atlassian.net/browse/DAOS-16008

github-actions[bot] avatar Jul 02 '24 20:07 github-actions[bot]

ftest LGTM. On the topic of cart_ctl, what does this buy us? I know we use it to write some string to the log file, but what's the point? not sure I understand the question. cart_ctl allows user to inject log messages on the engine side; this is used by tests to inject test name before they start. mostly useful when same engine is reuesd between test variants, as it allows then developer to know which line of engine corresponds to what test

frostedcmos avatar Jul 03 '24 16:07 frostedcmos

mostly useful when same engine is reuesd between test variants, as it allows then developer to know which line of engine corresponds to what test

This answers my question. Thanks :)

daltonbohning avatar Jul 03 '24 17:07 daltonbohning

Test stage Functional Hardware Medium completed with status FAILURE. https://build.hpdd.intel.com//job/daos-stack/job/daos/view/change-requests/job/PR-14686/2/execution/node/1505/log

daosbuild1 avatar Jul 08 '24 09:07 daosbuild1

Test stage Unit Test on EL 8.8 completed with status FAILURE. https://build.hpdd.intel.com/job/daos-stack/job/daos/job/PR-14686/4/display/redirect

daosbuild1 avatar Jul 10 '24 03:07 daosbuild1

Test stage Functional Hardware Medium completed with status UNSTABLE. https://build.hpdd.intel.com/job/daos-stack/job/daos//view/change-requests/job/PR-14686/6/testReport/

daosbuild1 avatar Jul 12 '24 09:07 daosbuild1

Test stage Functional Hardware Medium UCX Provider completed with status FAILURE. https://build.hpdd.intel.com//job/daos-stack/job/daos/view/change-requests/job/PR-14686/6/execution/node/1622/log

daosbuild1 avatar Jul 12 '24 09:07 daosbuild1

Due to a lab power issue I restarted the PR. I also enabled the UCX Provider stage (where the problem is being seen) w/ Build with Parameters (future commits should use Skip-func-hw-test-medium-ucx-provider: false).

The failures in https://build.hpdd.intel.com/job/daos-stack/job/daos/job/PR-14686/6:

  • 1x failure in https://build.hpdd.intel.com/job/daos-stack/job/daos/job/PR-14686/6/artifact/Functional%20Hardware%20Medium/dfuse/daos_build.py
    • Known issue https://daosio.atlassian.net/browse/DAOS-16215
  • 28x failures in https://build.hpdd.intel.com/job/daos-stack/job/daos/job/PR-14686/6/artifact/Functional%20Hardware%20Medium%20UCX%20Provider/daos_test/suite.py:
2024-07-12 09:15:40,302 process          L0604 INFO | Running 'cart_ctl add_log_msg --group-name daos_server -m "Test.name: 01-./daos_test/suite.py:DaosCoreTest.test_daos_degraded_mode;run-agent_config-transport_config-daos_tests-args-daos_test-num_clients-pools_created-stopped_ranks-test_name-dmg-hosts-pool-server_config-engines-0-storage-0-1-timeouts-01b6" True --rank all --use_daos_agent_env'
2024-07-12 09:15:46,709 process          L0416 DEBUG| [stderr] Init failed; rc=-1020
2024-07-12 09:15:46,710 process          L0416 DEBUG| [stdout] crtu_wait_for_ranks() failed with rc=-1
2024-07-12 09:15:46,713 process          L0686 INFO | Command 'cart_ctl add_log_msg --group-name daos_server -m "Test.name: 01-./daos_test/suite.py:DaosCoreTest.test_daos_degraded_mode;run-agent_config-transport_config-daos_tests-args-daos_test-num_clients-pools_created-stopped_ranks-test_name-dmg-hosts-pool-server_config-engines-0-storage-0-1-timeouts-01b6" True --rank all --use_daos_agent_env' finished with 255 after 6.407943487167358s
2024-07-12 09:15:46,713 general_utils    L0177 INFO | Error occurred running 'cart_ctl add_log_msg --group-name daos_server -m "Test.name: 01-./daos_test/suite.py:DaosCoreTest.test_daos_degraded_mode;run-agent_config-transport_config-daos_tests-args-daos_test-num_clients-pools_created-stopped_ranks-test_name-dmg-hosts-pool-server_config-engines-0-storage-0-1-timeouts-01b6" True --rank all --use_daos_agent_env': Command 'cart_ctl add_log_msg --group-name daos_server -m "Test.name: 01-./daos_test/suite.py:DaosCoreTest.test_daos_degraded_mode;run-agent_config-transport_config-daos_tests-args-daos_test-num_clients-pools_created-stopped_ranks-test_name-dmg-hosts-pool-server_config-engines-0-storage-0-1-timeouts-01b6" True --rank all --use_daos_agent_env' failed.
stdout: b'crtu_wait_for_ranks() failed with rc=-1\n'
stderr: b'Init failed; rc=-1020\n'
additional_info: None

phender avatar Jul 12 '24 13:07 phender

Due to a lab power issue I restarted the PR. I also enabled the UCX Provider stage (where the problem is being seen) w/ Build with Parameters (future commits should use Skip-func-hw-test-medium-ucx-provider: false).

The failures in https://build.hpdd.intel.com/job/daos-stack/job/daos/job/PR-14686/6:

  • 1x failure in https://build.hpdd.intel.com/job/daos-stack/job/daos/job/PR-14686/6/artifact/Functional%20Hardware%20Medium/dfuse/daos_build.py

    • Known issue https://daosio.atlassian.net/browse/DAOS-16215

  • 28x failures in https://build.hpdd.intel.com/job/daos-stack/job/daos/job/PR-14686/6/artifact/Functional%20Hardware%20Medium%20UCX%20Provider/daos_test/suite.py:

2024-07-12 09:15:40,302 process          L0604 INFO | Running 'cart_ctl add_log_msg --group-name daos_server -m "Test.name: 01-./daos_test/suite.py:DaosCoreTest.test_daos_degraded_mode;run-agent_config-transport_config-daos_tests-args-daos_test-num_clients-pools_created-stopped_ranks-test_name-dmg-hosts-pool-server_config-engines-0-storage-0-1-timeouts-01b6" True --rank all --use_daos_agent_env'
2024-07-12 09:15:46,709 process          L0416 DEBUG| [stderr] Init failed; rc=-1020
2024-07-12 09:15:46,710 process          L0416 DEBUG| [stdout] crtu_wait_for_ranks() failed with rc=-1
2024-07-12 09:15:46,713 process          L0686 INFO | Command 'cart_ctl add_log_msg --group-name daos_server -m "Test.name: 01-./daos_test/suite.py:DaosCoreTest.test_daos_degraded_mode;run-agent_config-transport_config-daos_tests-args-daos_test-num_clients-pools_created-stopped_ranks-test_name-dmg-hosts-pool-server_config-engines-0-storage-0-1-timeouts-01b6" True --rank all --use_daos_agent_env' finished with 255 after 6.407943487167358s
2024-07-12 09:15:46,713 general_utils    L0177 INFO | Error occurred running 'cart_ctl add_log_msg --group-name daos_server -m "Test.name: 01-./daos_test/suite.py:DaosCoreTest.test_daos_degraded_mode;run-agent_config-transport_config-daos_tests-args-daos_test-num_clients-pools_created-stopped_ranks-test_name-dmg-hosts-pool-server_config-engines-0-storage-0-1-timeouts-01b6" True --rank all --use_daos_agent_env': Command 'cart_ctl add_log_msg --group-name daos_server -m "Test.name: 01-./daos_test/suite.py:DaosCoreTest.test_daos_degraded_mode;run-agent_config-transport_config-daos_tests-args-daos_test-num_clients-pools_created-stopped_ranks-test_name-dmg-hosts-pool-server_config-engines-0-storage-0-1-timeouts-01b6" True --rank all --use_daos_agent_env' failed.
stdout: b'crtu_wait_for_ranks() failed with rc=-1\n'
stderr: b'Init failed; rc=-1020\n'
additional_info: None

Looks like cart_ctl.no_sync.value = True expands as a literal 'True' into the cart_ctl command, will push a patch to set it as to '-n' instead for value.

frostedcmos avatar Jul 15 '24 16:07 frostedcmos

Looks like cart_ctl.no_sync.value = True expands as a literal 'True' into the cart_ctl command, will push a patch to set it as to '-n' instead for value.

You'll want to change this: https://github.com/daos-stack/daos/blob/27f5e90b63a96b38251a01946ce107fb121e8ea2/src/tests/ftest/util/cart_ctl_utils.py#L63 To

self.no_sync = FormattedParameter("-n", False)

daltonbohning avatar Jul 15 '24 17:07 daltonbohning

Test stage NLT on EL 8.8 completed with status FAILURE. https://build.hpdd.intel.com/job/daos-stack/job/daos/job/PR-14686/7/display/redirect

daosbuild1 avatar Jul 15 '24 17:07 daosbuild1

Test stage Functional Hardware Medium UCX Provider completed with status FAILURE. https://build.hpdd.intel.com//job/daos-stack/job/daos/view/change-requests/job/PR-14686/8/execution/node/1516/log

daosbuild1 avatar Jul 16 '24 13:07 daosbuild1

Test stage NLT on EL 8.8 completed with status UNSTABLE. https://build.hpdd.intel.com/job/daos-stack/job/daos//view/change-requests/job/PR-14686/13/testReport/

daosbuild1 avatar Jul 16 '24 20:07 daosbuild1

Test stage Functional Hardware Medium UCX Provider completed with status FAILURE. https://build.hpdd.intel.com//job/daos-stack/job/daos/view/change-requests/job/PR-14686/14/execution/node/1546/log

daosbuild1 avatar Jul 17 '24 18:07 daosbuild1

Test stage NLT on EL 8.8 completed with status UNSTABLE. https://build.hpdd.intel.com/job/daos-stack/job/daos//view/change-requests/job/PR-14686/16/testReport/

daosbuild1 avatar Jul 17 '24 21:07 daosbuild1

Test stage Functional Hardware Medium UCX Provider completed with status FAILURE. https://build.hpdd.intel.com//job/daos-stack/job/daos/view/change-requests/job/PR-14686/16/execution/node/1551/log

daosbuild1 avatar Jul 19 '24 13:07 daosbuild1

Test stage Functional Hardware Medium completed with status FAILURE. https://build.hpdd.intel.com//job/daos-stack/job/daos/view/change-requests/job/PR-14686/17/execution/node/1416/log

daosbuild1 avatar Jul 21 '24 10:07 daosbuild1

Unrelated test timeout failure (https://daosio.atlassian.net/browse/DAOS-15608) in Test Hardware / Functional Hardware Medium / FTEST_osa.OSAOfflineReintegration.1-./osa/offline_reintegration.py:OSAOfflineReintegration.test_osa_offline_reintegration_without_checksum;run-agent_config-transport_config-aggregation-blank_node-checksum-container-dmg-hosts-ior-clientslots-iorflags-loop_test-mdtest-client_processes-wr_size-32K-pool-pool_capacity-rebuild-server_config-engines-0-storage-0-1-setup-snapshot-test_obj_class-test_rf-timeouts-c212

frostedcmos avatar Jul 22 '24 16:07 frostedcmos

ping reviewers

frostedcmos avatar Jul 22 '24 16:07 frostedcmos