sst-core icon indicating copy to clipboard operation
sst-core copied to clipboard

Published 20 hours ago • verified publisher icon sstsimulator

Improve SST test frameworks

Open jmlapre opened this issue 1 year ago • 55 comments

sst_unittest.py:

  • [x] Deprecate initializeClass() – it’s not needed and redundant (DONE)
  • [ ] run_sst(): Remove parameters num_ranks, num_threads, and global_args since those are supposed to be determined by arguments to sst-test-* not by tests themselves. Probably any test suite using these is doing so incorrectly.
  • [ ] Get_test_runtime_sec() is used internally. It’s not strictly a “get”, it modifies the run time variable so should ONLY be called by run_sst and never externally. Change to internal?

sst_unittest_support.py:

  • [ ] Move away from lightweight wrappers and implementing minor function variants in place of adding function arguments
  • [x] testing_is_PIN2_used() : Deprecate – I will remove from the memH test suites that currently call it
  • [x] host_os_get_*() : Eliminate the ones that are just wrapping a Python function in favor of the Python function. Most tests at most have to make a distinction between osx & linux. Occasionally Ubuntu vs RHEL needs to be treated differently but only w.r.t. Pin in my experience.
  • [ ] testing_check_is_scenario_filtering_enabled() : should be internal?
  • [ ] sst_core_config_include_file_get_value_*() : these functions only differ in the return type. Remove the wrapper and make the underlying function public.
  • [ ] sst_elements_config_include_file_get_value_*(): Same as above
  • [ ] sstsimulator_conf_get_value_*(): Same as above
  • [ ] log*(): Mostly used/useful. No distinction exists afaik between VERBOSE_NORMAL and VERBOSE_LOUD in the framework despite there being separate verbosity levels. Should we eliminate unused levels or leave them?
  • [ ] Combining MPI files is complicated but useful to keep this in the framework since different OpenMPI versions do it differently. We have two badly named functions – probably rename so their names are intuitive.
  • [ ] combine_per_rank_files() - For files like stat files that are named basename_X.ext where X is the rank ID)
  • [ ] testing_merge_mpi_files() – For stdout/stderr redirects where the filename & directory structure depends entirely on the OpenMPI version
  • [ ] testing_compare_* : There are four of these functions where we could have one that takes output file, reference file, filter list, and type of diff to do (exact, sorted, subset)
  • [ ] testing_stat_output_diff() : Is there a way (or should we even try) to unify this with the testing_compare*() functions?
  • [ ] OS commands (os_*) should probably take a timeout parameter since they mostly call OSCommand under the hood which has a timeout.
  • [ ] Os_awk_print() : Can we get rid of this?
  • [ ] Os_wc() : It would be more intuitive to return (line count, word count, char count) instead of the roundabout index shenanigans it is doing right now.
  • [ ] Os_simple_command() : should we replace this with OSCommand from test_engine_support? Or just tell ppl to create their own subprocess runners if they need to execute something? Or leave as is?

jmlapre avatar Aug 28 '24 14:08 jmlapre

CMAKE-FORMAT TEST - PASSED

github-actions[bot] avatar Aug 28 '24 14:08 github-actions[bot]

CLANG-FORMAT TEST - PASSED

github-actions[bot] avatar Aug 28 '24 14:08 github-actions[bot]

Status Flag 'Pre-Test Inspection' - Auto Inspected - Inspection is Not Necessary for this Pull Request.

sst-autotester avatar Aug 28 '24 15:08 sst-autotester

Status Flag 'Pull Request AutoTester' - Testing Jenkins Projects:

Pull Request Auto Testing STARTING (click to expand)

Build Information

Test Name: SST__AutotestGen2_NewFW_sst-test_OMPI-4.1.4_PY3.6_sst-elements

  • Build Num: 1558
  • Status: STARTED

Build Information

Test Name: SST__AutotestGen2_NewFW_sst-test_OMPI-4.1.4_PY3.6_sst-elements_MR-2

  • Build Num: 1540
  • Status: STARTED

Build Information

Test Name: SST__AutotestGen2_NewFW_sst-test_OMPI-4.1.4_PY3.6_sst-elements_MT-2

  • Build Num: 1540
  • Status: STARTED

Build Information

Test Name: SST__AutotestGen2_NewFW_sst-test_OMPI-4.1.4_PY3.6_sst-macro_withsstcore

  • Build Num: 651
  • Status: STARTED

Build Information

Test Name: SST__AutotestGen2_NewFW_sst-test_OMPI-4.1.4_PY3.6_sst-core_Make-Dist

  • Build Num: 508
  • Status: STARTED

Build Information

Test Name: SST__AutotestGen2_NewFW_sst-test_Clang-Format_sst-core

  • Build Num: 462
  • Status: STARTED

Build Information

Test Name: SST__AutotestGen2_NewFW_OSX-14-XC15-ARM2_OMPI-4.1.6_PY3.10_sst-elements

  • Build Num: 120
  • Status: STARTED

Build Information

Test Name: SST__AutotestGen2_NewFW_OSX-14-XC15-ARM2_OMPI-4.1.6_PY3.10_sst-macro_withsstcore

  • Build Num: 82
  • Status: STARTED

Using Repos:

Repo: CORE (jmlapre/sst-core)
  • Branch: 1-improve-sst-test-frameworks
  • SHA: 8799b8c40fd39b37c7bca49b1764fa52d87ea725
  • Mode: TEST_REPO
Repo: SQE (sstsimulator/sst-sqe)
  • Branch: devel
  • SHA: 40ef31399c0ef1473f23626bd5e3aa75d18d234e
  • Mode: SUPPORT_REPO
Repo: ELEMENTS (sstsimulator/sst-elements)
  • Branch: devel
  • SHA: 2664955f14ed0ce7d9b0c77d3a18436df5a2c72b
  • Mode: SUPPORT_REPO
Repo: MACRO (sstsimulator/sst-macro)
  • Branch: devel
  • SHA: 50a62170b3681ea20cc2f56abd2eb3911053f1fc
  • Mode: SUPPORT_REPO

Pull Request Author: jmlapre

sst-autotester avatar Aug 28 '24 15:08 sst-autotester

Status Flag 'Pull Request AutoTester' - Jenkins Testing: all Jobs PASSED

Pull Request Auto Testing has PASSED (click to expand)

Build Information

Test Name: SST__AutotestGen2_NewFW_sst-test_OMPI-4.1.4_PY3.6_sst-elements

  • Build Num: 1558
  • Status: PASSED

Build Information

Test Name: SST__AutotestGen2_NewFW_sst-test_OMPI-4.1.4_PY3.6_sst-elements_MR-2

  • Build Num: 1540
  • Status: PASSED

Build Information

Test Name: SST__AutotestGen2_NewFW_sst-test_OMPI-4.1.4_PY3.6_sst-elements_MT-2

  • Build Num: 1540
  • Status: PASSED

Build Information

Test Name: SST__AutotestGen2_NewFW_sst-test_OMPI-4.1.4_PY3.6_sst-macro_withsstcore

  • Build Num: 651
  • Status: PASSED

Build Information

Test Name: SST__AutotestGen2_NewFW_sst-test_OMPI-4.1.4_PY3.6_sst-core_Make-Dist

  • Build Num: 508
  • Status: PASSED

Build Information

Test Name: SST__AutotestGen2_NewFW_sst-test_Clang-Format_sst-core

  • Build Num: 462
  • Status: PASSED

Build Information

Test Name: SST__AutotestGen2_NewFW_OSX-14-XC15-ARM2_OMPI-4.1.6_PY3.10_sst-elements

  • Build Num: 120
  • Status: PASSED

Build Information

Test Name: SST__AutotestGen2_NewFW_OSX-14-XC15-ARM2_OMPI-4.1.6_PY3.10_sst-macro_withsstcore

  • Build Num: 82
  • Status: PASSED

sst-autotester avatar Aug 28 '24 16:08 sst-autotester

Status Flag 'Pre-Merge Inspection' - - This Pull Request Requires Inspection... The code must be inspected by a member of the Team before Testing/Merging WARNING: NO REVIEWERS HAVE BEEN REQUESTED FOR THIS PULL REQUEST!

sst-autotester avatar Aug 28 '24 16:08 sst-autotester

All Jobs Finished; status = PASSED, However Inspection must be performed before merge can occur...

sst-autotester avatar Aug 28 '24 16:08 sst-autotester

CMAKE-FORMAT TEST - PASSED

github-actions[bot] avatar Aug 28 '24 16:08 github-actions[bot]

CLANG-FORMAT TEST - PASSED

github-actions[bot] avatar Aug 28 '24 16:08 github-actions[bot]

CLANG-FORMAT TEST - PASSED

github-actions[bot] avatar Aug 29 '24 14:08 github-actions[bot]

CMAKE-FORMAT TEST - PASSED

github-actions[bot] avatar Aug 29 '24 14:08 github-actions[bot]

Status Flag 'Pre-Test Inspection' - Auto Inspected - Inspection is Not Necessary for this Pull Request.

sst-autotester avatar Aug 29 '24 14:08 sst-autotester

Status Flag 'Pull Request AutoTester' - Testing Jenkins Projects:

Pull Request Auto Testing STARTING (click to expand)

Build Information

Test Name: SST__AutotestGen2_NewFW_sst-test_OMPI-4.1.4_PY3.6_sst-elements

  • Build Num: 1559
  • Status: STARTED

Build Information

Test Name: SST__AutotestGen2_NewFW_sst-test_OMPI-4.1.4_PY3.6_sst-elements_MR-2

  • Build Num: 1541
  • Status: STARTED

Build Information

Test Name: SST__AutotestGen2_NewFW_sst-test_OMPI-4.1.4_PY3.6_sst-elements_MT-2

  • Build Num: 1541
  • Status: STARTED

Build Information

Test Name: SST__AutotestGen2_NewFW_sst-test_OMPI-4.1.4_PY3.6_sst-macro_withsstcore

  • Build Num: 652
  • Status: STARTED

Build Information

Test Name: SST__AutotestGen2_NewFW_sst-test_OMPI-4.1.4_PY3.6_sst-core_Make-Dist

  • Build Num: 509
  • Status: STARTED

Build Information

Test Name: SST__AutotestGen2_NewFW_sst-test_Clang-Format_sst-core

  • Build Num: 463
  • Status: STARTED

Build Information

Test Name: SST__AutotestGen2_NewFW_OSX-14-XC15-ARM2_OMPI-4.1.6_PY3.10_sst-elements

  • Build Num: 121
  • Status: STARTED

Build Information

Test Name: SST__AutotestGen2_NewFW_OSX-14-XC15-ARM2_OMPI-4.1.6_PY3.10_sst-macro_withsstcore

  • Build Num: 83
  • Status: STARTED

Using Repos:

Repo: CORE (jmlapre/sst-core)
  • Branch: 1-improve-sst-test-frameworks
  • SHA: 8a4b4d917472818590d0c28f0270ae2a965d65d0
  • Mode: TEST_REPO
Repo: SQE (sstsimulator/sst-sqe)
  • Branch: devel
  • SHA: 2574c98896598820227190149834172b962dc3fc
  • Mode: SUPPORT_REPO
Repo: ELEMENTS (sstsimulator/sst-elements)
  • Branch: devel
  • SHA: 2664955f14ed0ce7d9b0c77d3a18436df5a2c72b
  • Mode: SUPPORT_REPO
Repo: MACRO (sstsimulator/sst-macro)
  • Branch: devel
  • SHA: 50a62170b3681ea20cc2f56abd2eb3911053f1fc
  • Mode: SUPPORT_REPO

Pull Request Author: jmlapre

sst-autotester avatar Aug 29 '24 14:08 sst-autotester

Status Flag 'Pull Request AutoTester' - Jenkins Testing: all Jobs PASSED

Pull Request Auto Testing has PASSED (click to expand)

Build Information

Test Name: SST__AutotestGen2_NewFW_sst-test_OMPI-4.1.4_PY3.6_sst-elements

  • Build Num: 1559
  • Status: PASSED

Build Information

Test Name: SST__AutotestGen2_NewFW_sst-test_OMPI-4.1.4_PY3.6_sst-elements_MR-2

  • Build Num: 1541
  • Status: PASSED

Build Information

Test Name: SST__AutotestGen2_NewFW_sst-test_OMPI-4.1.4_PY3.6_sst-elements_MT-2

  • Build Num: 1541
  • Status: PASSED

Build Information

Test Name: SST__AutotestGen2_NewFW_sst-test_OMPI-4.1.4_PY3.6_sst-macro_withsstcore

  • Build Num: 652
  • Status: PASSED

Build Information

Test Name: SST__AutotestGen2_NewFW_sst-test_OMPI-4.1.4_PY3.6_sst-core_Make-Dist

  • Build Num: 509
  • Status: PASSED

Build Information

Test Name: SST__AutotestGen2_NewFW_sst-test_Clang-Format_sst-core

  • Build Num: 463
  • Status: PASSED

Build Information

Test Name: SST__AutotestGen2_NewFW_OSX-14-XC15-ARM2_OMPI-4.1.6_PY3.10_sst-elements

  • Build Num: 121
  • Status: PASSED

Build Information

Test Name: SST__AutotestGen2_NewFW_OSX-14-XC15-ARM2_OMPI-4.1.6_PY3.10_sst-macro_withsstcore

  • Build Num: 83
  • Status: PASSED

sst-autotester avatar Aug 29 '24 16:08 sst-autotester

Status Flag 'Pre-Merge Inspection' - - This Pull Request Requires Inspection... The code must be inspected by a member of the Team before Testing/Merging WARNING: NO REVIEWERS HAVE BEEN REQUESTED FOR THIS PULL REQUEST!

sst-autotester avatar Aug 29 '24 16:08 sst-autotester

All Jobs Finished; status = PASSED, However Inspection must be performed before merge can occur...

sst-autotester avatar Aug 29 '24 16:08 sst-autotester

All Jobs Finished; status = PASSED, However Inspection must be performed before merge can occur...

sst-autotester avatar Aug 29 '24 20:08 sst-autotester

All Jobs Finished; status = PASSED, However Inspection must be performed before merge can occur...

sst-autotester avatar Aug 30 '24 00:08 sst-autotester

All Jobs Finished; status = PASSED, However Inspection must be performed before merge can occur...

sst-autotester avatar Aug 30 '24 05:08 sst-autotester

All Jobs Finished; status = PASSED, However Inspection must be performed before merge can occur...

sst-autotester avatar Aug 30 '24 09:08 sst-autotester

All Jobs Finished; status = PASSED, However Inspection must be performed before merge can occur...

sst-autotester avatar Aug 30 '24 13:08 sst-autotester

All Jobs Finished; status = PASSED, However Inspection must be performed before merge can occur...

sst-autotester avatar Aug 30 '24 18:08 sst-autotester

All Jobs Finished; status = PASSED, However Inspection must be performed before merge can occur...

sst-autotester avatar Aug 30 '24 23:08 sst-autotester

All Jobs Finished; status = PASSED, However Inspection must be performed before merge can occur...

sst-autotester avatar Aug 31 '24 03:08 sst-autotester

All Jobs Finished; status = PASSED, However Inspection must be performed before merge can occur...

sst-autotester avatar Aug 31 '24 07:08 sst-autotester

All Jobs Finished; status = PASSED, However Inspection must be performed before merge can occur...

sst-autotester avatar Aug 31 '24 12:08 sst-autotester

All Jobs Finished; status = PASSED, However Inspection must be performed before merge can occur...

sst-autotester avatar Aug 31 '24 16:08 sst-autotester

All Jobs Finished; status = PASSED, However Inspection must be performed before merge can occur...

sst-autotester avatar Aug 31 '24 21:08 sst-autotester

All Jobs Finished; status = PASSED, However Inspection must be performed before merge can occur...

sst-autotester avatar Sep 01 '24 01:09 sst-autotester

All Jobs Finished; status = PASSED, However Inspection must be performed before merge can occur...

sst-autotester avatar Sep 01 '24 05:09 sst-autotester