hedera-services icon indicating copy to clipboard operation
hedera-services copied to clipboard

ci: Aggregate otter tests results for testContainer

Open rbarker-dev opened this issue 2 months ago • 6 comments

Description

This pull request introduces improvements to how output directories and test artifacts are managed and uploaded for Otter integration and container tests. The changes make the test environments more configurable, ensure logs and reports are better organized per test, and update CI workflows to use the new artifact locations.

Test Environment and Output Directory Enhancements:

  • Added an outputDirectory() method to the TestEnvironment interface and implemented it in ContainerTestEnvironment, allowing tests to specify and retrieve their output directories for artifacts. [1] [2]
  • Refactored ContainerTestEnvironment to accept a custom output directory, defaulting to build/aggregateTestContainer, and updated constructors accordingly. [1] [2] [3]
  • Introduced a new abstract BaseIntegrationTest class that sets up a unique output directory for each test run and provides factory methods for creating TurtleTestEnvironment and ContainerTestEnvironment instances with the correct output paths.

JUnit Extension and Lifecycle Improvements:

  • Updated the OtterTestExtension JUnit extension to set up a unique output directory for each test before execution and store it in the extension context, supporting both "turtle" and "container" environments via a factory function. [1] [2] [3]
  • Added a new Lifecycle record to manage the lifecycle of a TestEnvironment within the JUnit extension context.

CI/CD Workflow Updates:

  • Changed artifact upload paths in .github/workflows/node-zxc-compile-application-code.yaml and .github/workflows/zxc-execute-otter-tests.yaml to use the new aggregated directories, improving log organization and retention. [1] [2] [3]
  • Updated workflow manifest and test result file references to match new output locations and naming conventions. [1] [2] [3] [4]

These changes make test output management more robust and flexible, simplify artifact uploads, and ensure that logs and reports are consistently organized per test run.

Related Issue(s)

Closes #21957

rbarker-dev avatar Nov 03 '25 20:11 rbarker-dev

:white_check_mark: Snyk checks have passed. No issues have been found so far.

Status Scanner Critical High Medium Low Total (0)
:white_check_mark: Open Source Security 0 0 0 0 0 issues

:computer: Catch issues earlier using the plugins for VS Code, JetBrains IDEs, Visual Studio, and Eclipse.

lfdt-bot avatar Nov 03 '25 21:11 lfdt-bot

Codecov Report

:white_check_mark: All modified and coverable lines are covered by tests.

Impacted file tree graph

@@             Coverage Diff              @@
##               main   #21979      +/-   ##
============================================
+ Coverage     70.89%   70.94%   +0.04%     
  Complexity    24432    24432              
============================================
  Files          2668     2668              
  Lines        104410   104336      -74     
  Branches      10966    10955      -11     
============================================
- Hits          74020    74019       -1     
+ Misses        26344    26274      -70     
+ Partials       4046     4043       -3     

see 20 files with indirect coverage changes

Impacted file tree graph

:rocket: New features to boost your workflow:
  • :snowflake: Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

codecov[bot] avatar Nov 03 '25 21:11 codecov[bot]

Coverage summary from Codacy

See diff coverage on Codacy

Coverage variation Diff coverage
:white_check_mark: +0.05% (target: -1.00%) :white_check_mark:
Coverage variation details
Coverable lines Covered lines Coverage
Common ancestor commit (88c39cfc54018f6cf1afe7e052a7bf92f423fe15) 104315 78021 74.79%
Head commit (db4b1871f5304512a8e8b7ae7688701ae89c5962) 104241 (-74) 78017 (-4) 74.84% (+0.05%)

Coverage variation is the difference between the coverage for the head and common ancestor commits of the pull request branch: <coverage of head commit> - <coverage of common ancestor commit>

Diff coverage details
Coverable lines Covered lines Diff coverage
Pull request (#21979) 0 0 ∅ (not applicable)

Diff coverage is the percentage of lines that are covered by tests out of the coverable lines that the pull request added or modified: <covered lines added or modified>/<coverable lines added or modified> * 100%

See your quality gate settings    Change summary preferences

codacy-production[bot] avatar Nov 03 '25 21:11 codacy-production[bot]

Local testing passes:

image

rbarker-dev avatar Nov 04 '25 21:11 rbarker-dev

XTS Dry Run with Otter Tests After Rebase.

rbarker-dev avatar Nov 06 '25 15:11 rbarker-dev