[epic]: MATS Testing and Delivery Pipeline CI/CD Phase 1 tasks

[rbarkerSL]

Background

Refer to phase 1 design

The CI/CD team is supporting the effor to implement MATS tests and enable continuous delivery from the mainline of hashgraph/hedera-services. This estimate goes over the anticpated effort for the following tasks:

Task List

• [ ] #14571
  • look at all workflows currently running on push to develop
  • Runs minimal acceptable test suite
    • MATS runs do not use tags since they execute on every PR/commit merged to main
  • Look at breaking the unit test flows into independent parts
    • MATS
    • Hammer
    • Long Running
    • Performance
    • Smoke
    • Fuzz
  • look at how/where jacoco/codecov tasks are called?
  • look at how/where adhoc release is called
    • ensure adhoc release runs after all testing iff all pre-tests are passing
  • ensure publishing jacoco/codecov results as build artifact in github actions
• [ ] #14570
  • Execute once every 3 hours
  • Grab the latest untested commit from develop
  • Retrieve the latest jacoco/codecov artifacts
  • Run long unit tests (> 30 minutes)
  • Run Extended Test Suite
    • Hammer Tests
    • Fuzz Tests
    • Testnet migration tests - what environment? GCE?
    • Mainnet migration tests - what environment? GCE?
  • Capture jacoco/codecov coverage results as build artifact
  • Publish jacoco and codecov results for the commit
    • Publish the results from the commits MATS timeframe
    • Publish the results from the commits ETS timeframe
  • Scenario -- Tests pass:
    • Tag the build with a promotion marker (label?)
    • Add temporary tags to the commit that passed the 3 hour ETS
  • Scenario -- Tests fail:
    • Notify through pager duty
    • Notify through slack
    • Using rootly?
• [ ] #14650
  • The build ID should only be tagged once per day. Eg we produce a single build candidate daily and not every 3 hours.
  • Pick the latest passing ETS commit
  • Assign it a build ID
  • clear the temp tags from the other commits
• [ ] #14572
  • Document in hedera-services or notion
  • Release Engineering will create the tagging schema
  • Must be compatible with all four phases of the continuous deployment process
• [ ] #14573
  • Document in hedera-services or notion
  • add labels to hedera-services
  • Release Engineering will handle standardization of build promotion
• [ ] #14574
  • Make updates in hedera-services as needed
• [ ] Test standardization effort
  • Standardardize test, test suite, and test class names
• [ ] #11564
  • Gradle changes to support MATS and other test types
  • Gradle changes to restructure tests into MATS and other test types
  • Requires engagement with other engineering teams
  • Split tests into appropriate source sets
    • MATS
    • Hammer
    • Long Running
    • Performance
    • Smoke
    • Fuzz

Estimate

1. Modifying existing workflows - 2 weeks
2. Gradle Tasks - 3 weeks
3. Create a new flow that executes on main - 2 week
4. Create a new flow that executes daily to facilitate automated tagging and build promotion - 2 weeks
5. Standardize/document tagging scheme for MATS/ETS commits - 2 weeks
6. Standardize/document promotion labels/tags - 2 weeks
7. Standardize/update existing workflow files - 3 weeks

---

• Estiamted time to complete: 16-20 weeks