Add jinja preprocessing to YamlTemplate
Adds ability to perform jinja preprocessing to Beam YAML jobs run on YamlTemplate until the functionaiity is added to vanilla Beam YAML in Beam 2.47.0
Adds template parameter --jinja_variables
Codecov Report
All modified and coverable lines are covered by tests :white_check_mark:
Project coverage is 40.18%. Comparing base (
8936b1d) to head (b961be5). Report is 3 commits behind head on main.
Additional details and impacted files
@@ Coverage Diff @@
## main #1500 +/- ##
============================================
+ Coverage 39.88% 40.18% +0.29%
- Complexity 2782 2792 +10
============================================
Files 750 737 -13
Lines 42568 42768 +200
Branches 4555 4578 +23
============================================
+ Hits 16980 17186 +206
+ Misses 24121 24101 -20
- Partials 1467 1481 +14
| Components | Coverage Δ | |
|---|---|---|
| spanner-templates | 56.79% <46.86%> (+0.07%) |
:arrow_up: |
| spanner-import-export | 65.50% <ø> (-0.07%) |
:arrow_down: |
| spanner-live-forward-migration | 63.17% <94.73%> (+1.93%) |
:arrow_up: |
| spanner-live-reverse-replication | 42.95% <93.61%> (+0.42%) |
:arrow_up: |
| spanner-bulk-migration | 69.97% <45.05%> (-2.03%) |
:arrow_down: |
@damccorm @robertwb After reconsidering, I decided just to copy over the main from https://github.com/apache/beam/pull/30976 for the time being. There were too many issues involving overridden pipeline args, and it makes more sense to me to just copy over the same logic that will be in Beam 2.57.0 instead of trying to stitch pieces together. Once 2.57.0 is release, assuming the jinja PR is merged, I will revert the template back to how it was before.
Could we add a GitHub issue to track the work to remove these pieces before proceeding? Otherwise LGTM
#1554