DataflowTemplates icon indicating copy to clipboard operation
DataflowTemplates copied to clipboard

Add jinja preprocessing to YamlTemplate

Open Polber opened this issue 1 year ago • 1 comments

Adds ability to perform jinja preprocessing to Beam YAML jobs run on YamlTemplate until the functionaiity is added to vanilla Beam YAML in Beam 2.47.0

Adds template parameter --jinja_variables

Polber avatar May 02 '24 00:05 Polber

Codecov Report

All modified and coverable lines are covered by tests :white_check_mark:

Project coverage is 40.18%. Comparing base (8936b1d) to head (b961be5). Report is 3 commits behind head on main.

Additional details and impacted files
@@             Coverage Diff              @@
##               main    #1500      +/-   ##
============================================
+ Coverage     39.88%   40.18%   +0.29%     
- Complexity     2782     2792      +10     
============================================
  Files           750      737      -13     
  Lines         42568    42768     +200     
  Branches       4555     4578      +23     
============================================
+ Hits          16980    17186     +206     
+ Misses        24121    24101      -20     
- Partials       1467     1481      +14     
Components Coverage Δ
spanner-templates 56.79% <46.86%> (+0.07%) :arrow_up:
spanner-import-export 65.50% <ø> (-0.07%) :arrow_down:
spanner-live-forward-migration 63.17% <94.73%> (+1.93%) :arrow_up:
spanner-live-reverse-replication 42.95% <93.61%> (+0.42%) :arrow_up:
spanner-bulk-migration 69.97% <45.05%> (-2.03%) :arrow_down:

see 35 files with indirect coverage changes

codecov[bot] avatar May 02 '24 00:05 codecov[bot]

@damccorm @robertwb After reconsidering, I decided just to copy over the main from https://github.com/apache/beam/pull/30976 for the time being. There were too many issues involving overridden pipeline args, and it makes more sense to me to just copy over the same logic that will be in Beam 2.57.0 instead of trying to stitch pieces together. Once 2.57.0 is release, assuming the jinja PR is merged, I will revert the template back to how it was before.

Polber avatar May 07 '24 14:05 Polber

Could we add a GitHub issue to track the work to remove these pieces before proceeding? Otherwise LGTM

#1554

Polber avatar May 13 '24 20:05 Polber