DataflowTemplates icon indicating copy to clipboard operation
DataflowTemplates copied to clipboard

Session file support for bulk migration using JSON node

Open Deep1998 opened this issue 1 year ago • 1 comments

Contains the session file support in the bulk template using JSON node as the standard format. If we decide to finalise avro, we will update this PR with avro support.

This PR supports:

  • Name transformations, synth PK handling and dropped columns using session file.
  • Almost all mysql data types except MySQL YEAR and BIT.

This PR does not support:

  • Sharding support (shard id population)
  • Transformation context

Deep1998 avatar Mar 27 '24 11:03 Deep1998

Codecov Report

Merging #1400 (fe123de) into main (f8653ef) will increase coverage by 0.07%. Report is 11 commits behind head on main. The diff coverage is 100.00%.

Additional details and impacted files
@@             Coverage Diff              @@
##               main    #1400      +/-   ##
============================================
+ Coverage     38.92%   38.99%   +0.07%     
+ Complexity     2761     2759       -2     
============================================
  Files           743      745       +2     
  Lines         41939    41992      +53     
  Branches       4516     4527      +11     
============================================
+ Hits          16324    16375      +51     
  Misses        24127    24127              
- Partials       1488     1490       +2     
Components Coverage Δ
spanner-templates 52.46% <100.00%> (+0.28%) :arrow_up:
spanner-import-export 65.22% <ø> (-0.03%) :arrow_down:
spanner-live-forward-migration 57.20% <100.00%> (+0.97%) :arrow_up:
spanner-live-reverse-replication 38.44% <100.00%> (+0.91%) :arrow_up:
spanner-bulk-migration 59.40% <100.00%> (+1.27%) :arrow_up:
Files Coverage Δ
...eport/v2/spanner/migrations/schema/LazyMapper.java 100.00% <100.00%> (ø)
.../spanner/migrations/schema/SessionBasedMapper.java 100.00% <100.00%> (ø)

... and 4 files with indirect coverage changes

codecov[bot] avatar Mar 27 '24 11:03 codecov[bot]