jitsu icon indicating copy to clipboard operation
jitsu copied to clipboard

Improve patch table function

Open absorbb opened this issue 3 years ago • 0 comments

Problem

Currently patchTable checks difference between table schema in destination and table schema derived from data. But it ignores column type difference completely. So if column types differ we attempt to do insert and fail with error that not always conclusive.

Solution

  • tableHelper.EnsureTable: must return table schema with column types derived from current data (right now types comes from destination table)
  • tableHelper.EnsureTable: must additionally return column types diffs object
  • if consequent insert fails we need to print extra error about column types diffs that could cause this error and offer solution: sql_type mapping or alter table

Optional:

  • For sources sync only we can find common type parents for conflicting columns, create new table with fixed types and fill it with data from destination_table and tmp_table

absorbb avatar Apr 12 '22 18:04 absorbb