airbyte icon indicating copy to clipboard operation
airbyte copied to clipboard

Published 20 hours ago • verified publisher icon airbytehq

Destination bigquery: airbyte_meta/sync_id/generation_id

Open edgao opened this issue 9 months ago • 2 comments

  • create new raw tables with meta+gen ID (also partition on gen ID - this seems like a good idea? but could be convinced to skip it for now)
  • write meta+gen ID to raw tables in direct upload mode
    • gcs upload mode is handled by instantiating StagingStreamOperations using V2_WITH_GENERATION (see BigqueryDestination)
  • pass generation ID through from raw to final table
  • concat airbyte_meta.changes from raw to final table
  • giant pile of test fixture updates, including a new test case to exercise the migration directly

edgao avatar May 20 '24 18:05 edgao