Destination bigquery: airbyte_meta/sync_id/generation_id

[edgao]

• create new raw tables with meta+gen ID (also partition on gen ID - this seems like a good idea? but could be convinced to skip it for now)
• write meta+gen ID to raw tables in direct upload mode
  • gcs upload mode is handled by instantiating StagingStreamOperations using V2_WITH_GENERATION (see BigqueryDestination)
• pass generation ID through from raw to final table
• concat airbyte_meta.changes from raw to final table
• giant pile of test fixture updates, including a new test case to exercise the migration directly

[vercel[bot]]

The latest updates on your projects. Learn more about Vercel for Git ↗︎

Name	Status	Preview	Comments	Updated (UTC)
airbyte-docs	✅ Ready (Inspect)	Visit Preview	💬 Add feedback	May 29, 2024 6:32pm

[edgao]

• #38359 👈
• #38358
• #38738 : 1 other dependent PR (#38658 )
• master

This stack of pull requests is managed by Graphite. Learn more about stacking.

Join @edgao and the rest of your teammates on Graphite