airbyte icon indicating copy to clipboard operation
airbyte copied to clipboard

Java CDK updates for mysql

Open edgao opened this issue 10 months ago • 4 comments

ported from https://github.com/airbytehq/airbyte/pull/34608 to kotlin.

  • TestDatabase
    • general cleanup (nullability, etc)
    • ConfigBuilder.build() now declares ObjectNode as its return type for convenience
  • AbstractJdbcDestination - allow injecting a custom v1v2 migrator
  • JdbcDestinationHandler
    • Make databaseName nullable (see comment for reason)
    • allow overriding the data type for _airbyte_destination_state.updated_at (mysql doesn't have a timestamp_with_timezone type)
    • allow overriding a few DB metadata-related things (mysql doesn't have schemas, so need to pass namespace into the database argument)
    • type name comparison now uses .equals(..., ignoreCase=true)
  • JdbcSqlGenerator
    • add a formatTimestampLiteral method
    • kill the createIndexes method - implementing classes can just return a Sql object with multiple statements
    • use dslContext everywhere instead of static DSL
    • allow overriding renameTable and createV2RawTableFromV1Table (once again, to support mysql quirks)
  • JdbcV1V2Migrator - more open stuff, make databaseName nullable
  • JdbcSqlGeneratorIntegrationTest - nullability, make dslContext protected instead of private
  • JdbcTypingDedupingTest
    • make more things open/protected
    • call convertStreamName on the raw table namespace (this is to match the JdbcSqlGenerator behavior, so that tests can correctly find the raw table)
    • more DSL -> dslContext changes
  • JdbcBufferedConsumerFactory - use ParsedCatalog as source of truth for raw table name
  • JdbcSqlOperations: modify airbyte_meta handling (write explicit value when null)
  • CatalogParser - correctly propagate the original stream name+namespace when there are table name collisions
  • BaseTypingDedupingTest
    • add more tests/assertions
  • misc code cleanup
    • lots of making things nonnullable

edgao avatar Apr 09 '24 16:04 edgao

The latest updates on your projects. Learn more about Vercel for Git ↗︎

1 Ignored Deployment
Name Status Preview Comments Updated (UTC)
airbyte-docs ⬜️ Ignored (Inspect) Visit Preview May 3, 2024 7:38pm

vercel[bot] avatar Apr 09 '24 16:04 vercel[bot]

  • #36935 Graphite: 6 dependent PRs (#34611 Graphite, #34612 Graphite, #34613 Graphite and 3 others)
  • #36929 Graphite 👈
  • #36926 Graphite
  • master

This stack of pull requests is managed by Graphite. Learn more about stacking.

Join @edgao and the rest of your teammates on Graphite Graphite

edgao avatar Apr 11 '24 21:04 edgao

made a few minor changes to handle mysql's spec using database instead of schema for the default namespace field, plus merge conflicts (more nullability stuff, and a new @Suppress(deprecation) thing)

edgao avatar Apr 23 '24 22:04 edgao

... and fixed SpecModifyingDestination to correctly proxy the isV2Destination call to the underlying destination object

edgao avatar Apr 24 '24 16:04 edgao

/publish-java-cdk

:clock2: https://github.com/airbytehq/airbyte/actions/runs/8944047470 :white_check_mark: Successfully published Java CDK version=0.32.0!

edgao avatar May 03 '24 19:05 edgao