airbyte icon indicating copy to clipboard operation
airbyte copied to clipboard

✨Source Outreach: Migrate Python CDK to Low-code CDK

Open btkcodedev opened this issue 10 months ago • 14 comments

What

Migrating Source Outreach to Low-Code CDK Closes airbytehq/airbyte-internal-issues#6916

How

Developed using (Configuration Based Source) low-code CDK

Recommended reading order

  1. spec.yaml
  2. manifest.yaml
  3. schemas/*

Tests

Integration & Acceptance
  • Test successful with incremental syncs

image

🚨 User Impact 🚨

  • Migration to low-code

btkcodedev avatar Apr 10 '24 07:04 btkcodedev

The latest updates on your projects. Learn more about Vercel for Git ↗︎

Name Status Preview Comments Updated (UTC)
airbyte-docs ✅ Ready (Inspect) Visit Preview 💬 Add feedback May 17, 2024 3:20pm

vercel[bot] avatar Apr 10 '24 07:04 vercel[bot]

Inline schema :white_check_mark: Poetry lock file ✅ Breaking change referred in metadata :white_check_mark: Add migration.md docs file :white_check_mark: Docs updated ✅ Formatted :white_check_mark:

airbyte-ci test - All pass image

btkcodedev avatar Apr 10 '24 07:04 btkcodedev

Marking PR as ready to merge

btkcodedev avatar Apr 10 '24 07:04 btkcodedev

@btkcodedev can you run airbyte-ci format fix all to fix the formatting issues?

girarda avatar Apr 10 '24 19:04 girarda

Done @girarda !!

btkcodedev avatar Apr 11 '24 01:04 btkcodedev

@lazebnyi,

  1. those schemas looks exactly the same as previous version json, Could you please look again where is the fault, the connection looks good in local and also in Airbyte ci to,

  2. There is a schema change from backend in sequence_steps_schema where changed its ['properties']['creator']['items']['type'] field to include integers, some of them are integers instead of strings

  3. As the scope are not defined in the previous version and is hardcoded, user may need to access the resources, so for safety I've included all the scopes necessary for the streams.

btkcodedev avatar Apr 16 '24 18:04 btkcodedev

@btkcodedev

  1. Ok, I'll check and get back to you.
  2. In that way we need use string in schema and use TypeTransformer for integers
  3. However, having all scopes is not necessary to sync a specific stream. By default, the availability check will filter the stream if it's not accessible to the user.

lazebnyi avatar Apr 17 '24 12:04 lazebnyi

@lazebnyi Any updates🙂

btkcodedev avatar Apr 21 '24 15:04 btkcodedev

@btkcodedev Please check my comment above.

You need update schema and use TypeTransformer and delete scopes list in the code as it is not require to have all scopes to start sync for some special streams.

lazebnyi avatar Apr 24 '24 22:04 lazebnyi

@lazebnyi,

  1. The Schema look good, and it already includes all the mentioned fields in https://github.com/airbytehq/airbyte/pull/36954#pullrequestreview-2002425832

  2. There is no need for TypeTransformation not needed as it includes only integers instead of strings,

  3. Scope has been reset according to the given LastPass notes

-> The connection is failing as the refresh token is expired I think, The creds need to be refreshed image

btkcodedev avatar Apr 25 '24 05:04 btkcodedev

@lazebnyi Scopes has been reset to the ones in LastPass, Connection check and all other checks are passing in the CI, Schema looks same as mentioned, Please continue the merge process

btkcodedev avatar Apr 28 '24 04:04 btkcodedev

CI Passed perfectly image

btkcodedev avatar Apr 28 '24 04:04 btkcodedev

/format-fix

Format-fix job started... Check job output.

✅ Changes applied successfully. (263f800b7dfe93993428f5dccc367618ae86db4b)

btkcodedev avatar May 09 '24 08:05 btkcodedev

@lazebnyi, Resolved comments and CI green :white_check_mark: Ready to merge :bow:

btkcodedev avatar May 16 '24 03:05 btkcodedev

@katmarkham, Docs checked, Ready for the approval :bow:

btkcodedev avatar May 17 '24 16:05 btkcodedev