airbyte
airbyte copied to clipboard
airbytehq
✨Source Aws Cloudtrail: Migrate Python CDK to Low-code CDK
What
Migrating Source Aws Cloudtrail to Low-Code CDK Closes airbytehq/airbyte-internal-issues#7042
How
Developed using (Configuration Based Source) low-code CDK
Recommended reading order
-
spec.yaml -
manifest.yaml -
schemas/*
Tests
Integration & Acceptance
- Test sucessfull
🚨 User Impact 🚨
- Spec Breaking changes, migration to low-code
The latest updates on your projects. Learn more about Vercel for Git ↗︎
| Name | Status | Preview | Comments | Updated (UTC) |
|---|---|---|---|---|
| airbyte-docs | ✅ Ready (Inspect) | Visit Preview | 💬 Add feedback | Jun 25, 2024 7:13pm |
![vercel[bot] avatar](https://avatars.githubusercontent.com/in/8329?v=4 "Published by a pub.dev verified publisher"){kind=small}
Inline Schema ✅ Poetry lock file ✅ Breaking change referred in metadata ✅ Add migration.md docs file ✅ Docs updated ✅ Incremental Sync :white_check_mark: Formatted :white_check_mark:
airbyte-ci test All pass state:
Results:
The region is hardcoded as the other regions us-west cannot be used while using Rest endpoints,the previous connector was using boto3 client for creating sessions, it was changed to custom Rest API approach, The Aws documentation is very poor regarding this, Other regions were tested as API base URL but not found working correctly. Thus the API URL could only use US-east region
The region is hardcoded as the other regions us-west cannot be used while using Rest endpoints,the previous connector was using boto3 client for creating sessions, it was changed to custom Rest API approach, The Aws documentation is very poor regarding this, Other regions were tested as API base URL but not found working correctly. Thus the API URL could only use US-east region
@btkcodedev What do you mean when you say that it is not working properly for the REST API approach? I have used us-west-2 for your implementation, with deleted the LookupAttributes filter, and it is working as expected.
Oh I haven't tried with filter deleted 😅, thanks for the info, I'll fix the comments 🙇
@lazebnyi Thanks for the info about the filter, I've removed it,
Currently the CI is getting timeout due to the high amount of record fetch with incremental sync,
There is a need to change the Start_Date in config to the today's date to limit the records to pass the CI Thanks!
There is a need to change the Start_Date in config to the today's date to limit the records to pass the CI Thanks!
@btkcodedev is today's date 17/04/2024 ?
@lazebnyi The current date, example today- 30/04/2024 to fetch minimal records
@lazebnyi, The connector is always getting timeout due to the high amount of records, Do you have any idea how to resolve this? adding a filter or something
The region is hardcoded as the other regions us-west cannot be used while using Rest endpoints,the previous connector was using boto3 client for creating sessions, it was changed to custom Rest API approach
If it needs boto3, perhaps we should move that to be a file-based source instead?
If using REST limits the connector to be available for just one region, that's bad and not what we want.
No the problem is solved, it works on any regions, the problem here is its getting timed out as there are very high amount of records @natikgadzhi
@btkcodedev The acceptance tests fail with timeout because the connector no longer limits the number of records during the tests. We need to find a new way to limit records.
I've added a new filtering capability and sent credentials via slack, hope that does the job
I've updated the connector to take the default start date at the current date and sent the config to Daniel Jablonski. Waiting for an update from him.
@btkcodedev Please update the docs with the new default
start_date.
Done