airbyte

airbyte copied to clipboard

…overy

What

We had a lot of trouble duplicating HTTP error handling. Using the HttpClient will allow us to make sure most of the error handling logic is just at one place and once HTTP requests are added or modified, they will leverage the same error handling.

The scope for now is bulk streams, authentication and schema discovery. REST is excluded because it relies on HttpStream which is not updated yet.

How

Update the airbyte_cdk to leverage HttpClient. Instantiate a HttpClient and use it for HTTP requests.

Review guide

1. For authentication and schema discover:airbyte-integrations/connectors/source-salesforce/source_salesforce/api.py
2. Use in bulk stream: airbyte-integrations/connectors/source-salesforce/source_salesforce/streams.py
3. Generalized error handling: airbyte-integrations/connectors/source-salesforce/source_salesforce/rate_limiting.py

User Impact

There should be none as this will improve the way we maintain the source.

Note that there is a small change in behavior where REST errors will retry differently but we don't expect this to have an impact on the end user

Can this PR be safely reverted and rolled back?

• [X] YES 💚
• [ ] NO ❌

The latest updates on your projects. Learn more about Vercel for Git ↗︎

Name	Status	Preview	Comments	Updated (UTC)
airbyte-docs	✅ Ready (Inspect)	Visit Preview	💬 Add feedback	May 24, 2024 0:50am

Note: there is an issue with retrying on authentication. I'll fix that soon and update the PR

Suspect Issues

This pull request was deployed and Sentry observed the following issues:

• ‼️ airbyte_cdk.utils.traced_exception.AirbyteTracedException: 'GET' request to 'https://energy-force-6851.my.salesforce.com/services/data/v57.0/sobjects' faile... /usr/local/lib/python3.9/site-packages/airbyte_... View Issue

_{Did you find this useful? React with a 👍 or 👎}