perf(airbyte-cdk): replace `json` with `orjson`
What
Resolve https://github.com/airbytehq/airbyte-internal-issues/issues/9428 Reason: orjson is much faster than standard json
How
replace json with orjson
Review guide
User Impact
Can this PR be safely reverted and rolled back?
- [ ] YES 💚
- [ ] NO ❌
The latest updates on your projects. Learn more about Vercel for Git ↗︎
1 Skipped Deployment
| Name | Status | Preview | Comments | Updated (UTC) |
|---|---|---|---|---|
| airbyte-docs | ⬜️ Ignored (Inspect) | Visit Preview | Aug 27, 2024 0:15am |
⚡️ Codeflash found optimizations for this PR
📄 SimpleRetriever._parse_response() in airbyte-cdk/python/airbyte_cdk/sources/declarative/retrievers/simple_retriever.py
📈 Performance improved by 20% (0.20x faster)
⏱️ Runtime went down from 30.0 microseconds to 24.9 microseconds
I created a new dependent PR with the suggested changes. Please review:
-
#44814
If you approve, it will be merged into this PR (branch artem1205/airbyte-cdk-migrate-json-to-orjson).
@artem1205 do we have numbers on how much it improves things in our case?
@maxi297, this is a good question, while i'm not sure whether we'll see the real difference in all cases.
Most important/heavily used is change in JsonlDecoder https://github.com/airbytehq/airbyte/pull/44589/files#diff-404f304b08b4a327259c7999f75283e1f111edaac2bb09196cb2c536f141f2deL74-L75
The difference for the test test_jsonl_decoder_memory_usage is about 20%.
⚡️ Codeflash found optimizations for this PR
📄 HttpRequest._to_mapping() in airbyte-cdk/python/airbyte_cdk/test/mock_http/request.py
📈 Performance improved by 31% (0.31x faster)
⏱️ Runtime went down from 663 microseconds to 504 microseconds
I created a new dependent PR with the suggested changes. Please review:
-
#44821
If you approve, it will be merged into this PR (branch artem1205/airbyte-cdk-migrate-json-to-orjson).