airbyte icon indicating copy to clipboard operation
airbyte copied to clipboard

perf(airbyte-cdk): replace `json` with `orjson`

Open artem1205 opened this issue 1 year ago • 1 comments

What

Resolve https://github.com/airbytehq/airbyte-internal-issues/issues/9428 Reason: orjson is much faster than standard json

How

replace json with orjson

Review guide

User Impact

Can this PR be safely reverted and rolled back?

  • [ ] YES 💚
  • [ ] NO ❌

artem1205 avatar Aug 23 '24 14:08 artem1205

The latest updates on your projects. Learn more about Vercel for Git ↗︎

1 Skipped Deployment
Name Status Preview Comments Updated (UTC)
airbyte-docs ⬜️ Ignored (Inspect) Visit Preview Aug 27, 2024 0:15am

vercel[bot] avatar Aug 23 '24 14:08 vercel[bot]

⚡️ Codeflash found optimizations for this PR

📄 SimpleRetriever._parse_response() in airbyte-cdk/python/airbyte_cdk/sources/declarative/retrievers/simple_retriever.py

📈 Performance improved by 20% (0.20x faster)

⏱️ Runtime went down from 30.0 microseconds to 24.9 microseconds

I created a new dependent PR with the suggested changes. Please review:

  • #44814

If you approve, it will be merged into this PR (branch artem1205/airbyte-cdk-migrate-json-to-orjson).

codeflash-ai[bot] avatar Aug 27 '24 13:08 codeflash-ai[bot]

@artem1205 do we have numbers on how much it improves things in our case?

maxi297 avatar Aug 27 '24 13:08 maxi297

@maxi297, this is a good question, while i'm not sure whether we'll see the real difference in all cases.

Most important/heavily used is change in JsonlDecoder https://github.com/airbytehq/airbyte/pull/44589/files#diff-404f304b08b4a327259c7999f75283e1f111edaac2bb09196cb2c536f141f2deL74-L75 The difference for the test test_jsonl_decoder_memory_usage is about 20%.

artem1205 avatar Aug 27 '24 13:08 artem1205

⚡️ Codeflash found optimizations for this PR

📄 HttpRequest._to_mapping() in airbyte-cdk/python/airbyte_cdk/test/mock_http/request.py

📈 Performance improved by 31% (0.31x faster)

⏱️ Runtime went down from 663 microseconds to 504 microseconds

I created a new dependent PR with the suggested changes. Please review:

  • #44821

If you approve, it will be merged into this PR (branch artem1205/airbyte-cdk-migrate-json-to-orjson).

codeflash-ai[bot] avatar Aug 27 '24 15:08 codeflash-ai[bot]