PyAirbyte icon indicating copy to clipboard operation
PyAirbyte copied to clipboard

Refactor: new "Writers" interface for destinations and caches

Open aaronsteers opened this issue 6 months ago • 1 comments

Summary by CodeRabbit

  • New Features

    • Enhanced stream management with new methods for primary key and cursor key retrieval, as well as improved write strategy resolution in the CatalogProvider.
    • Streamlined processing of Airbyte messages in the SqlProcessorBase, supporting various message types and improving state management.
    • Expanded telemetry capabilities to accommodate a broader range of destination types, including support for AirbyteWriterInterface.
    • Introduced AirbyteWriterInterface for flexible message writing across various integrations.
    • Added the _write_airbyte_message_stream method to CacheBase to facilitate reading and writing of Airbyte messages.
  • Bug Fixes

    • Improved clarity in the Destination class's write method documentation and handling of write strategies.
  • Documentation

    • Updated method docstrings to clarify functionality and expected behavior for enhanced user understanding.
  • Refactor

    • Restructured the FileWriterBase class to align with the AirbyteWriterInterface, enhancing its integration within the Airbyte framework.

aaronsteers avatar Aug 05 '24 20:08 aaronsteers