[Versioning]: Explore Kedro + Iceberg for versioning

[ElenaKhaustova]

Description

At the current stage by versioning we assume mapping a single version number to the corresponding versions of parameters, I/O data, and code. So one is able to retrieve a full project state including data at any point in time.

The goal is to check if we can use Iceberg to map a single version number to code, parameters, and I/O data within Kedro and how it aligns with Kedro’s workflow.

As a result, we expect a working example of kedro project used with Iceberg for versioning and some assumptions on:

• whether it solves the main task and what are the constraints;
• how easy is to set up;
• how the workflow looks like;
• whether any changes are required on the kedro side;
• what data formats are supported;
• how easy is to work with local/remote storage;
• how demanding is it in terms of dependencies.

Context

https://github.com/kedro-org/kedro/issues/4199

Market research