Joel

If that's the case, we should be able to write a `kedro upgrade` function?

This is slightly unscientific, but I trust the vibes in the industry enough to say Iceberg will clearly be the winner in long term.  Plus people saying things like...

Delta is 100% more mature, Iceberg is the horse to back. This is the thread I was trying to find earlier: https://x.com/sean_lynch/status/1845500735842390276 I also don't think we should be wedded...

So I'm actually being bullish and saying we _should_ pick one of these when it comes to **our idea of versioned data**. We simply don't have capacity to integrate everywhere...

Super cool application of these concepts https://datamonkeysite.com/2024/11/10/smart-data-pipeline-design-check-for-delta-table-changes-with-minimal-overhead/

>I'm with @deepyaman on this one. There should be a layer in Kedro that is format-agnostic. We can be more opinionated in a higher layer. I just want to warn...

My view: >1. Does it make sense to use it for anything else but versioning tabular data? I'm willing to bet >95% of use cases fall into this. > 2....

Thanks Nok this is super helpful, in general `upserts` don't canonically make sense in Kedro

Also search by kind - if I wanted to find all Parquet files today I'd have to get very creative. Retrieving paths associated with those would be super complicated.

Why not expose `catalog.keys()` I think they need a way of accessing things declared in the catalog in interactive environments