Upgrade to Delta 3.1.0

[osopardo1]

Within the past weeks, Delta Lake released its version 3.1.0 with some exciting opportunities for integrations with the ecosystem.

Some of the highlights are:

• Preview of Liquid Clustering. Now Delta allows clustering the data in a Delta table for better data skipping.
• Auto-Compaction to address the small files problem during table writes.
• Support for optimizing min/max aggregations using the table metadata
• Fixed some issues with column renames in Merge.

Find the full notes here.

We should:

• [ ] Check which APIs have changed and if they affect our code.
• [ ] Agreed on which version of Qbeast-Spark should we add this change. The initial idea is to upgrade it on top of 1.0.0, but then there will be no version for Delta 3.0.0 (unless we publish within a different version number). We agree to constantly support the latest versions, and for the size of the project, it's ok.
• [ ] Upgrade Delta Library on the branch.
• [ ] Update the APIs.
• [ ] Run tests
• [ ] Publish SNAPSHOT with the last commit ID.
• [ ] Update other deployments that might contain the Qbeast and Delta Lake images.