[DISCUSS] DataFusion upgrade policy

[andygrove]

What is the problem the feature request solves?

During the Comet community call this morning, we discussed some options around how we keep up to date with changes in DataFusion core.

There are two main options:

1. Always have the Comet main branch use an official DataFusion release from crates.io
2. Between Comet releases, keep updating the main branch to use a recent revision of DataFusion and update it as needed to pick up new features / bug fixes

There are pros and cons to each approach.

Option 1 has less overhead, but means we have to wait longer for fixes, and could also find out that we are affected by breaking changes after DataFusion is already released.

Option 2 allows us to catch regressions earlier but means we cannot release a new version until DataFusion releases (although we would still be able to create patch releases from the previous release branch, using an older version of DataFusion).

Let's use this issue to discuss.

[viirya]

1. Always have the Comet main branch use an official DataFusion release from crates.io
2. Between Comet releases, keep updating the main branch to use a recent revision of DataFusion and update it as needed to pick up new features / bug fixes

I think we can always use an official DataFusion release from crates.io for a release of Comet.

But it is hard to guarantee that the Comet main branch always uses an official DataFusion release at any moment. For example, we might temporarily switch to revisions of DataFusion/arrow-rs to add workarounds/fixes or get features which are not available yet in the official releases.