datafusion-comet Switch back to released version of DataFusion and arrow-rs after Arrow Java 16 is released

What is the problem the feature request solves?

In #239, we moved to use specified forked of DataFusion and arrow-rs as a temporary workaround to the Arrow Java bug which sends invalid offset buffers through C Data Interface.

The fix to the bug was merged to Arrow Java. Once Arrow Java 16 is released, we should move back to released versions of DataFusion and arrow-rs.

Describe the potential solution

No response

Additional context

No response

Apr 08 '24 18:04 viirya

Arrow 16 has now been released:

https://arrow.apache.org/release/16.0.0.html

Apr 25 '24 20:04 andygrove

Got it. Thanks.

Apr 25 '24 20:04 viirya

I would like to hold this until #250 is merged. Assigned to myself.

Apr 26 '24 18:04 viirya

@viirya Is there anything stopping us from depending on the datafusion repo directly now (once https://github.com/apache/datafusion/pull/10702 is merged) rather than your fork?

May 30 '24 14:05 andygrove

Let me try to switch to official DataFusion repo.

May 30 '24 15:05 viirya

There are some changes from latest DataFusion repo. I'm working on to update Comet with it.

May 30 '24 18:05 viirya

The release vote for arrow-rs 52 has started, and there is a draft PR in DataFusion to upgrade to this version:

https://github.com/apache/datafusion/pull/10765

Jun 03 '24 16:06 andygrove

Cool. I will update the PR accordingly once we get DataFusion and arrow-rs releases.

Jun 03 '24 16:06 viirya