Improve DataFusion CI

[alamb]

Is your feature request related to a problem or challenge? Please describe what you are trying to do. The CI system in arrow-datafusion largely dates to the time when it was part of the arrow repository. It is all in a single workflow, which among other things prevents more specific test triggering and also retriggering the entire run if just a single check fails

Describe the solution you'd like I want to clean up the DataFusion CI system in a as-yet-to-be-determined way to make it faster and more-efficient

Describe alternatives you've considered leave it alone

Additional context I recently went through the same exercise in https://github.com/apache/arrow-rs/issues/2149. I am not as

Inspired by https://github.com/apache/arrow-datafusion/pull/3043