chore: Comet + Iceberg (1.8.1) CI

[hsiang-c]

Which issue does this PR close?

Closes #. https://github.com/apache/datafusion-comet/issues/1685

Rationale for this change

Run Iceberg Spark' tests as part of Comet CI

What changes are included in this PR?

• Produce a git diff bases on Iceberg version 1.8.1 (will work on other Iceberg versions (e.g. 1.9.x) later)
• Change the default value of Parquet Reader Type from ICEBERG to COMET
• Disable testMergeSchemaIgnoreCastingLongToInt and testMergeSchemaIgnoreCastingDoubleToFloat in TestDataFrameWriterV2 for both Iceberg Spark 3.4 and Iceberg Spark 3.5
• Run Iceberg Spark's tests, based on Iceberg's GitHub workflow: https://github.com/apache/iceberg/blob/main/.github/workflows/spark-ci.yml

How are these changes tested?

At the moment, locally:

# Spark 3.5
./gradlew -DsparkVersions=3.5 -DscalaVersion=2.12 -DflinkVersions= -DkafkaVersions= :iceberg-spark:iceberg-spark-3.5_2.12:check -Pquick=true -x javadoc

BUILD SUCCESSFUL in 26m 10s
46 actionable tasks: 7 executed, 39 up-to-date

./gradlew -DsparkVersions=3.5 -DscalaVersion=2.12 -DflinkVersions= -DkafkaVersions= :iceberg-spark:iceberg-spark-extensions-3.5_2.12:check -Pquick=true -x javadoc

BUILD SUCCESSFUL in 23m 44s
52 actionable tasks: 9 executed, 4 from cache, 39 up-to-date

./gradlew -DsparkVersions=3.5 -DscalaVersion=2.12 -DflinkVersions= -DkafkaVersions= :iceberg-spark:iceberg-spark-runtime-3.5_2.12:check -Pquick=true -x javadoc

BUILD SUCCESSFUL in 15s
65 actionable tasks: 4 executed, 61 up-to-date

# Spark 3.4
./gradlew -DsparkVersions=3.4 -DscalaVersion=2.12 -DflinkVersions= -DkafkaVersions= :iceberg-spark:iceberg-spark-3.4_2.12:check -Pquick=true -x javadoc

BUILD SUCCESSFUL in 21m 32s
45 actionable tasks: 7 executed, 1 from cache, 37 up-to-date

./gradlew -DsparkVersions=3.4 -DscalaVersion=2.12 -DflinkVersions= -DkafkaVersions= :iceberg-spark:iceberg-spark-extensions-3.4_2.12:check -Pquick=true -x javadoc

BUILD SUCCESSFUL in 22m 6s
51 actionable tasks: 5 executed, 2 from cache, 44 up-to-date

./gradlew -DsparkVersions=3.4 -DscalaVersion=2.12 -DflinkVersions= -DkafkaVersions= :iceberg-spark:iceberg-spark-runtime-3.4_2.12:check -Pquick=true -x javadoc

BUILD SUCCESSFUL in 18s
64 actionable tasks: 4 executed, 1 from cache, 59 up-to-date

[codecov-commenter]

Codecov Report

All modified and coverable lines are covered by tests :white_check_mark:

Project coverage is 58.79%. Comparing base (f09f8af) to head (738bebb). Report is 276 commits behind head on main.

Additional details and impacted files

@@             Coverage Diff              @@
##               main    #1715      +/-   ##
============================================
+ Coverage     56.12%   58.79%   +2.66%     
- Complexity      976     1140     +164     
============================================
  Files           119      130      +11     
  Lines         11743    12836    +1093     
  Branches       2251     2415     +164     
============================================
+ Hits           6591     7547     +956     
- Misses         4012     4065      +53     
- Partials       1140     1224      +84     

:umbrella: View full report in Codecov by Sentry.
:loudspeaker: Have feedback on the report? Share it here.

:rocket: New features to boost your workflow:

• :snowflake: Test Analytics: Detect flaky tests, report on failures, and find test suite problems.