iceberg
iceberg copied to clipboard
Spark 3.5: Update Spark to use planned Avro reads
Moving to the planned reader adds default value support. This is the same basic change as in #9366 and #11108.
Here are the benchmark results:
## main
Benchmark Mode Cnt Score Error Units
IcebergSourceFlatAvroDataReadBenchmark.readIceberg ss 5 8.883 ± 0.321 s/op
IcebergSourceFlatAvroDataReadBenchmark.readWithProjectionIceberg ss 5 7.173 ± 0.254 s/op
## this PR
Benchmark Mode Cnt Score Error Units
IcebergSourceFlatAvroDataReadBenchmark.readIceberg ss 5 3.718 ± 0.177 s/op
IcebergSourceFlatAvroDataReadBenchmark.readWithProjectionIceberg ss 5 3.777 ± 0.776 s/op
## main
Benchmark Mode Cnt Score Error Units
IcebergSourceNestedAvroDataReadBenchmark.readIceberg ss 5 2.616 ± 0.110 s/op
IcebergSourceNestedAvroDataReadBenchmark.readWithProjectionIceberg ss 5 2.365 ± 0.037 s/op
## this PR
Benchmark Mode Cnt Score Error Units
IcebergSourceNestedAvroDataReadBenchmark.readIceberg ss 5 1.969 ± 0.079 s/op
IcebergSourceNestedAvroDataReadBenchmark.readWithProjectionIceberg ss 5 1.637 ± 0.045 s/op