iceberg icon indicating copy to clipboard operation
iceberg copied to clipboard

Spark 3.5: Update Spark to use planned Avro reads

Open rdblue opened this issue 1 year ago • 1 comments

Moving to the planned reader adds default value support. This is the same basic change as in #9366 and #11108.

rdblue avatar Oct 10 '24 23:10 rdblue

Here are the benchmark results:

## main
Benchmark                                                         Mode  Cnt  Score   Error  Units
IcebergSourceFlatAvroDataReadBenchmark.readIceberg                  ss    5  8.883 ± 0.321   s/op
IcebergSourceFlatAvroDataReadBenchmark.readWithProjectionIceberg    ss    5  7.173 ± 0.254   s/op

## this PR
Benchmark                                                         Mode  Cnt  Score   Error  Units
IcebergSourceFlatAvroDataReadBenchmark.readIceberg                  ss    5  3.718 ± 0.177   s/op
IcebergSourceFlatAvroDataReadBenchmark.readWithProjectionIceberg    ss    5  3.777 ± 0.776   s/op

## main
Benchmark                                                           Mode  Cnt  Score   Error  Units
IcebergSourceNestedAvroDataReadBenchmark.readIceberg                  ss    5  2.616 ± 0.110   s/op
IcebergSourceNestedAvroDataReadBenchmark.readWithProjectionIceberg    ss    5  2.365 ± 0.037   s/op

## this PR
Benchmark                                                           Mode  Cnt  Score   Error  Units
IcebergSourceNestedAvroDataReadBenchmark.readIceberg                  ss    5  1.969 ± 0.079   s/op
IcebergSourceNestedAvroDataReadBenchmark.readWithProjectionIceberg    ss    5  1.637 ± 0.045   s/op

rdblue avatar Oct 11 '24 20:10 rdblue