[SPARK-39927][BUILD] Upgrade to Avro 1.11.1

[iemejia]

What changes were proposed in this pull request?

Update the Avro version to 1.11.1

Why are the changes needed?

To stay up to date with upstream

Does this PR introduce any user-facing change?

No

How was this patch tested?

Unit tests

[HyukjinKwon]

Seems like the test failure looks related:

AvroV1Suite.support user provided avro schema for writing non-nullable enum type
java.lang.AssertionError: assertion failed: Exception tree doesn't contain the expected exception of type org.apache.avro.AvroTypeException with message: Not an enum: null
org.apache.avro.AvroTypeException: value null is not a SuitEnumType
	at org.apache.avro.generic.GenericDatumWriter.writeEnum(GenericDatumWriter.java:269)
	at org.apache.avro.specific.SpecificDatumWriter.writeEnum(SpecificDatumWriter.java:66)
	at org.apache.avro.generic.GenericDatumWriter.writeWithoutConversion(GenericDatumWriter.java:148)
	at org.apache.avro.generic.GenericDatumWriter.write(GenericDatumWriter.java:95)
	at org.apache.avro.reflect.ReflectDatumWriter.write(ReflectDatumWriter.java:158)
	at org.apache.avro.generic.GenericDatumWriter.writeField(GenericDatumWriter.java:245)
	at org.apache.avro.specific.SpecificDatumWriter.writeField(SpecificDatumWriter.java:117)
	at org.apache.avro.reflect.ReflectDatumWriter.writeField(ReflectDatumWriter.java:184)
	at org.apache.avro.generic.GenericDatumWriter.writeRecord(GenericDatumWriter.java:234)
	at org.apache.avro.specific.SpecificDatumWriter.writeRecord(SpecificDatumWriter.java:92)
	at org.apache.avro.generic.GenericDatumWriter.writeWithoutConversion(GenericDatumWriter.java:145)
	at org.apache.avro.generic.GenericDatumWriter.write(GenericDatumWriter.java:95)
	at org.apache.avro.reflect.ReflectDatumWriter.write(ReflectDatumWriter.java:158)
	at org.apache.avro.generic.GenericDatumWriter.write(GenericDatumWriter.java:82)
	at org.apache.avro.file.DataFileWriter.append(DataFileWriter.java:314)
	at org.apache.spark.sql.avro.SparkAvroKeyRecordWriter.write(SparkAvroKeyOutputFormat.java:86)
	at org.apache.spark.sql.avro.SparkAvroKeyRecordWriter.write(SparkAvroKeyOutputFormat.java:63)
	at org.apache.spark.sql.avro.AvroOutputWriter.write(AvroOutputWriter.scala:86)
	at org.apache.spark.sql.execution.datasources.SingleDirectoryDataWriter.write(FileFormatDataWriter.scala:175)
	at org.apache.spark.sql.execution.datasources.FileFormatDataWriter.writeWithMetrics(FileFormatDataWriter.scala:85)
	at org.apache.spark.sql.execution.datasources.FileFormatDataWriter.writeWithIterator(FileFormatDataWriter.scala:92)
	at org.apache.spark.sql.execution.datasources.FileFormatWriter$.$anonfun$executeTask$1(FileFormatWriter.scala:323)
	at org.apache.spark.util.Utils$.tryWithSafeFinallyAndFailureCallbacks(Utils.scala:1524)

[xkrogen]

Is this an official release? The documentation links you updated (such as this one) give a 404 error and I don't see Avro 1.11.1 listed on the Avro Releases page.

[AmplabJenkins]

Can one of the admins verify this patch?

[dongjoon-hyun]

Gentle ping, @iemejia .

[martin-g]

I don't see Avro 1.11.1 listed on the Avro Releases page.

Also, we work on replacing the Avro website with a new one. Probably it will be done later this week!

[LuciferYang]

seems support user provided non-nullable avro schema for nullable catalyst schema without any null record *** FAILED *** (181 milliseconds) still failed

[dongjoon-hyun]

Yes, it does. @LuciferYang .

To @iemejia , as I mentioned here, please take a look at the test case. Otherwise, Avro 1.11.1 cannot pass the CIs.

[dongjoon-hyun]

Merged to master for Apache Spark 3.4.0.

[dongjoon-hyun]

Thank you, @iemejia , @HyukjinKwon , @xkrogen , @martin-g , @LuciferYang .

[iemejia]

Thanks as usual @dongjoon-hyun !