kafka-connect-fs
kafka-connect-fs copied to clipboard
Parquet dependency should be updated to 1.12+ to support the INT96 compatibility flag
The current parquet dependency is for the 1.11.x version, which doesn't support INT96 types. Older datasets generated by Spark applications may have used this type. Parquet 1.12+ has added support for this. Updating to 1.12.3 allows the user to add "policy.fs.parquet.avro.readInt96AsFixed": "true",
for example.
See https://spark.apache.org/docs/latest/sql-data-sources-parquet.html#configuration, https://issues.apache.org/jira/browse/PARQUET-1928, https://github.com/apache/parquet-mr/blob/master/CHANGES.md.