spark-avro
spark-avro copied to clipboard
Support for LogicalTypes like Decimal type
Avro doesn't support very big numbers directly. It supports it through logicalTypes where you can specify value as string type but send the actual data type of the field as property logicalType. Following is the example of a decimal type
{"type": "record",
"name": "test",
"fields" : [
{"name": "a","type": "string",
"logicalType": "decimal",
"precision": 38,
"scale": 2
},
{"name": "b", "type": "string"}
]}
I have raised a pull request to support for reading such datatype. For now I have added only decimal type, but we can add more logicalTypes. In dataframe it actually appears as decimal(38,2) type.
Following is the pull request https://github.com/databricks/spark-avro/pull/121