spark-avro icon indicating copy to clipboard operation
spark-avro copied to clipboard

Support for LogicalTypes like Decimal type

Open cpbhagtani opened this issue 8 years ago • 1 comments

Avro doesn't support very big numbers directly. It supports it through logicalTypes where you can specify value as string type but send the actual data type of the field as property logicalType. Following is the example of a decimal type

{"type": "record", 
 "name": "test",
 "fields" : [
 {"name": "a","type": "string",
  "logicalType": "decimal",
  "precision": 38,
  "scale": 2
},
 {"name": "b", "type": "string"}
]}

I have raised a pull request to support for reading such datatype. For now I have added only decimal type, but we can add more logicalTypes. In dataframe it actually appears as decimal(38,2) type.

cpbhagtani avatar Mar 09 '16 06:03 cpbhagtani

Following is the pull request https://github.com/databricks/spark-avro/pull/121

cpbhagtani avatar Mar 09 '16 06:03 cpbhagtani