spark-avro Logical datatype support in spark-avro

Hi ,

Is there a functionality here which can deserialize data to support reflect data apis.

For instance if my type schema for field is

   {"name": "a", "type": "string" , "java-class": "java.math.BigInteger"},

I want the return type to be of BigInteger type .

Mar 02 '16 12:03 rahulmittal91

Nope, it looks like we don't support this currently. I'd be happy to review a PR to add support for this, though.

Nov 21 '16 23:11 JoshRosen

We have a pull request https://github.com/databricks/spark-avro/pull/121 which supports LogicalData types in avro. Currently we have added support for BigDecimals with scale and precision , timestamps and date . This code is also running in our production from past 8 months without any issues .

Nov 28 '16 11:11 rahulmittal91

To clarify, it looks like #121 is concerned with logicalType, whereas it sounds like this issue is discussing ReflectData / the java-class attribute. These are related but slightly orthogonal features, right?

Nov 28 '16 19:11 JoshRosen

Yes these are exaclty not the same features . We came up with a design where logical types were more suitable . We wanted to represent db types(Bigdecimals, dates and timestamps) in avro types, serialize and deserialize them (This question was posted before https://github.com/databricks/spark-avro/pull/121 was raised).

Nov 29 '16 05:11 rahulmittal91

@JoshRosen, me and Rahul and the contributors to this PR. This issues was raised as a question if there was such support in spark-avro. Since there was no response, so we implemented this feature. Initially we thought of doing it through ReflectData, but logical types suited best to our use case.

@rahulmittal91 Can you change the subject of this issue?

Nov 29 '16 09:11 cpbhagtani