spark-avro
spark-avro copied to clipboard
Error converting MapType to avro
I currently have a map of datapoints that I am trying to write. The schema for this field is as follows:
MapType(LongType,DoubleType,false)
Since the map is keyed on longs and not strings, I get an error during schema conversion to avro:
com.databricks.spark.avro.SchemaConverters$IncompatibleSchemaException: Unexpected type MapType(LongType,DoubleType,false).
at com.databricks.spark.avro.SchemaConverters$.com$databricks$spark$avro$SchemaConverters$$convertFieldTypeToAvro(SchemaConverters.scala:391)
at com.databricks.spark.avro.SchemaConverters$$anonfun$convertStructToAvro$1.apply(SchemaConverters.scala:122)
at com.databricks.spark.avro.SchemaConverters$$anonfun$convertStructToAvro$1.apply(SchemaConverters.scala:118)
at scala.collection.IndexedSeqOptimized$class.foreach(IndexedSeqOptimized.scala:33)
at scala.collection.mutable.ArrayOps$ofRef.foreach(ArrayOps.scala:186)
at com.databricks.spark.avro.SchemaConverters$.convertStructToAvro(SchemaConverters.scala:118)
at com.databricks.spark.avro.DefaultSource.prepareWrite(DefaultSource.scala:116)
at org.apache.spark.sql.execution.datasources.FileFormatWriter$.write(FileFormatWriter.scala:142)
Are maps such as the above supported?
I would be grateful for any input on this issue.