spark-avro icon indicating copy to clipboard operation
spark-avro copied to clipboard

Error converting MapType to avro

Open davecromberge opened this issue 7 years ago • 0 comments

I currently have a map of datapoints that I am trying to write. The schema for this field is as follows: MapType(LongType,DoubleType,false)

Since the map is keyed on longs and not strings, I get an error during schema conversion to avro:

com.databricks.spark.avro.SchemaConverters$IncompatibleSchemaException: Unexpected type MapType(LongType,DoubleType,false).
  at com.databricks.spark.avro.SchemaConverters$.com$databricks$spark$avro$SchemaConverters$$convertFieldTypeToAvro(SchemaConverters.scala:391)
  at com.databricks.spark.avro.SchemaConverters$$anonfun$convertStructToAvro$1.apply(SchemaConverters.scala:122)
  at com.databricks.spark.avro.SchemaConverters$$anonfun$convertStructToAvro$1.apply(SchemaConverters.scala:118)
  at scala.collection.IndexedSeqOptimized$class.foreach(IndexedSeqOptimized.scala:33)
  at scala.collection.mutable.ArrayOps$ofRef.foreach(ArrayOps.scala:186)
  at com.databricks.spark.avro.SchemaConverters$.convertStructToAvro(SchemaConverters.scala:118)
  at com.databricks.spark.avro.DefaultSource.prepareWrite(DefaultSource.scala:116)
  at org.apache.spark.sql.execution.datasources.FileFormatWriter$.write(FileFormatWriter.scala:142)

Are maps such as the above supported?

I would be grateful for any input on this issue.

davecromberge avatar Oct 04 '17 10:10 davecromberge