spark-dynamodb
spark-dynamodb copied to clipboard
ItemConverter to support DynamoDB List
https://github.com/traviscrawford/spark-dynamodb/blob/master/src/main/scala/com/github/traviscrawford/spark/dynamodb/ItemConverter.scala#L17
When hitting an unsupported type it will immediately throw an Exception. By your comments there I assume this is expected :)
17/08/01 12:41:06 ERROR DynamoDBRelation: Failed converting item to row: {"id":"xxxx","sections":["AAA","BBB","CCC"]}
scala.MatchError: ArrayType(StringType,true) (of class org.apache.spark.sql.types.ArrayType)
at com.github.traviscrawford.spark.dynamodb.ItemConverter$$anonfun$1.apply(ItemConverter.scala:30)
I'm quite new to Scala but the following works for my current needs:
case _ =>
field.dataType match {
case ArrayType(StringType,true) => jsonFieldValue.extract[List[String]]
case IntegerType => jsonFieldValue.extract[Int]
case LongType => jsonFieldValue.extract[Long]
case DoubleType => jsonFieldValue.extract[Double]
case StringType => jsonFieldValue.extract[String]
}
I wonder if you are planning to update the logic with something more robust
Agreed that ItemConverter
is not fully featured. It was enough for my needs at the time and this has never been revisited. I just added support for simple arrays in branch https://github.com/traviscrawford/spark-dynamodb/compare/travis/improve-item-converter . Can you try that branch on your table and test that it works for you? If it does I'll merge to master and publish a new release.