spark-dynamodb icon indicating copy to clipboard operation
spark-dynamodb copied to clipboard

ItemConverter to support DynamoDB List

Open pmminov opened this issue 7 years ago • 1 comments

https://github.com/traviscrawford/spark-dynamodb/blob/master/src/main/scala/com/github/traviscrawford/spark/dynamodb/ItemConverter.scala#L17

When hitting an unsupported type it will immediately throw an Exception. By your comments there I assume this is expected :)

17/08/01 12:41:06 ERROR DynamoDBRelation: Failed converting item to row: {"id":"xxxx","sections":["AAA","BBB","CCC"]}
scala.MatchError: ArrayType(StringType,true) (of class org.apache.spark.sql.types.ArrayType)
	at com.github.traviscrawford.spark.dynamodb.ItemConverter$$anonfun$1.apply(ItemConverter.scala:30)

I'm quite new to Scala but the following works for my current needs:

case _ =>
          field.dataType match {
            case ArrayType(StringType,true) => jsonFieldValue.extract[List[String]]
            case IntegerType => jsonFieldValue.extract[Int]
            case LongType => jsonFieldValue.extract[Long]
            case DoubleType => jsonFieldValue.extract[Double]
            case StringType => jsonFieldValue.extract[String]
          }

I wonder if you are planning to update the logic with something more robust

pmminov avatar Aug 01 '17 02:08 pmminov

Agreed that ItemConverter is not fully featured. It was enough for my needs at the time and this has never been revisited. I just added support for simple arrays in branch https://github.com/traviscrawford/spark-dynamodb/compare/travis/improve-item-converter . Can you try that branch on your table and test that it works for you? If it does I'll merge to master and publish a new release.

traviscrawford avatar Aug 02 '17 19:08 traviscrawford