scio icon indicating copy to clipboard operation
scio copied to clipboard

Issue with encoding Map with complex types

Open klyashko opened this issue 1 year ago • 0 comments

Hello.

I believe there is a bug in SchemaMaterializer if type of key or value isn't a primitive.

getCollectionElementType returns field type only for lists for map getMapKeyType and getMapValueType needs to be used.

So as a result I have a NullPointerException if I have a map with complex types.

A unit test to reproduce the issue:

  case class K(someIntField: Int)

  case class V(andLong: Long)

"com.spotify.scio.schemas.SchemaMaterializer" should "throw NPE when it's serializing map with objects" in {
    val objectMap = Map(K(1) -> V(2)) 
    val (_, objectMapToRow, _) = SchemaMaterializer.materialize(Schema[Map[K, V]])

    assertThrows[NullPointerException] {
      println(objectMapToRow(objectMap))
    }

    val map = Map("s" -> "e")
    val (_, toRow, _) = SchemaMaterializer.materialize(Schema[Map[String, String]])

    // Working fine for primitives ¯\_(ツ)_/¯
    noException should be thrownBy toRow(map)
  }

klyashko avatar May 18 '23 12:05 klyashko