scio icon indicating copy to clipboard operation
scio copied to clipboard

Spanner read and write example

Open trickster opened this issue 1 year ago • 0 comments

A spanner read sample can be made with (works fine)

sc.spannerQuery(config, "select id1, firstname, lastname from users limit 10", false, false)
  .map(r => (r.getString(0), r.getString(1), r.getString(2)))
  .map[Row](Row.tupled)
  .map(_.toString)
  .saveAsTextFile(output, 1)

Spanner write gives an unsupported exception

    case class Row(name: String, firstname: String, lastname: String)
    sc
      .parallelize(Seq(Row("a1", "xxx", "yyy"), Row("a2", "xxx", "yyy")))
      .map(r =>
        Mutation
          .newInsertOrUpdateBuilder("users")
          .set("id1")
          .to(r.name)
          .set("firstname")
          .to(r.firstname)
          .set("lastname")
          .to(r.lastname)
          .build()
      )
      .write(SpannerWrite(config))(SpannerWrite.WriteParam())

Detailed stacktrace in write sample

[error] Exception in thread "main" org.apache.beam.sdk.Pipeline$PipelineExecutionException: com.esotericsoftware.kryo.KryoException: java.lang.UnsupportedOperationException
[error] Serialization trace:
[error] columns (com.google.cloud.spanner.Mutation)
[error] mutations (org.apache.beam.sdk.io.gcp.spanner.MutationGroup)
[error] 	at org.apache.beam.runners.direct.DirectRunner$DirectPipelineResult.waitUntilFinish(DirectRunner.java:374)
[error] 	at org.apache.beam.runners.direct.DirectRunner$DirectPipelineResult.waitUntilFinish(DirectRunner.java:342)
[error] 	at org.apache.beam.runners.direct.DirectRunner.run(DirectRunner.java:218)
[error] 	at org.apache.beam.runners.direct.DirectRunner.run(DirectRunner.java:67)
[error] 	at org.apache.beam.sdk.Pipeline.run(Pipeline.java:323)
[error] 	at org.apache.beam.sdk.Pipeline.run(Pipeline.java:309)
[error] 	at com.spotify.scio.ScioContext.execute(ScioContext.scala:587)
[error] 	at com.spotify.scio.ScioContext$$anonfun$run$1.apply(ScioContext.scala:574)
[error] 	at com.spotify.scio.ScioContext$$anonfun$run$1.apply(ScioContext.scala:562)
[error] 	at com.spotify.scio.ScioContext.requireNotClosed(ScioContext.scala:653)
[error] 	at com.spotify.scio.ScioContext.run(ScioContext.scala:562)
[error] 	at com.rakutentraveldio.SpannerRead$.main(SpannerRead.scala:106)
[error] 	at com.rakutentraveldio.SpannerRead.main(SpannerRead.scala)

trickster avatar Dec 06 '22 02:12 trickster