scio
scio copied to clipboard
Spanner read and write example
A spanner read sample can be made with (works fine)
sc.spannerQuery(config, "select id1, firstname, lastname from users limit 10", false, false)
.map(r => (r.getString(0), r.getString(1), r.getString(2)))
.map[Row](Row.tupled)
.map(_.toString)
.saveAsTextFile(output, 1)
Spanner write gives an unsupported exception
case class Row(name: String, firstname: String, lastname: String)
sc
.parallelize(Seq(Row("a1", "xxx", "yyy"), Row("a2", "xxx", "yyy")))
.map(r =>
Mutation
.newInsertOrUpdateBuilder("users")
.set("id1")
.to(r.name)
.set("firstname")
.to(r.firstname)
.set("lastname")
.to(r.lastname)
.build()
)
.write(SpannerWrite(config))(SpannerWrite.WriteParam())
Detailed stacktrace in write sample
[error] Exception in thread "main" org.apache.beam.sdk.Pipeline$PipelineExecutionException: com.esotericsoftware.kryo.KryoException: java.lang.UnsupportedOperationException
[error] Serialization trace:
[error] columns (com.google.cloud.spanner.Mutation)
[error] mutations (org.apache.beam.sdk.io.gcp.spanner.MutationGroup)
[error] at org.apache.beam.runners.direct.DirectRunner$DirectPipelineResult.waitUntilFinish(DirectRunner.java:374)
[error] at org.apache.beam.runners.direct.DirectRunner$DirectPipelineResult.waitUntilFinish(DirectRunner.java:342)
[error] at org.apache.beam.runners.direct.DirectRunner.run(DirectRunner.java:218)
[error] at org.apache.beam.runners.direct.DirectRunner.run(DirectRunner.java:67)
[error] at org.apache.beam.sdk.Pipeline.run(Pipeline.java:323)
[error] at org.apache.beam.sdk.Pipeline.run(Pipeline.java:309)
[error] at com.spotify.scio.ScioContext.execute(ScioContext.scala:587)
[error] at com.spotify.scio.ScioContext$$anonfun$run$1.apply(ScioContext.scala:574)
[error] at com.spotify.scio.ScioContext$$anonfun$run$1.apply(ScioContext.scala:562)
[error] at com.spotify.scio.ScioContext.requireNotClosed(ScioContext.scala:653)
[error] at com.spotify.scio.ScioContext.run(ScioContext.scala:562)
[error] at com.trickster.SpannerRead$.main(SpannerRead.scala:106)
[error] at com.trickster.SpannerRead.main(SpannerRead.scala)