DataflowTemplates
DataflowTemplates copied to clipboard
GoogleCloudPlatform
[Bug]: Spanner staging tests are breaking for ImportPipelineIT and ExportPipelineIT
Related Template(s)
Cloud_Spanner_to_GCS_Avro, GCS_Avro_to_Cloud_Spanner
Template Version
HEAD
What happened?
The ImportPipelineIT and ExportPipelineIT tests have been failing for Spanner Staging cases. There were a series of errors like the following:
Caused by: java.lang.IllegalStateException: Column 1 is not of correct type: expected BOOL but was STRING
com.google.common.base.Preconditions.checkState(Preconditions.java:857)
com.google.cloud.spanner.AbstractStructReader.checkNonNullOfType(AbstractStructReader.java:616)
com.google.cloud.spanner.AbstractStructReader.getBoolean(AbstractStructReader.java:155)
com.google.cloud.teleport.spanner.ddl.InformationSchemaScanner.listPlacements(InformationSchemaScanner.java:1450)
com.google.cloud.teleport.spanner.ddl.InformationSchemaScanner.scan(InformationSchemaScanner.java:105)
com.google.cloud.teleport.spanner.ReadInformationSchema$ReadInformationSchemaFn.processElement(ReadInformationSchema.java:96)
that seem to have been resolved as of #1792, however the below log output shows the current output.
Since this appears to only occur on staging spanner tests, it may be due to an incompatible change on the staged spanner instance, in which case disabling these tests will be the temporary fix until a patch is released, or the ITs are updated.
Relevant log output
org.apache.beam.it.gcp.spanner.SpannerResourceManagerException: Failed to execute statement.
at org.apache.beam.it.gcp.spanner.SpannerResourceManager.executeDdlStatements(SpannerResourceManager.java:307)
at org.apache.beam.it.gcp.spanner.SpannerResourceManager.executeDdlStatement(SpannerResourceManager.java:283)
at com.google.cloud.teleport.spanner.ExportPipelineIT.testSpannerToGCSAvroBase(ExportPipelineIT.java:210)
at com.google.cloud.teleport.spanner.ExportPipelineIT.testSpannerToGCSAvroStaging(ExportPipelineIT.java:168)
at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.base/java.lang.reflect.Method.invoke(Method.java:566)
at org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:59)
at org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12)
at org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:56)
at org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17)
at org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:26)
at org.junit.internal.runners.statements.RunAfters.evaluate(RunAfters.java:27)
at org.junit.rules.TestWatcher$1.evaluate(TestWatcher.java:61)
at org.junit.runners.ParentRunner$3.evaluate(ParentRunner.java:306)
at org.junit.runners.BlockJUnit4ClassRunner$1.evaluate(BlockJUnit4ClassRunner.java:100)
at org.junit.runners.ParentRunner.runLeaf(ParentRunner.java:366)
at org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:103)
at org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:63)
at org.junit.runners.ParentRunner$4.run(ParentRunner.java:331)
at org.apache.maven.surefire.junitcore.pc.Scheduler$1.run(Scheduler.java:410)
at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515)
at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264)
at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
at java.base/java.lang.Thread.run(Thread.java:829)
Caused by: java.util.concurrent.ExecutionException: com.google.cloud.spanner.SpannerException: INVALID_ARGUMENT: Operation with name "projects/cloud-teleport-testing/instances/teleport/databases/testspa_20240819_204705_ruvcft/operations/r58f2a880_153b_470d_a95a_66c0b992ae2a" failed with status = GrpcStatusCode{transportCode=INVALID_ARGUMENT} and message = Token key MyTokens in search index testSpannerToGCSAvroStaging_SearchIndex does not support order.
at com.google.common.util.concurrent.AbstractFuture.getDoneValue(AbstractFuture.java:594)
at com.google.common.util.concurrent.AbstractFuture.get(AbstractFuture.java:573)
at com.google.common.util.concurrent.FluentFuture$TrustedFuture.get(FluentFuture.java:91)
at com.google.common.util.concurrent.ForwardingFuture.get(ForwardingFuture.java:67)
at com.google.api.gax.longrunning.OperationFutureImpl.get(OperationFutureImpl.java:125)
at org.apache.beam.it.gcp.spanner.SpannerResourceManager.executeDdlStatements(SpannerResourceManager.java:304)
... 26 more
Caused by: com.google.cloud.spanner.SpannerException: INVALID_ARGUMENT: Operation with name "projects/cloud-teleport-testing/instances/teleport/databases/testspa_20240819_204705_ruvcft/operations/r58f2a880_153b_470d_a95a_66c0b992ae2a" failed with status = GrpcStatusCode{transportCode=INVALID_ARGUMENT} and message = Token key MyTokens in search index testSpannerToGCSAvroStaging_SearchIndex does not support order.
at com.google.cloud.spanner.SpannerExceptionFactory.newSpannerExceptionPreformatted(SpannerExceptionFactory.java:291)
at com.google.cloud.spanner.SpannerExceptionFactory.fromApiException(SpannerExceptionFactory.java:311)
at com.google.cloud.spanner.SpannerExceptionFactory.newSpannerException(SpannerExceptionFactory.java:174)
at com.google.cloud.spanner.SpannerExceptionFactory.newSpannerException(SpannerExceptionFactory.java:110)
at com.google.cloud.spanner.DatabaseAdminClientImpl.lambda$updateDatabaseDdl$11(DatabaseAdminClientImpl.java:470)
at com.google.api.core.ApiFutures$ApiFunctionToGuavaFunction.apply(ApiFutures.java:396)
at com.google.common.util.concurrent.AbstractCatchingFuture$CatchingFuture.doFallback(AbstractCatchingFuture.java:237)
at com.google.common.util.concurrent.AbstractCatchingFuture$CatchingFuture.doFallback(AbstractCatchingFuture.java:225)
at com.google.common.util.concurrent.AbstractCatchingFuture.run(AbstractCatchingFuture.java:135)
at com.google.common.util.concurrent.DirectExecutor.execute(DirectExecutor.java:31)
at com.google.common.util.concurrent.AbstractFuture.executeListener(AbstractFuture.java:1298)
at com.google.common.util.concurrent.AbstractFuture.complete(AbstractFuture.java:1059)
at com.google.common.util.concurrent.AbstractFuture.setException(AbstractFuture.java:809)
at com.google.common.util.concurrent.AbstractTransformFuture.run(AbstractTransformFuture.java:129)
at com.google.common.util.concurrent.DirectExecutor.execute(DirectExecutor.java:31)
at com.google.common.util.concurrent.AbstractFuture.executeListener(AbstractFuture.java:1298)
at com.google.common.util.concurrent.AbstractFuture.complete(AbstractFuture.java:1059)
at com.google.common.util.concurrent.AbstractFuture.set(AbstractFuture.java:784)
at com.google.api.gax.retrying.BasicRetryingFuture.handleAttempt(BasicRetryingFuture.java:203)
at com.google.api.gax.retrying.CallbackChainRetryingFuture$AttemptCompletionListener.handle(CallbackChainRetryingFuture.java:135)
at com.google.api.gax.retrying.CallbackChainRetryingFuture$AttemptCompletionListener.run(CallbackChainRetryingFuture.java:115)
at com.google.common.util.concurrent.DirectExecutor.execute(DirectExecutor.java:31)
at com.google.common.util.concurrent.AbstractFuture.executeListener(AbstractFuture.java:1298)
at com.google.common.util.concurrent.AbstractFuture.complete(AbstractFuture.java:1059)
at com.google.common.util.concurrent.AbstractFuture.set(AbstractFuture.java:784)
at com.google.common.util.concurrent.AbstractTransformFuture$TransformFuture.setResult(AbstractTransformFuture.java:259)
at com.google.common.util.concurrent.AbstractTransformFuture.run(AbstractTransformFuture.java:171)
at com.google.common.util.concurrent.DirectExecutor.execute(DirectExecutor.java:31)
at com.google.common.util.concurrent.AbstractFuture.executeListener(AbstractFuture.java:1298)
at com.google.common.util.concurrent.AbstractFuture.complete(AbstractFuture.java:1059)
at com.google.common.util.concurrent.AbstractFuture.set(AbstractFuture.java:784)
at com.google.api.gax.retrying.BasicRetryingFuture.handleAttempt(BasicRetryingFuture.java:203)
at com.google.api.gax.retrying.CallbackChainRetryingFuture$AttemptCompletionListener.handle(CallbackChainRetryingFuture.java:135)
at com.google.api.gax.retrying.CallbackChainRetryingFuture$AttemptCompletionListener.run(CallbackChainRetryingFuture.java:115)
at com.google.common.util.concurrent.DirectExecutor.execute(DirectExecutor.java:31)
at com.google.common.util.concurrent.AbstractFuture.executeListener(AbstractFuture.java:1298)
at com.google.common.util.concurrent.AbstractFuture.complete(AbstractFuture.java:1059)
at com.google.common.util.concurrent.AbstractFuture.set(AbstractFuture.java:784)
at com.google.api.core.AbstractApiFuture$InternalSettableFuture.set(AbstractApiFuture.java:87)
at com.google.api.core.AbstractApiFuture.set(AbstractApiFuture.java:70)
at com.google.api.gax.grpc.GrpcExceptionCallable$ExceptionTransformingFuture.onSuccess(GrpcExceptionCallable.java:88)
at com.google.api.core.ApiFutures$1.onSuccess(ApiFutures.java:89)
at com.google.common.util.concurrent.Futures$CallbackListener.run(Futures.java:1137)
at com.google.common.util.concurrent.DirectExecutor.execute(DirectExecutor.java:31)
at com.google.common.util.concurrent.AbstractFuture.executeListener(AbstractFuture.java:1298)
at com.google.common.util.concurrent.AbstractFuture.complete(AbstractFuture.java:1059)
at com.google.common.util.concurrent.AbstractFuture.set(AbstractFuture.java:784)
at io.grpc.stub.ClientCalls$GrpcFuture.set(ClientCalls.java:563)
at io.grpc.stub.ClientCalls$UnaryStreamToFuture.onClose(ClientCalls.java:536)
at io.grpc.PartialForwardingClientCallListener.onClose(PartialForwardingClientCallListener.java:39)
at io.grpc.ForwardingClientCallListener.onClose(ForwardingClientCallListener.java:23)
at io.grpc.ForwardingClientCallListener$SimpleForwardingClientCallListener.onClose(ForwardingClientCallListener.java:40)
at com.google.api.gax.grpc.ChannelPool$ReleasingClientCall$1.onClose(ChannelPool.java:570)
at io.grpc.PartialForwardingClientCallListener.onClose(PartialForwardingClientCallListener.java:39)
at io.grpc.ForwardingClientCallListener.onClose(ForwardingClientCallListener.java:23)
at io.grpc.ForwardingClientCallListener$SimpleForwardingClientCallListener.onClose(ForwardingClientCallListener.java:40)
at io.grpc.PartialForwardingClientCallListener.onClose(PartialForwardingClientCallListener.java:39)
at io.grpc.ForwardingClientCallListener.onClose(ForwardingClientCallListener.java:23)
at io.grpc.ForwardingClientCallListener$SimpleForwardingClientCallListener.onClose(ForwardingClientCallListener.java:40)
at com.google.cloud.spanner.spi.v1.SpannerErrorInterceptor$1$1.onClose(SpannerErrorInterceptor.java:100)
at io.grpc.PartialForwardingClientCallListener.onClose(PartialForwardingClientCallListener.java:39)
at io.grpc.ForwardingClientCallListener.onClose(ForwardingClientCallListener.java:23)
at io.grpc.ForwardingClientCallListener$SimpleForwardingClientCallListener.onClose(ForwardingClientCallListener.java:40)
at io.grpc.census.CensusStatsModule$StatsClientInterceptor$1$1.onClose(CensusStatsModule.java:814)
at io.grpc.PartialForwardingClientCallListener.onClose(PartialForwardingClientCallListener.java:39)
at io.grpc.ForwardingClientCallListener.onClose(ForwardingClientCallListener.java:23)
at io.grpc.ForwardingClientCallListener$SimpleForwardingClientCallListener.onClose(ForwardingClientCallListener.java:40)
at io.grpc.census.CensusTracingModule$TracingClientInterceptor$1$1.onClose(CensusTracingModule.java:494)
at io.grpc.internal.ClientCallImpl.closeObserver(ClientCallImpl.java:574)
at io.grpc.internal.ClientCallImpl.access$300(ClientCallImpl.java:72)
at io.grpc.internal.ClientCallImpl$ClientStreamListenerImpl$1StreamClosed.runInternal(ClientCallImpl.java:742)
at io.grpc.internal.ClientCallImpl$ClientStreamListenerImpl$1StreamClosed.runInContext(ClientCallImpl.java:723)
at io.grpc.internal.ContextRunnable.run(ContextRunnable.java:37)
at io.grpc.internal.SerializingExecutor.run(SerializingExecutor.java:133)
... 3 more
https://github.com/GoogleCloudPlatform/DataflowTemplates/pull/1808 disables this for now.
The original issue should be fixed in https://github.com/GoogleCloudPlatform/DataflowTemplates/pull/1792
I believe the issue was due to the rollout of a change that introduced new information schema views for Postgres dialect databases in our staging environment, but with a different type for the is_default column.
Should be fixed by the following logic: https://github.com/GoogleCloudPlatform/DataflowTemplates/blob/e58646a1d11dfe5036b0719c5add1bd7f1d8a459/v1/src/main/java/com/google/cloud/teleport/spanner/ddl/InformationSchemaScanner.java#L1472-L1484
The second issue seems to be related to token indexes.
Hi @sgorse123 @VardhanThigle, testSpannerToGCSAvroStaging is failing again on main branch, see https://github.com/GoogleCloudPlatform/DataflowTemplates/pull/1867#issuecomment-2350767191 could you please take a look?
