airbyte
airbyte copied to clipboard
Published
20 hours ago •
airbytehq
airbytehq
source-postgres (1.0.30) Cannot convert the column of type TIMESTAMP_ARRAY to requested type java.time.LocalDateTime.
Environment
- Airbyte version: 0.40.14
- OS Version / Instance: ubuntu
- Deployment: kubernetes helm chart
- Source Connector and version: airbyte/source-postgres | 1.0.30
- Destination Connector and version: airbyte/destination-postgres | 0.3.26
- Step where error happened:
- After upgrading source connector to 1.0.30 , some field mapping changed to array
- It works quite good, but one fiel that contains an array of timestamps fails with belows error.
- after removing the table with this field, the sync of the other tables work again
- adding the table to the sync set, it fails again
- was propably introduced with:
- 1.0.27 | 2022-11-28 | 16990 | Handle arrays data types
Current Behavior
Sync fails with: Failure Origin: source, Message: Something went wrong in the connector. See the logs for more details.
Expected Behavior
Sync succeeds
Logs
[44msource[0m > SQLState: 42821, Message: Cannot convert the column of type TIMESTAMP_ARRAY to requested type java.time.LocalDateTime.
2022-12-04 09:55:51 [44msource[0m > Closing database connection pool.
2022-12-04 09:55:51 [44msource[0m > HikariPool-2 - Shutdown initiated...
2022-12-04 09:55:51 [44msource[0m > HikariPool-2 - Shutdown completed.
2022-12-04 09:55:51 [44msource[0m > Closed database connection pool.
2022-12-04 09:55:51 [44msource[0m > Something went wrong in the connector. See the logs for more details.
Stack Trace: java.lang.RuntimeException: org.postgresql.util.PSQLException: Cannot convert the column of type TIMESTAMP_ARRAY to requested type java.time.LocalDateTime.
at io.airbyte.db.jdbc.StreamingJdbcDatabase$1.tryAdvance(StreamingJdbcDatabase.java:105)
at java.base/java.util.Spliterators$1Adapter.hasNext(Spliterators.java:681)
at io.airbyte.commons.util.DefaultAutoCloseableIterator.computeNext(DefaultAutoCloseableIterator.java:38)
at com.google.common.collect.AbstractIterator.tryToComputeNext(AbstractIterator.java:146)
at com.google.common.collect.AbstractIterator.hasNext(AbstractIterator.java:141)
at io.airbyte.commons.util.LazyAutoCloseableIterator.computeNext(LazyAutoCloseableIterator.java:42)
at com.google.common.collect.AbstractIterator.tryToComputeNext(AbstractIterator.java:146)
at com.google.common.collect.AbstractIterator.hasNext(AbstractIterator.java:141)
at com.google.common.collect.TransformedIterator.hasNext(TransformedIterator.java:46)
at io.airbyte.commons.util.DefaultAutoCloseableIterator.computeNext(DefaultAutoCloseableIterator.java:38)
at com.google.common.collect.AbstractIterator.tryToComputeNext(AbstractIterator.java:146)
at com.google.common.collect.AbstractIterator.hasNext(AbstractIterator.java:141)
at com.google.common.collect.TransformedIterator.hasNext(TransformedIterator.java:46)
at io.airbyte.commons.util.DefaultAutoCloseableIterator.computeNext(DefaultAutoCloseableIterator.java:38)
at com.google.common.collect.AbstractIterator.tryToComputeNext(AbstractIterator.java:146)
at com.google.common.collect.AbstractIterator.hasNext(AbstractIterator.java:141)
at io.airbyte.commons.util.CompositeIterator.computeNext(CompositeIterator.java:63)
at com.google.common.collect.AbstractIterator.tryToComputeNext(AbstractIterator.java:146)
at com.google.common.collect.AbstractIterator.hasNext(AbstractIterator.java:141)
at io.airbyte.commons.util.DefaultAutoCloseableIterator.computeNext(DefaultAutoCloseableIterator.java:38)
at com.google.common.collect.AbstractIterator.tryToComputeNext(AbstractIterator.java:146)
at com.google.common.collect.AbstractIterator.hasNext(AbstractIterator.java:141)
at io.airbyte.commons.util.DefaultAutoCloseableIterator.computeNext(DefaultAutoCloseableIterator.java:38)
at com.google.common.collect.AbstractIterator.tryToComputeNext(AbstractIterator.java:146)
at com.google.common.collect.AbstractIterator.hasNext(AbstractIterator.java:141)
at java.base/java.util.Iterator.forEachRemaining(Iterator.java:132)
at io.airbyte.integrations.base.IntegrationRunner.lambda$produceMessages$0(IntegrationRunner.java:188)
at io.airbyte.integrations.base.IntegrationRunner.watchForOrphanThreads(IntegrationRunner.java:238)
at io.airbyte.integrations.base.IntegrationRunner.produceMessages(IntegrationRunner.java:187)
at io.airbyte.integrations.base.IntegrationRunner.runInternal(IntegrationRunner.java:140)
at io.airbyte.integrations.base.IntegrationRunner.run(IntegrationRunner.java:99)
at io.airbyte.integrations.base.adaptive.AdaptiveSourceRunner$Runner.run(AdaptiveSourceRunner.java:86)
at io.airbyte.integrations.source.postgres.PostgresSourceRunner.main(PostgresSourceRunner.java:15)
Caused by: org.postgresql.util.PSQLException: Cannot convert the column of type TIMESTAMP_ARRAY to requested type java.time.LocalDateTime.
at org.postgresql.jdbc.PgResultSet.getLocalDateTime(PgResultSet.java:738)
at org.postgresql.jdbc.PgResultSet.getObject(PgResultSet.java:3760)
at io.airbyte.db.jdbc.AbstractJdbcCompatibleSourceOperations.getObject(AbstractJdbcCompatibleSourceOperations.java:262)
at io.airbyte.integrations.source.postgres.PostgresSourceOperations.putTimestampArray(PostgresSourceOperations.java:268)
at io.airbyte.integrations.source.postgres.PostgresSourceOperations.setJsonField(PostgresSourceOperations.java:210)
at io.airbyte.integrations.source.postgres.PostgresSourceOperations.rowToJson(PostgresSourceOperations.java:95)
at io.airbyte.integrations.source.postgres.PostgresSourceOperations.rowToJson(PostgresSourceOperations.java:57)
at io.airbyte.db.jdbc.StreamingJdbcDatabase$1.tryAdvance(StreamingJdbcDatabase.java:99)
... 32 more
Steps to Reproduce
- Create a postgres source table with a column of type:
timestamp without time zone[] - Synchronize to a destination table of type postgres
- No transformation used, default normalization
- sync fails
Hey @quorak are you still experiencing this issue? Could you attach the full sync log to help with investigation?
Also please try with the latest connector and Airbyte versions to see if it helps
Hey @sh4sh ,
thanks for the reply, yes it is still happening, also with latest
- Source Connector and version: airbyte/source-postgres | 1.0.39
- Destination Connector and version: airbyte/destination-postgres | 0.3.26
Cannot convert the column of type TIMESTAMP_ARRAY to requested type java.time.LocalDateTime.
at io.airbyte.commons.util.CompositeIterator.computeNext(CompositeIterator.java:67)
at com.google.common.collect.AbstractIterator.tryToComputeNext(AbstractIterator.java:146)
at com.google.common.collect.AbstractIterator.hasNext(AbstractIterator.java:141)
at io.airbyte.commons.util.DefaultAutoCloseableIterator.computeNext(DefaultAutoCloseableIterator.java:38)
at com.google.common.collect.AbstractIterator.tryToComputeNext(AbstractIterator.java:146)
at com.google.common.collect.AbstractIterator.hasNext(AbstractIterator.java:141)
at io.airbyte.commons.util.DefaultAutoCloseableIterator.computeNext(DefaultAutoCloseableIterator.java:38)
at com.google.common.collect.AbstractIterator.tryToComputeNext(AbstractIterator.java:146)
at com.google.common.collect.AbstractIterator.hasNext(AbstractIterator.java:141)
at java.base/java.util.Iterator.forEachRemaining(Iterator.java:132)
at io.airbyte.integrations.base.IntegrationRunner.lambda$produceMessages$0(IntegrationRunner.java:187)
at io.airbyte.integrations.base.IntegrationRunner.watchForOrphanThreads(IntegrationRunner.java:237)
at io.airbyte.integrations.base.IntegrationRunner.produceMessages(IntegrationRunner.java:186)
at io.airbyte.integrations.base.IntegrationRunner.runInternal(IntegrationRunner.java:139)
at io.airbyte.integrations.base.IntegrationRunner.run(IntegrationRunner.java:98)
at io.airbyte.integrations.base.adaptive.AdaptiveSourceRunner$Runner.run(AdaptiveSourceRunner.java:86)
at io.airbyte.integrations.source.postgres.PostgresSourceRunner.main(PostgresSourceRunner.java:15)
Caused by: java.lang.RuntimeException: org.postgresql.util.PSQLException: Cannot convert the column of type TIMESTAMP_ARRAY to requested type java.time.LocalDateTime.
at io.airbyte.db.jdbc.StreamingJdbcDatabase.lambda$unsafeQuery$0(StreamingJdbcDatabase.java:75)
at java.base/java.util.stream.AbstractPipeline.close(AbstractPipeline.java:323)
at io.airbyte.commons.concurrency.VoidCallable.call(VoidCallable.java:15)
at io.airbyte.commons.util.DefaultAutoCloseableIterator.close(DefaultAutoCloseableIterator.java:49)
at io.airbyte.commons.util.LazyAutoCloseableIterator.close(LazyAutoCloseableIterator.java:52)
at io.airbyte.commons.concurrency.VoidCallable.call(VoidCallable.java:15)
at io.airbyte.commons.util.DefaultAutoCloseableIterator.close(DefaultAutoCloseableIterator.java:49)
at io.airbyte.commons.concurrency.VoidCallable.call(VoidCallable.java:15)
at io.airbyte.commons.util.DefaultAutoCloseableIterator.close(DefaultAutoCloseableIterator.java:49)
at io.airbyte.commons.util.CompositeIterator.computeNext(CompositeIterator.java:65)
... 16 more
Caused by: org.postgresql.util.PSQLException: Cannot convert the column of type TIMESTAMP_ARRAY to requested type java.time.LocalDateTime.
at org.postgresql.jdbc.PgResultSet.getLocalDateTime(PgResultSet.java:738)
at org.postgresql.jdbc.PgResultSet.getObject(PgResultSet.java:3760)
at io.airbyte.db.jdbc.AbstractJdbcCompatibleSourceOperations.getObject(AbstractJdbcCompatibleSourceOperations.java:244)
at io.airbyte.integrations.source.postgres.PostgresSourceOperations.putTimestampArray(PostgresSourceOperations.java:256)
at io.airbyte.integrations.source.postgres.PostgresSourceOperations.copyToJsonField(PostgresSourceOperations.java:198)
at io.airbyte.integrations.source.postgres.PostgresSourceOperations.rowToJson(PostgresSourceOperations.java:94)
at io.airbyte.integrations.source.postgres.PostgresSourceOperations.rowToJson(PostgresSourceOperations.java:56)
at io.airbyte.db.jdbc.StreamingJdbcDatabase$1.tryAdvance(StreamingJdbcDatabase.java:102)
at java.base/java.util.Spliterators$1Adapter.hasNext(Spliterators.java:681)
at io.airbyte.commons.util.DefaultAutoCloseableIterator.computeNext(DefaultAutoCloseableIterator.java:38)
at com.google.common.collect.AbstractIterator.tryToComputeNext(AbstractIterator.java:146)
at com.google.common.collect.AbstractIterator.hasNext(AbstractIterator.java:141)
at io.airbyte.commons.util.LazyAutoCloseableIterator.computeNext(LazyAutoCloseableIterator.java:42)
at com.google.common.collect.AbstractIterator.tryToComputeNext(AbstractIterator.java:146)
at com.google.common.collect.AbstractIterator.hasNext(AbstractIterator.java:141)
at com.google.common.collect.TransformedIterator.hasNext(TransformedIterator.java:46)
at io.airbyte.commons.util.DefaultAutoCloseableIterator.computeNext(DefaultAutoCloseableIterator.java:38)
at com.google.common.collect.AbstractIterator.tryToComputeNext(AbstractIterator.java:146)
at com.google.common.collect.AbstractIterator.hasNext(AbstractIterator.java:141)
at com.google.common.collect.TransformedIterator.hasNext(TransformedIterator.java:46)
at io.airbyte.commons.util.DefaultAutoCloseableIterator.computeNext(DefaultAutoCloseableIterator.java:38)
at com.google.common.collect.AbstractIterator.tryToComputeNext(AbstractIterator.java:146)
at com.google.common.collect.AbstractIterator.hasNext(AbstractIterator.java:141)
at io.airbyte.commons.util.CompositeIterator.computeNext(CompositeIterator.java:63)
... 16 more
,retryable=<null>,timestamp=1674590389685], io.airbyte.config.FailureReason@462be4e0[failureOrigin=source,failureType=<null>,internalMessage=io.airbyte.workers.internal.exception.SourceException: Source process exited with non-zero exit code 1,externalMessage=Something went wrong within the source connector,metadata=io.airbyte.config.Metadata@29a76976[additionalProperties={attemptNumber=0, jobId=1, connector_command=read}],stacktrace=java.util.concurrent.CompletionException: io.airbyte.workers.internal.exception.SourceException: Source process exited with non-zero exit code 1
at java.base/java.util.concurrent.CompletableFuture.encodeThrowable(CompletableFuture.java:315)
at java.base/java.util.concurrent.CompletableFuture.completeThrowable(CompletableFuture.java:320)
at java.base/java.util.concurrent.CompletableFuture$AsyncRun.run(CompletableFuture.java:1807)
at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1144)
at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:642)
at java.base/java.lang.Thread.run(Thread.java:1589)
Caused by: io.airbyte.workers.internal.exception.SourceException: Source process exited with non-zero exit code 1
at io.airbyte.workers.general.DefaultReplicationWorker.lambda$readFromSrcAndWriteToDstRunnable$6(DefaultReplicationWorker.java:406)
at java.base/java.util.concurrent.CompletableFuture$AsyncRun.run(CompletableFuture.java:1804)
... 3 more
,retryable=<null>,timestamp=1674590390616], io.airbyte.config.FailureReason@69c94d87[failureOrigin=replication,failureType=<null>,internalMessage=java.lang.RuntimeException: java.io.UncheckedIOException: java.net.SocketException: Socket closed,externalMessage=Something went wrong during replication,metadata=io.airbyte.config.Metadata@9dc2644[additionalProperties={attemptNumber=0, jobId=1}],stacktrace=java.util.concurrent.CompletionException: java.lang.RuntimeException: java.io.UncheckedIOException: java.net.SocketException: Socket closed
at java.base/java.util.concurrent.CompletableFuture.encodeThrowable(CompletableFuture.java:315)
at java.base/java.util.concurrent.CompletableFuture.completeThrowable(CompletableFuture.java:320)
at java.base/java.util.concurrent.CompletableFuture$AsyncRun.run(CompletableFuture.java:1807)
at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1144)
at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:642)
at java.base/java.lang.Thread.run(Thread.java:1589)
Caused by: java.lang.RuntimeException: java.io.UncheckedIOException: java.net.SocketException: Socket closed
at io.airbyte.workers.general.DefaultReplicationWorker.lambda$readFromDstRunnable$4(DefaultReplicationWorker.java:312)
at java.base/java.util.concurrent.CompletableFuture$AsyncRun.run(CompletableFuture.java:1804)
... 3 more
Caused by: java.io.UncheckedIOException: java.net.SocketException: Socket closed
at java.base/java.io.BufferedReader$1.hasNext(BufferedReader.java:685)
at java.base/java.util.Spliterators$IteratorSpliterator.tryAdvance(Spliterators.java:1931)
at java.base/java.util.stream.StreamSpliterators$WrappingSpliterator.lambda$initPartialTraversalState$0(StreamSpliterators.java:292)
at java.base/java.util.stream.StreamSpliterators$AbstractWrappingSpliterator.fillBuffer(StreamSpliterators.java:206)
at java.base/java.util.stream.StreamSpliterators$AbstractWrappingSpliterator.doAdvance(StreamSpliterators.java:161)
at java.base/java.util.stream.StreamSpliterators$WrappingSpliterator.tryAdvance(StreamSpliterators.java:298)
at java.base/java.util.Spliterators$1Adapter.hasNext(Spliterators.java:681)
at io.airbyte.workers.internal.DefaultAirbyteDestination.isFinished(DefaultAirbyteDestination.java:157)
at io.airbyte.workers.general.DefaultReplicationWorker.lambda$readFromDstRunnable$4(DefaultReplicationWorker.java:275)
... 4 more
Caused by: java.net.SocketException: Socket closed
at java.base/sun.nio.ch.NioSocketImpl.endRead(NioSocketImpl.java:238)
at java.base/sun.nio.ch.NioSocketImpl.implRead(NioSocketImpl.java:317)
at java.base/sun.nio.ch.NioSocketImpl.read(NioSocketImpl.java:340)
at java.base/sun.nio.ch.NioSocketImpl$1.read(NioSocketImpl.java:789)
at java.base/java.net.Socket$SocketInputStream.read(Socket.java:1025)
at java.base/sun.nio.cs.StreamDecoder.readBytes(StreamDecoder.java:333)
at java.base/sun.nio.cs.StreamDecoder.implRead(StreamDecoder.java:376)
at java.base/sun.nio.cs.StreamDecoder.lockedRead(StreamDecoder.java:219)
at java.base/sun.nio.cs.StreamDecoder.read(StreamDecoder.java:173)
at java.base/java.io.InputStreamReader.read(InputStreamReader.java:189)
at java.base/java.io.BufferedReader.fill(BufferedReader.java:161)
at java.base/java.io.BufferedReader.implReadLine(BufferedReader.java:371)
at java.base/java.io.BufferedReader.readLine(BufferedReader.java:348)
at java.base/java.io.BufferedReader.readLine(BufferedReader.java:437)
at java.base/java.io.BufferedReader$1.hasNext(BufferedReader.java:682)
... 12 more
,retryable=<null>,timestamp=1674590450661]]]
Is there a way how I can get the full logs to you in private? It contains a lot of private data. thank you best
we have a array of type Datetime[] in postgres, which is likely to be the problem here.