airbyte
airbyte copied to clipboard
[destination-snowflake] java.lang.RuntimeException: Unable to deserialize PartialAirbyteMessage.
Connector Name
destination-snowflake
Connector Version
3.6.6
What step the error happened?
During the sync
Relevant information
Environment:
Source: airbyte/source-postgres:3.3.27
Destination: airbyte/destination-snowflake:3.6.6 (upgraded from 3.4.10)
Platform: 0.56.0 (upgraded from 0.50.33)
Issue:
When attempting to ingest all fields, the process fails with the following error:
2024-04-26 08:27:19 destination > WARN main i.a.c.j.Jsons(handleDeserThrowable):440 Failed to deserialize json due to class com.fasterxml.jackson.databind.JsonMappingException
at com.fasterxml.jackson.databind.JsonMappingException.wrapWithPath(JsonMappingException.java:402)
at com.fasterxml.jackson.databind.JsonMappingException.wrapWithPath(JsonMappingException.java:361)
at com.fasterxml.jackson.databind.deser.BeanDeserializerBase.wrapAndThrow(BeanDeserializerBase.java:1853)
at com.fasterxml.jackson.databind.deser.BeanDeserializer.vanillaDeserialize(BeanDeserializer.java:316)
at com.fasterxml.jackson.databind.deser.BeanDeserializer.deserialize(BeanDeserializer.java:177)
at com.fasterxml.jackson.databind.deser.impl.MethodProperty.deserializeAndSet(MethodProperty.java:129)
at com.fasterxml.jackson.databind.deser.BeanDeserializer.vanillaDeserialize(BeanDeserializer.java:314)
at com.fasterxml.jackson.databind.deser.BeanDeserializer.deserialize(BeanDeserializer.java:177)
at com.fasterxml.jackson.databind.deser.DefaultDeserializationContext.readRootValue(DefaultDeserializationContext.java:323)
at com.fasterxml.jackson.databind.ObjectMapper._readMapAndClose(ObjectMapper.java:4825)
at com.fasterxml.jackson.databind.ObjectMapper.readValue(ObjectMapper.java:3772)
at com.fasterxml.jackson.databind.ObjectMapper.readValue(ObjectMapper.java:3740)
at io.airbyte.commons.json.Jsons.tryDeserializeExact(Jsons.java:146)
at io.airbyte.cdk.integrations.destination.async.deser.DeserializationUtil.deserializeAirbyteMessage(DeserializationUtil.kt:31)
at io.airbyte.cdk.integrations.destination.async.AsyncStreamConsumer.accept(AsyncStreamConsumer.kt:204)
at io.airbyte.cdk.integrations.destination.async.AsyncStreamConsumer.accept(AsyncStreamConsumer.kt:46)
at io.airbyte.cdk.integrations.base.IntegrationRunner.consumeWriteStream(IntegrationRunner.java:309)
at io.airbyte.cdk.integrations.base.IntegrationRunner.consumeWriteStream(IntegrationRunner.java:289)
at io.airbyte.cdk.integrations.base.IntegrationRunner.runInternal(IntegrationRunner.java:190)
at io.airbyte.cdk.integrations.base.IntegrationRunner.run(IntegrationRunner.java:125)
at io.airbyte.cdk.integrations.base.adaptive.AdaptiveDestinationRunner$Runner.run(AdaptiveDestinationRunner.java:88)
at io.airbyte.integrations.destination.snowflake.SnowflakeDestinationRunner.main(SnowflakeDestinationRunner.java:20)
Additional message
2024-04-26 08:28:14 destination > ERROR main i.a.c.i.b.AirbyteExceptionHandler(uncaughtException):64 Something went wrong in the connector. See the logs for more details. java.lang.RuntimeException: Unable to deserialize PartialAirbyteMessage.
at io.airbyte.cdk.integrations.destination.async.deser.DeserializationUtil.deserializeAirbyteMessage$lambda$0(DeserializationUtil.kt:32) ~[airbyte-cdk-core-0.27.7.jar:?]
at java.base/java.util.Optional.orElseThrow(Optional.java:403) ~[?:?]
at io.airbyte.cdk.integrations.destination.async.deser.DeserializationUtil.deserializeAirbyteMessage(DeserializationUtil.kt:32) ~[airbyte-cdk-core-0.27.7.jar:?]
at io.airbyte.cdk.integrations.destination.async.AsyncStreamConsumer.accept(AsyncStreamConsumer.kt:204) ~[airbyte-cdk-core-0.27.7.jar:?]
at io.airbyte.cdk.integrations.destination.async.AsyncStreamConsumer.accept(AsyncStreamConsumer.kt:46) ~[airbyte-cdk-core-0.27.7.jar:?]
at io.airbyte.cdk.integrations.base.IntegrationRunner.consumeWriteStream(IntegrationRunner.java:309) ~[airbyte-cdk-core-0.27.7.jar:?]
at io.airbyte.cdk.integrations.base.IntegrationRunner.consumeWriteStream(IntegrationRunner.java:289) ~[airbyte-cdk-core-0.27.7.jar:?]
at io.airbyte.cdk.integrations.base.IntegrationRunner.runInternal(IntegrationRunner.java:190) ~[airbyte-cdk-core-0.27.7.jar:?]
at io.airbyte.cdk.integrations.base.IntegrationRunner.run(IntegrationRunner.java:125) ~[airbyte-cdk-core-0.27.7.jar:?]
at io.airbyte.cdk.integrations.base.adaptive.AdaptiveDestinationRunner$Runner.run(AdaptiveDestinationRunner.java:88) ~[airbyte-cdk-core-0.27.7.jar:?]
at io.airbyte.integrations.destination.snowflake.SnowflakeDestinationRunner.main(SnowflakeDestinationRunner.java:20) ~[io.airbyte.airbyte-integrations.connectors-destination-snowflake.jar:?]
Observation:
The process succeeds when one of the large-sized fields is discarded. So, we assume the issue is with the column length, but it is an assumption. Has something on the snowflake connector been changed?
Request:
I'm looking for help obtaining more detailed logs for debugging purposes. To facilitate debugging, it would be helpful to identify the affected rows or similar information. The lack of detailed error messages makes it challenging to pinpoint the exact cause of failure.
Log information
attempt_1.txt attempt_2.txt attempt_3.txt attempt_4.txt attempt_5.txt
Relevant log output
No response
Contribute
- [X] Yes, I want to contribute
Is this a new sync or it was workign before?
When we tested the sync in the past with the config 👇, it worked.
| platform version | destination |
|---|---|
| airbyte 0.50.33 | airbyte/destination-snowflake:3.4.10 |
Then we upgraded to:
| platform version | destination |
|---|---|
| airbyte 0.56.0 | airbyte/destination-snowflake: 3.6.6 |
Could something change on the connector? Or is it related to the message size?
hello, I encountered the same error while syncing mysql to snowflake, the error message is at java.base/java.lang.Thread.run(Thread.java:1583) 2024-05-06 21:15:48 destination > ERROR main i.a.c.i.b.AirbyteExceptionHandler(uncaughtException):64 Something went wrong in the connector. See the logs for more details. java.lang.RuntimeException: Unable to deserialize PartialAirbyteMessage. at io.airbyte.cdk.integrations.destination.async.deser.DeserializationUtil.deserializeAirbyteMessage$lambda$0(DeserializationUtil.kt:32) ~[airbyte-cdk-core-0.27.7.jar:?] at java.base/java.util.Optional.orElseThrow(Optional.java:403) ~[?:?].
Is there any workaround for this? can we get older version of the connector to try?
Thanks,
We're encountering the same issue with the following versions:
| Platform Version | Destination Version |
|---|---|
| airbyte 0.59.0 | airbyte/destination-snowflake: 3.7.4 |
2024-05-10 10:55:12 platform > failures: [ { "failureOrigin" : "destination", "failureType" : "system_error", "internalMessage" : "java.lang.RuntimeException: Unable to deserialize PartialAirbyteMessage.", "externalMessage" : "Something went wrong in the connector. See the logs for more details.", "metadata" : { "attemptNumber" : 4, "jobId" : 13, "from_trace_message" : true, "connector_command" : "write" },
I raised this to the destination team. Hope to get an updated about what is causing this errors. @marcraminv if you return to the previous version it works?
We haven't tried because we migrated everything to the last version on the Platform, but we tried to downgrade the destination and are still having the same issue.
Hello everyone! 👋 Please upgrade to the latest version of Destination Snowflake 3.8.0. This version uses the most recent Airbyte Destination CDK, which should resolve the issue you're currently facing. Let me know if the problem is solved after the upgrade. Thanks!
Hey @marcosmarxm, We upgraded to:
| platform version | destination |
|---|---|
| airbyte 0.59.1 | airbyte/destination-snowflake:3.8.0 |
The problem is still there:
Can you also confirm that you are using the latest version of the Airbyte platform? we changed how we handle large records recently
Hey, @evantahler, I can confirm that we upgraded to the last platform version, 0.59.1.
I can provide more context and narrow down the issue. After reading your message, I found the column that was causing the issue (due to the large record). I don't know if it helps, but I can tell you that the conflicted column is a JSON type on Postgres.
data_type: JSON
character_maximum_length: Empty
is_nullable: YES
column_default: Empty
https://github.com/airbytehq/airbyte-internal-issues/issues/7113 will make the error clearer
Hey @evantahler, I think the link points to a private repository. A part of this, thank you so much taking care of the issue, really appreciated.
The next version of Snowflake will solve this issue (as well as implement refreshes) - stay tuned.
Hey @evantahler , I have been following this thread. Is it possible that the new release will also fix the issues as described in #38641 since it seems like both the issues might be related to inability to deserialise some message?
Got this error when syncing postgres (3.3.27) -> BigQuery (2.4.12) on Airbyte 0.60.0.
at io.airbyte.cdk.integrations.base.IntegrationRunner.run(IntegrationRunner.java:125)
at io.airbyte.integrations.destination.bigquery.BigQueryDestination.main(BigQueryDestination.java:477)
2024-06-19 09:12:30 platform > readFromDestination: exception caught
io.airbyte.workers.internal.exception.DestinationException: Destination process exited with non-zero exit code 1
at io.airbyte.workers.general.BufferedReplicationWorker.readFromDestination(BufferedReplicationWorker.java:493) ~[io.airbyte-airbyte-commons-worker-0.60.0.jar:?]
at io.airbyte.workers.general.BufferedReplicationWorker.lambda$runAsync$2(BufferedReplicationWorker.java:235) ~[io.airbyte-airbyte-commons-worker-0.60.0.jar:?]
at java.base/java.util.concurrent.CompletableFuture$AsyncRun.run(CompletableFuture.java:1804) ~[?:?]
at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1144) ~[?:?]
at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:642) ~[?:?]
at java.base/java.lang.Thread.run(Thread.java:1583) [?:?]
2024-06-19 09:12:30 platform > writeToDestination: exception caught
java.io.IOException: Broken pipe
at java.base/java.io.FileOutputStream.writeBytes(Native Method) ~[?:?]
at java.base/java.io.FileOutputStream.write(FileOutputStream.java:367) ~[?:?]
at java.base/java.io.BufferedOutputStream.implWrite(BufferedOutputStream.java:217) ~[?:?]
at java.base/java.io.BufferedOutputStream.write(BufferedOutputStream.java:206) ~[?:?]
at java.base/sun.nio.cs.StreamEncoder.writeBytes(StreamEncoder.java:309) ~[?:?]
at java.base/sun.nio.cs.StreamEncoder.implFlushBuffer(StreamEncoder.java:405) ~[?:?]
at java.base/sun.nio.cs.StreamEncoder.implFlush(StreamEncoder.java:410) ~[?:?]
at java.base/sun.nio.cs.StreamEncoder.lockedFlush(StreamEncoder.java:214) ~[?:?]
at java.base/sun.nio.cs.StreamEncoder.flush(StreamEncoder.java:201) ~[?:?]
at java.base/java.io.OutputStreamWriter.flush(OutputStreamWriter.java:262) ~[?:?]
at java.base/java.io.BufferedWriter.implFlush(BufferedWriter.java:372) ~[?:?]
at java.base/java.io.BufferedWriter.flush(BufferedWriter.java:359) ~[?:?]
at io.airbyte.workers.internal.DefaultAirbyteMessageBufferedWriter.flush(DefaultAirbyteMessageBufferedWriter.java:31) ~[io.airbyte-airbyte-commons-worker-0.60.0.jar:?]
at io.airbyte.workers.internal.DefaultAirbyteDestination.notifyEndOfInputWithNoTimeoutMonitor(DefaultAirbyteDestination.java:155) ~[io.airbyte-airbyte-commons-worker-0.60.0.jar:?]
at io.airbyte.workers.internal.DefaultAirbyteDestination.notifyEndOfInput(DefaultAirbyteDestination.java:145) ~[io.airbyte-airbyte-commons-worker-0.60.0.jar:?]
at io.airbyte.workers.general.BufferedReplicationWorker.writeToDestination(BufferedReplicationWorker.java:459) ~[io.airbyte-airbyte-commons-worker-0.60.0.jar:?]
at io.airbyte.workers.general.BufferedReplicationWorker.lambda$runAsyncWithTimeout$5(BufferedReplicationWorker.java:263) ~[io.airbyte-airbyte-commons-worker-0.60.0.jar:?]
at java.base/java.util.concurrent.CompletableFuture$AsyncRun.run(CompletableFuture.java:1804) ~[?:?]
at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1144) ~[?:?]
at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:642) ~[?:?]
at java.base/java.lang.Thread.run(Thread.java:1583) [?:?]
2024-06-19 09:12:30 platform > writeToDestination: done. (forDest.isDone:false, isDestRunning:true)
2024-06-19 09:12:30 platform > readFromDestination: done. (writeToDestFailed:true, dest.isFinished:true)
2024-06-19 09:12:30 platform > processMessage: done. (fromSource.isDone:false, forDest.isClosed:true)
2024-06-19 09:12:30 platform > readFromSource: exception caught
java.lang.IllegalStateException: Source process is still alive, cannot retrieve exit value.
at com.google.common.base.Preconditions.checkState(Preconditions.java:515) ~[guava-33.1.0-jre.jar:?]
at io.airbyte.workers.internal.DefaultAirbyteSource.getExitValue(DefaultAirbyteSource.java:136) ~[io.airbyte-airbyte-commons-worker-0.60.0.jar:?]
at io.airbyte.workers.general.BufferedReplicationWorker.readFromSource(BufferedReplicationWorker.java:375) ~[io.airbyte-airbyte-commons-worker-0.60.0.jar:?]
at io.airbyte.workers.general.BufferedReplicationWorker.lambda$runAsyncWithHeartbeatCheck$3(BufferedReplicationWorker.java:242) ~[io.airbyte-airbyte-commons-worker-0.60.0.jar:?]
at java.base/java.util.concurrent.CompletableFuture$AsyncRun.run(CompletableFuture.java:1804) ~[?:?]
at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1144) ~[?:?]
at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:642) ~[?:?]
at java.base/java.lang.Thread.run(Thread.java:1583) [?:?]
2024-06-19 09:12:30 platform > readFromSource: done. (source.isFinished:false, fromSource.isClosed:true)
2024-06-19 09:13:30 platform > airbyte-source gobbler IOException: Stream closed. Typically happens when cancelling a job.
@killthekitten this issue is for snowflake. The latest version of the bigquery connector (v2.8.0) has the fix for that connector.
Closed by https://github.com/airbytehq/airbyte/pull/39473 - v3.11.0
I am getting same error on some connectors, following are the version I am using, if someone could point me in the right direction to solve this will be appreciated.
Airbyte version: 0.64.1
Source postgres version: 3.6.16
Destination snowflake: 3.11.10
java.lang.RuntimeException: Could not deserialize PartialAirbyteMessage: class com.fasterxml.jackson.databind.JsonMappingException at com.fasterxml.jackson.databind.JsonMappingException.wrapWithPath(JsonMappingException.java:402) at com.fasterxml.jackson.databind.JsonMappingException.wrapWithPath(JsonMappingException.java:361) at com.fasterxml.jackson.databind.deser.BeanDeserializerBase.wrapAndThrow(BeanDeserializerBase.java:1853) at com.fasterxml.jackson.databind.deser.BeanDeserializer.vanillaDeserialize(BeanDeserializer.java:316) at com.fasterxml.jackson.databind.deser.BeanDeserializer.deserialize(BeanDeserializer.java:177) at com.fasterxml.jackson.databind.deser.impl.MethodProperty.deserializeAndSet(MethodProperty.java:129) at com.fasterxml.jackson.databind.deser.BeanDeserializer.vanillaDeserialize(BeanDeserializer.java:314) at com.fasterxml.jackson.databind.deser.BeanDeserializer.deserialize(BeanDeserializer.java:177) at com.fasterxml.jackson.databind.deser.DefaultDeserializationContext.readRootValue(DefaultDeserializationContext.java:323) at com.fasterxml.jackson.databind.ObjectMapper._readMapAndClose(ObjectMapper.java:4825) at com.fasterxml.jackson.databind.ObjectMapper.readValue(ObjectMapper.java:3772) at com.fasterxml.jackson.databind.ObjectMapper.readValue(ObjectMapper.java:3740) at io.airbyte.commons.json.Jsons.deserializeExactUnchecked(Jsons.kt:147) at io.airbyte.cdk.integrations.destination.async.deser.AirbyteMessageDeserializer.deserializeAirbyteMessage(AirbyteMessageDeserializer.kt:48) at io.airbyte.cdk.integrations.destination.async.AsyncStreamConsumer.accept(AsyncStreamConsumer.kt:107) at io.airbyte.cdk.integrations.base.IntegrationRunner$Companion$consumeWriteStream$2$1.invoke(IntegrationRunner.kt:385) at io.airbyte.cdk.integrations.base.IntegrationRunner$Companion$consumeWriteStream$2$1.invoke(IntegrationRunner.kt:383) at io.airbyte.cdk.integrations.base.IntegrationRunner$Companion.consumeWriteStream$lambda$1$lambda$0(IntegrationRunner.kt:383) at java.base/java.util.Iterator.forEachRemaining(Iterator.java:133) at java.base/java.util.Spliterators$IteratorSpliterator.forEachRemaining(Spliterators.java:1939) at java.base/java.util.stream.ReferencePipeline$Head.forEach(ReferencePipeline.java:762) at io.airbyte.cdk.integrations.base.IntegrationRunner$Companion.consumeWriteStream$io_airbyte_airbyte_cdk_java_airbyte_cdk_airbyte_cdk_core(IntegrationRunner.kt:383) at io.airbyte.cdk.integrations.base.IntegrationRunner$Companion.consumeWriteStream$io_airbyte_airbyte_cdk_java_airbyte_cdk_airbyte_cdk_core$default(IntegrationRunner.kt:375) at io.airbyte.cdk.integrations.base.IntegrationRunner.runInternal(IntegrationRunner.kt:215) at io.airbyte.cdk.integrations.base.IntegrationRunner.run(IntegrationRunner.kt:119) at io.airbyte.cdk.integrations.base.IntegrationRunner.run$default(IntegrationRunner.kt:113) at io.airbyte.cdk.integrations.base.adaptive.AdaptiveDestinationRunner$Runner.run(AdaptiveDestinationRunner.kt:68) at io.airbyte.integrations.destination.snowflake.SnowflakeDestinationKt.main(SnowflakeDestination.kt:350) Caused by class com.fasterxml.jackson.core.io.JsonEOFException at com.fasterxml.jackson.core.base.ParserMinimalBase._reportInvalidEOF(ParserMinimalBase.java:697) at com.fasterxml.jackson.core.json.ReaderBasedJsonParser._skipColon2(ReaderBasedJsonParser.java:2405) at com.fasterxml.jackson.core.json.ReaderBasedJsonParser._skipColon(ReaderBasedJsonParser.java:2320) at com.fasterxml.jackson.core.json.ReaderBasedJsonParser.nextFieldName(ReaderBasedJsonParser.java:971) at com.fasterxml.jackson.databind.deser.std.BaseNodeDeserializer._deserializeContainerNoRecursion(JsonNodeDeserializer.java:539) at com.fasterxml.jackson.databind.deser.std.JsonNodeDeserializer.deserialize(JsonNodeDeserializer.java:100) at com.fasterxml.jackson.databind.deser.std.JsonNodeDeserializer.deserialize(JsonNodeDeserializer.java:25) at com.fasterxml.jackson.databind.deser.impl.MethodProperty.deserializeAndSet(MethodProperty.java:129) at com.fasterxml.jackson.databind.deser.BeanDeserializer.vanillaDeserialize(BeanDeserializer.java:314) at com.fasterxml.jackson.databind.deser.BeanDeserializer.deserialize(BeanDeserializer.java:177) at com.fasterxml.jackson.databind.deser.impl.MethodProperty.deserializeAndSet(MethodProperty.java:129) at com.fasterxml.jackson.databind.deser.BeanDeserializer.vanillaDeserialize(BeanDeserializer.java:314) at com.fasterxml.jackson.databind.deser.BeanDeserializer.deserialize(BeanDeserializer.java:177) at com.fasterxml.jackson.databind.deser.DefaultDeserializationContext.readRootValue(DefaultDeserializationContext.java:323) at com.fasterxml.jackson.databind.ObjectMapper._readMapAndClose(ObjectMapper.java:4825) at com.fasterxml.jackson.databind.ObjectMapper.readValue(ObjectMapper.java:3772) at com.fasterxml.jackson.databind.ObjectMapper.readValue(ObjectMapper.java:3740) at io.airbyte.commons.json.Jsons.deserializeExactUnchecked(Jsons.kt:147) at io.airbyte.cdk.integrations.destination.async.deser.AirbyteMessageDeserializer.deserializeAirbyteMessage(AirbyteMessageDeserializer.kt:48) at io.airbyte.cdk.integrations.destination.async.AsyncStreamConsumer.accept(AsyncStreamConsumer.kt:107) at io.airbyte.cdk.integrations.base.IntegrationRunner$Companion$consumeWriteStream$2$1.invoke(IntegrationRunner.kt:385) at io.airbyte.cdk.integrations.base.IntegrationRunner$Companion$consumeWriteStream$2$1.invoke(IntegrationRunner.kt:383) at io.airbyte.cdk.integrations.base.IntegrationRunner$Companion.consumeWriteStream$lambda$1$lambda$0(IntegrationRunner.kt:383) at java.base/java.util.Iterator.forEachRemaining(Iterator.java:133) at java.base/java.util.Spliterators$IteratorSpliterator.forEachRemaining(Spliterators.java:1939) at java.base/java.util.stream.ReferencePipeline$Head.forEach(ReferencePipeline.java:762) at io.airbyte.cdk.integrations.base.IntegrationRunner$Companion.consumeWriteStream$io_airbyte_airbyte_cdk_java_airbyte_cdk_airbyte_cdk_core(IntegrationRunner.kt:383) at io.airbyte.cdk.integrations.base.IntegrationRunner$Companion.consumeWriteStream$io_airbyte_airbyte_cdk_java_airbyte_cdk_airbyte_cdk_core$default(IntegrationRunner.kt:375) at io.airbyte.cdk.integrations.base.IntegrationRunner.runInternal(IntegrationRunner.kt:215) at io.airbyte.cdk.integrations.base.IntegrationRunner.run(IntegrationRunner.kt:119) at io.airbyte.cdk.integrations.base.IntegrationRunner.run$default(IntegrationRunner.kt:113) at io.airbyte.cdk.integrations.base.adaptive.AdaptiveDestinationRunner$Runner.run(AdaptiveDestinationRunner.kt:68) at io.airbyte.integrations.destination.snowflake.SnowflakeDestinationKt.main(SnowflakeDestination.kt:350)