Local / S3 to RedShift Normalization Fails

[akshat-crypto]

Environment

• Airbyte version: example is 0.22.0-alpha
• OS Version / Instance: Ubuntu 20.04, AWS EC2
• Deployment: Docker-compose env
• Source Connector and version: File 0.2.33
• Destination Connector and version: Redshift 0.3.53
• Step where error happened: Sync job / Setup new connection

Current Behavior

From Local source or from s3 bucket I am not able to sync the csv files into the redshift cluster In the logs its mentioned somewhere:

Checking if airbyte/normalization-redshift:0.2.25 exists... 2023-01-28 05:34:32 [32mINFO[m i.a.c.i.LineGobbler(voidCall):114 - airbyte/normalization-redshift:0.2.25 not found locally. Attempting to pull the image... 2023-01-28 05:34:36 [32mINFO[m i.a.c.i.LineGobbler(voidCall):114 - Image does not exist. 2023-01-28 05:34:36 [1;31mERROR[m i.a.w.g.DefaultNormalizationWorker(run):83 - Normalization failed for job 21. io.airbyte.workers.exception.WorkerException: Could not find image: airbyte/normalization-redshift:0.2.25 at io.airbyte.workers.process.DockerProcessFactory.create(DockerProcessFactory.java:98) ~[io.airbyte-airbyte-commons-worker-0.40.28.jar:?] at io.airbyte.workers.normalization.DefaultNormalizationRunner.runProcess(DefaultNormalizationRunner.java:126) ~[io.airbyte-airbyte-common

2023-01-28 05:34:43 [32mINFO[m i.a.c.i.LineGobbler(voidCall):114 - Checking if airbyte/destination-redshift:0.3.53 exists... 2023-01-28 05:34:43 [32mINFO[m i.a.c.i.LineGobbler(voidCall):114 - airbyte/destination-redshift:0.3.53 was found locally. 2023-01-28 05:34:43 [32mINFO[m i.a.w.p.DockerProcessFactory(create):120 - Creating docker container = destination-redshift-write-21-2-nehbn with resources io.airbyte.config.ResourceRequirements@42f1ad8b[cpuRequest=,cpuLimit=,memoryRequest=,memoryLimit=] 2023-01-28 05:34:43 [32mINFO[m i.a.w.p.DockerProcessFactory(create):164 - Preparing command: docker run --rm --init -i -w /data/21/2 --log-driver none --name destination-redshift-write-21-2-nehbn --network host -v airbyte_workspace:/data -v /tmp/airbyte_local:/local -e DEPLOYMENT_MODE=OSS -e USE_STREAM_CAPABLE_STATE=true -e FIELD_SELECTION_WORKSPACES= -e WORKER_ENVIRONMENT=DOCKER -e AIRBYTE_ROLE= -e APPLY_FIELD_SELECTION=false -e WORKER_JOB_ATTEMPT=2 -e WORKER_CONNECTOR_IMAGE=airbyte/destination-redshift:0.3.53 -e AUTO_DETECT_SCHEMA=false -e AIRBYTE_VERSION=0.40.28 -e WORKER_JOB_ID=21 airbyte/destination-redshift:0.3.53 write --config destination_config.json --catalog destination_catalog.json 2023-01-28 05:34:43 [32mINFO[m i.a.w.i.VersionedAirbyteMessageBufferedWriterFactory(createWriter):33 - Writing messages to protocol version 0.2.0

Expected Behavior

It must copy all the data in the csv files into a redshift cluster database under the given schema name but the data is not being transferred directly there

Logs

Caused by: io.temporal.serviceclient.CheckedExceptionWrapper: java.util.concurrent.ExecutionException: io.airbyte.workers.exception.WorkerException: Normalization Failed. at io.temporal.serviceclient.CheckedExceptionWrapper.wrap(CheckedExceptionWrapper.java:57) ~[temporal-serviceclient-1.17.0.jar:?] at io.temporal.internal.sync.WorkflowInternal.wrap(WorkflowInternal.java:461) ~[temporal-sdk-1.17.0.jar:?] at io.temporal.activity.Activity.wrap(Activity.java:52) ~[temporal-sdk-1.17.0.jar:?] at io.airbyte.workers.temporal.TemporalAttemptExecution.get(TemporalAttemptExecution.java:167) ~[io.airbyte-airbyte-workers-0.40.28.jar:?] at io.airbyte.workers.temporal.sync.NormalizationActivityImpl.lambda$normalize$3(NormalizationActivityImpl.java:133) ~[io.airbyte-airbyte-workers-0.40.28.jar:?] at io.airbyte.commons.temporal.TemporalUtils.withBackgroundHeartbeat(TemporalUtils.java:276) ~[io.airbyte-airbyte-commons-temporal-0.40.28.jar:?] ... 14 more Caused by: java.util.concurrent.ExecutionException: io.airbyte.workers.exception.WorkerException: Normalization Failed. at java.util.concurrent.CompletableFuture.reportGet(CompletableFuture.java:396) ~[?:?] at java.util.concurrent.CompletableFuture.get(CompletableFuture.java:2073) ~[?:?] at io.airbyte.workers.temporal.TemporalAttemptExecution.get(TemporalAttemptExecution.java:161) ~[io.airbyte-airbyte-workers-0.40.28.jar:?] at io.airbyte.workers.temporal.sync.NormalizationActivityImpl.lambda$normalize$3(NormalizationActivityImpl.java:133) ~[io.airbyte-airbyte-workers-0.40.28.jar:?] at io.airbyte.commons.temporal.TemporalUtils.withBackgroundHeartbeat(TemporalUtils.java:276) ~[io.airbyte-airbyte-commons-temporal-0.40.28.jar:?] ... 14 more Caused by: io.airbyte.workers.exception.WorkerException: Normalization Failed. at io.airbyte.workers.general.DefaultNormalizationWorker.run(DefaultNormalizationWorker.java:103) ~[io.airbyte-airbyte-commons-worker-0.40.28.jar:?] at io.airbyte.workers.general.DefaultNormalizationWorker.run(DefaultNormalizationWorker.java:34) ~[io.airbyte-airbyte-commons-worker-0.40.28.jar:?] at io.airbyte.workers.temporal.TemporalAttemptExecution.lambda$getWorkerThread$5(TemporalAttemptExecution.java:195) ~[io.airbyte-airbyte-workers-0.40.28.jar:?] ... 1 more 2023-01-28 05:34:54 INFO i.a.c.f.EnvVariableFeatureFlags(autoDisablesFailingConnections):27 - Auto Disable Failing Connections: false

airbyte_logs.txt

Steps to Reproduce

1. Use the docker-compose to run the airbyte

2. After setting up the connection use - Files/S3 as source and upload a demo csv file. jaffle_shop_orders.csv

3. Give details for the normal Redshift cluster to connect.

4. Run the connection setup and check logs