airbyte icon indicating copy to clipboard operation
airbyte copied to clipboard

Published 20 hours ago • verified publisher icon airbytehq

Source DynamoDB - Reserved Keyword Issue

Open piyushsingariya opened this issue 2 years ago • 5 comments

Environment

  • Source Connector and version: dynamodb 0.1.0
  • Step where error happened: Sync job

Current Behavior

If DynamoDB tables contains any reserved keyword in dynamo expressions it results in this error while running sync.

Expected Behavior

This should not happen, field_names should be added in dynamo query as placeholder shown in the solution on StackOverflow List of reserved keywords here

Logs

2022-11-23 05:35:35 [DEBUG] -
{"type":"TRACE","trace":{"type":"ERROR","emitted_at":1.669181735072E12,"error":{"message":"Something went wrong in the connector. See the logs for more
details.","internal_message":"software.amazon.awssdk.services.dynamodb.model.DynamoDbException: Invalid ProjectionExpression: Attribute name is a reserved keyword; reserved
keyword: name (Service: DynamoDb, Status Code: 400, Request ID:
NHCNIL7CDBVM1ABAU2573P0C4VVV4KQNSO5AEMVJF66Q9ASUAAJG)","stack_trace":"software.amazon.awssdk.services.dynamodb.model.DynamoDbException: Invalid ProjectionExpression: Attribute name is a reserved keyword; reserved
keyword: name (Service: DynamoDb, Status Code: 400, Request ID:
NHCNIL7CDBVM1ABAU2573P0C4VVV4KQNSO5AEMVJF66Q9ASUAAJG)\n\tat
software.amazon.awssdk.core.internal.http.CombinedResponseHandler.handleErrorResponse(CombinedResponseHandler.java:125)\n\tat
software.amazon.awssdk.core.internal.http.CombinedResponseHandler.handleResponse(CombinedResponseHandler.java:82)\n\tat
software.amazon.awssdk.core.internal.http.CombinedResponseHandler.handle(CombinedResponseHandler.java:60)\n\tat
software.amazon.awssdk.core.internal.http.CombinedResponseHandler.handle(CombinedResponseHandler.java:41)\n\tat
software.amazon.awssdk.core.internal.http.pipeline.stages.HandleResponseStage.execute(HandleResponseStage.java:40)\n\tat
software.amazon.awssdk.core.internal.http.pipeline.stages.HandleResponseStage.execute(HandleResponseStage.java:30)\n\tat
software.amazon.awssdk.core.internal.http.pipeline.RequestPipelineBuilder$ComposingRequestPipelineStage.execute(RequestPipelineBuilder.java:206)\n\tat
software.amazon.awssdk.core.internal.http.pipeline.stages.ApiCallAttemptTimeoutTrackingStage.execute(ApiCallAttemptTimeoutTrackingStage.java:73)\n\tat
software.amazon.awssdk.core.internal.http.pipeline.stages.ApiCallAttemptTimeoutTrackingStage.execute(ApiCallAttemptTimeoutTrackingStage.java:42)\n\tat
software.amazon.awssdk.core.internal.http.pipeline.stages.TimeoutExceptionHandlingStage.execute(TimeoutExceptionHandlingStage.java:78)\n\tat
software.amazon.awssdk.core.internal.http.pipeline.stages.TimeoutExceptionHandlingStage.execute(TimeoutExceptionHandlingStage.java:40)\n\tat
software.amazon.awssdk.core.internal.http.pipeline.stages.ApiCallAttemptMetricCollectionStage.execute(ApiCallAttemptMetricCollectionStage.java:50)\n\tat
software.amazon.awssdk.core.internal.http.pipeline.stages.ApiCallAttemptMetricCollectionStage.execute(ApiCallAttemptMetricCollectionStage.java:36)\n\tat
software.amazon.awssdk.core.internal.http.pipeline.stages.RetryableStage.execute(RetryableStage.java:81)\n\tat
software.amazon.awssdk.core.internal.http.pipeline.stages.RetryableStage.execute(RetryableStage.java:36)\n\tat
software.amazon.awssdk.core.internal.http.pipeline.RequestPipelineBuilder$ComposingRequestPipelineStage.execute(RequestPipelineBuilder.java:206)\n\tat
software.amazon.awssdk.core.internal.http.StreamManagingStage.execute(StreamManagingStage.java:56)\n\tat
software.amazon.awssdk.core.internal.http.StreamManagingStage.execute(StreamManagingStage.java:36)\n\tat
software.amazon.awssdk.core.internal.http.pipeline.stages.ApiCallTimeoutTrackingStage.executeWithTimer(ApiCallTimeoutTrackingStage.java:80)\n\tat
software.amazon.awssdk.core.internal.http.pipeline.stages.ApiCallTimeoutTrackingStage.execute(ApiCallTimeoutTrackingStage.java:60)\n\tat
software.amazon.awssdk.core.internal.http.pipeline.stages.ApiCallTimeoutTrackingStage.execute(ApiCallTimeoutTrackingStage.java:42)\n\tat
software.amazon.awssdk.core.internal.http.pipeline.stages.ApiCallMetricCollectionStage.execute(ApiCallMetricCollectionStage.java:48)\n\tat
software.amazon.awssdk.core.internal.http.pipeline.stages.ApiCallMetricCollectionStage.execute(ApiCallMetricCollectionStage.java:31)\n\tat
software.amazon.awssdk.core.internal.http.pipeline.RequestPipelineBuilder$ComposingRequestPipelineStage.execute(RequestPipelineBuilder.java:206)\n\tat
software.amazon.awssdk.core.internal.http.pipeline.RequestPipelineBuilder$ComposingRequestPipelineStage.execute(RequestPipelineBuilder.java:206)\n\tat
software.amazon.awssdk.core.internal.http.pipeline.stages.ExecutionFailureExceptionReportingStage.execute(ExecutionFailureExceptionReportingStage.java:37)\n\tat
software.amazon.awssdk.core.internal.http.pipeline.stages.ExecutionFailureExceptionReportingStage.execute(ExecutionFailureExceptionReportingStage.java:26)\n\tat
software.amazon.awssdk.core.internal.http.AmazonSyncHttpClient$RequestExecutionBuilderImpl.execute(AmazonSyncHttpClient.java:193)\n\tat
software.amazon.awssdk.core.internal.handler.BaseSyncClientHandler.invoke(BaseSyncClientHandler.java:103)\n\tat
software.amazon.awssdk.core.internal.handler.BaseSyncClientHandler.doExecute(BaseSyncClientHandler.java:171)\n\tat
software.amazon.awssdk.core.internal.handler.BaseSyncClientHandler.lambda$execute$1(BaseSyncClientHandler.java:82)\n\tat
software.amazon.awssdk.core.internal.handler.BaseSyncClientHandler.measureApiCallSuccess(BaseSyncClientHandler.java:179)\n\tat
software.amazon.awssdk.core.internal.handler.BaseSyncClientHandler.execute(BaseSyncClientHandler.java:76)\n\tat
software.amazon.awssdk.core.client.handler.SdkSyncClientHandler.execute(SdkSyncClientHandler.java:45)\n\tat
software.amazon.awssdk.awscore.client.handler.AwsSyncClientHandler.execute(AwsSyncClientHandler.java:56)\n\tat
software.amazon.awssdk.services.dynamodb.DefaultDynamoDbClient.scan(DefaultDynamoDbClient.java:4876)\n\tat
software.amazon.awssdk.services.dynamodb.paginators.ScanIterable$ScanResponseFetcher.nextPage(ScanIterable.java:134)\n\tat
software.amazon.awssdk.services.dynamodb.paginators.ScanIterable$ScanResponseFetcher.nextPage(ScanIterable.java:125)\n\tat
software.amazon.awssdk.core.pagination.sync.PaginatedResponsesIterator.next(PaginatedResponsesIterator.java:58)\n\tat
io.airbyte.integrations.source.dynamodb.DynamodbOperations.scanTable(DynamodbOperations.java:139)\n\tat
io.airbyte.integrations.source.dynamodb.DynamodbSource.scanFullRefresh(DynamodbSource.java:181)\n\tat
io.airbyte.integrations.source.dynamodb.DynamodbSource.lambda$read$1(DynamodbSource.java:106)\n\tat
java.base/java.util.stream.ReferencePipeline$3$1.accept(ReferencePipeline.java:197)\n\tat
java.base/java.util.ArrayList$ArrayListSpliterator.forEachRemaining(ArrayList.java:1625)\n\tat
java.base/java.util.stream.AbstractPipeline.copyInto(AbstractPipeline.java:509)\n\tat
java.base/java.util.stream.AbstractPipeline.wrapAndCopyInto(AbstractPipeline.java:499)\n\tat
java.base/java.util.stream.AbstractPipeline.evaluate(AbstractPipeline.java:575)\n\tat
java.base/java.util.stream.AbstractPipeline.evaluateToArrayNode(AbstractPipeline.java:260)\n\tat
java.base/java.util.stream.ReferencePipeline.toArray(ReferencePipeline.java:616)\n\tat
java.base/java.util.stream.ReferencePipeline.toArray(ReferencePipeline.java:622)\n\tat
java.base/java.util.stream.ReferencePipeline.toList(ReferencePipeline.java:627)\n\tat
io.airbyte.integrations.source.dynamodb.DynamodbSource.read(DynamodbSource.java:108)\n\tat
io.airbyte.integrations.base.IntegrationRunner.runInternal(IntegrationRunner.java:136)\n\tat
io.airbyte.integrations.base.IntegrationRunner.run(IntegrationRunner.java:97)\n\tat
io.airbyte.integrations.source.dynamodb.DynamodbSource.main(DynamodbSource.java:46)\n","failure_type":"system_error"}}}

Are you willing to submit a PR?

I am, but I'm no good at Java. Help would be much appreciated.

piyushsingariya avatar Nov 23 '22 06:11 piyushsingariya

mentioning previous contributors of source-dynamodb @marcosmarxm @itaseskii @suhomud

piyushsingariya avatar Nov 23 '22 06:11 piyushsingariya

Hey @piyushsingariya thanks for your feedback and help in making this connector more robust. I have linked a PR that should fix the reported issue. Feel free to look into it.

itaseskii avatar Dec 07 '22 00:12 itaseskii

@itaseskii will this PR fix also issue with ttl as reserved keyword? We are currently facing issue with ttl:

Invalid ProjectionExpression: Attribute name is a reserved keyword; reserved keyword: ttl.

We have two choices:

  • wait for this PR to be merged and released or
  • go through code and change the attribute name and replace it in DynamoDb.

I would prefer the 1st option, if possible and if this wont take too long to release. I am not aware of the release cycles for airbyte

hrvacadaIM avatar Jan 18 '23 10:01 hrvacadaIM

@hrvacadaIM it should fix the issue. The PR will be merged really soon so I don't think you need to go with the second option.

itaseskii avatar Jan 19 '23 19:01 itaseskii

is there some type of ETA for this release?

rodrigo-montalegre avatar Feb 01 '23 07:02 rodrigo-montalegre

Sorry to bother @itaseskii, is there any progress on this? I noticed the PR is done and ready for review but it's failed some of it's CICD tests

Rhiyo avatar Feb 05 '23 07:02 Rhiyo

@itaseskii It looks like that PR has been stuck for about 2 weeks now. It seems there are some edge cases that are currently failing. Any status report on that end?

hrvacadaIM avatar Feb 06 '23 09:02 hrvacadaIM

@Rhiyo @hrvacadaIM I'll look into this again today.

itaseskii avatar Feb 06 '23 10:02 itaseskii

Hello everyone :wave: thanks for being patient! I have just pushed another commit in the PR which should address having reserved words and attribute names having special characters both in the filter and projection expression when performing scan queries on the database. Since the connector can't know how many attributes do you have satisfying that criteria I have included a new configuration property named reserved_attribute_names which will allow you to enter a comma separated list of attribute names that should be treated as being reserved words. The implementation was a bit tricky and full of edge-cases but hopefully it will cover every scenario.

itaseskii avatar Feb 07 '23 20:02 itaseskii

The fix PR for this has been merged into source-dynamodb:0.1.2

sh4sh avatar Mar 02 '23 16:03 sh4sh

Hi! I'm still getting the same error, connector version is source-dynamodb:0.1.2 @sh4sh @itaseskii

2023-05-30 13:41:06 [32mINFO[m i.a.c.i.LineGobbler(voidCall):149 - Checking if airbyte/source-dynamodb:0.1.2 exists...
2023-05-30 13:41:06 [32mINFO[m i.a.c.i.LineGobbler(voidCall):149 - airbyte/source-dynamodb:0.1.2 was found locally.
...
...
2023-05-30 13:41:21 [32mINFO[m i.a.w.g.DefaultReplicationWorker(getReplicationOutput):539 - failures: [ {
  "failureOrigin" : "source",
  "failureType" : "system_error",
  "internalMessage" : "software.amazon.awssdk.services.dynamodb.model.DynamoDbException: Invalid ProjectionExpression: Attribute name is a reserved keyword; reserved keyword: timestamp (Service: DynamoDb, Status Code: 400, Request ID: SKS2KKQ0CVL153RD2KPDSDMCLJVV4KQNSO5AEMVJF66Q9ASUAAJG)",
  "externalMessage" : "Something went wrong in the connector. See the logs for more details.",
  "metadata" : {
    "attemptNumber" : 0,
    "jobId" : 5187,
    "from_trace_message" : true,
    "connector_command" : "read"
  },
  "stacktrace" : "software.amazon.awssdk.services.dynamodb.model.DynamoDbException: Invalid ProjectionExpression: Attribute name is a reserved keyword; reserved keyword: timestamp (Service: DynamoDb, Status Code: 400, Request ID: SKS2KKQ0CVL153RD2KPDSDMCLJVV4KQNSO5AEMVJF66Q9ASUAAJG)\n\tat software.amazon.awssdk.core.internal.http.CombinedResponseHandler.handleErrorResponse(CombinedResponseHandler.java:125)\n\tat software.amazon.awssdk.core.internal.http.CombinedResponseHandler.handleResponse(CombinedResponseHandler.java:82)\n\tat software.amazon.awssdk.core.internal.http.CombinedResponseHandler.handle(CombinedResponseHandler.java:60)\n\tat software.amazon.awssdk.core.internal.http.CombinedResponseHandler.handle(CombinedResponseHandler.java:41)\n\tat software.amazon.awssdk.core.internal.http.pipeline.stages.HandleResponseStage.execute(HandleResponseStage.java:40)\n\tat software.amazon.awssdk.core.internal.http.pipeline.stages.HandleResponseStage.execute(HandleResponseStage.java:30)\n\tat software.amazon.awssdk.core.internal.http.pipeline.RequestPipelineBuilder$ComposingRequestPipelineStage.execute(RequestPipelineBuilder.java:206)\n\tat software.amazon.awssdk.core.internal.http.pipeline.stages.ApiCallAttemptTimeoutTrackingStage.execute(ApiCallAttemptTimeoutTrackingStage.java:73)\n\tat software.amazon.awssdk.core.internal.http.pipeline.stages.ApiCallAttemptTimeoutTrackingStage.execute(ApiCallAttemptTimeoutTrackingStage.java:42)\n\tat software.amazon.awssdk.core.internal.http.pipeline.stages.TimeoutExceptionHandlingStage.execute(TimeoutExceptionHandlingStage.java:78)\n\tat software.amazon.awssdk.core.internal.http.pipeline.stages.TimeoutExceptionHandlingStage.execute(TimeoutExceptionHandlingStage.java:40)\n\tat software.amazon.awssdk.core.internal.http.pipeline.stages.ApiCallAttemptMetricCollectionStage.execute(ApiCallAttemptMetricCollectionStage.java:50)\n\tat software.amazon.awssdk.core.internal.http.pipeline.stages.ApiCallAttemptMetricCollectionStage.execute(ApiCallAttemptMetricCollectionStage.java:36)\n\tat software.amazon.awssdk.core.internal.http.pipeline.stages.RetryableStage.execute(RetryableStage.java:81)\n\tat software.amazon.awssdk.core.internal.http.pipeline.stages.RetryableStage.execute(RetryableStage.java:36)\n\tat software.amazon.awssdk.core.internal.http.pipeline.RequestPipelineBuilder$ComposingRequestPipelineStage.execute(RequestPipelineBuilder.java:206)\n\tat software.amazon.awssdk.core.internal.http.StreamManagingStage.execute(StreamManagingStage.java:56)\n\tat software.amazon.awssdk.core.internal.http.StreamManagingStage.execute(StreamManagingStage.java:36)\n\tat software.amazon.awssdk.core.internal.http.pipeline.stages.ApiCallTimeoutTrackingStage.executeWithTimer(ApiCallTimeoutTrackingStage.java:80)\n\tat software.amazon.awssdk.core.internal.http.pipeline.stages.ApiCallTimeoutTrackingStage.execute(ApiCallTimeoutTrackingStage.java:60)\n\tat software.amazon.awssdk.core.internal.http.pipeline.stages.ApiCallTimeoutTrackingStage.execute(ApiCallTimeoutTrackingStage.java:42)\n\tat software.amazon.awssdk.core.internal.http.pipeline.stages.ApiCallMetricCollectionStage.execute(ApiCallMetricCollectionStage.java:48)\n\tat software.amazon.awssdk.core.internal.http.pipeline.stages.ApiCallMetricCollectionStage.execute(ApiCallMetricCollectionStage.java:31)\n\tat software.amazon.awssdk.core.internal.http.pipeline.RequestPipelineBuilder$ComposingRequestPipelineStage.execute(RequestPipelineBuilder.java:206)\n\tat software.amazon.awssdk.core.internal.http.pipeline.RequestPipelineBuilder$ComposingRequestPipelineStage.execute(RequestPipelineBuilder.java:206)\n\tat software.amazon.awssdk.core.internal.http.pipeline.stages.ExecutionFailureExceptionReportingStage.execute(ExecutionFailureExceptionReportingStage.java:37)\n\tat software.amazon.awssdk.core.internal.http.pipeline.stages.ExecutionFailureExceptionReportingStage.execute(ExecutionFailureExceptionReportingStage.java:26)\n\tat software.amazon.awssdk.core.internal.http.AmazonSyncHttpClient$RequestExecutionBuilderImpl.execute(AmazonSyncHttpClient.java:193)\n\tat software.amazon.awssdk.core.internal.handler.BaseSyncClientHandler.invoke(BaseSyncClientHandler.java:103)\n\tat software.amazon.awssdk.core.internal.handler.BaseSyncClientHandler.doExecute(BaseSyncClientHandler.java:171)\n\tat software.amazon.awssdk.core.internal.handler.BaseSyncClientHandler.lambda$execute$1(BaseSyncClientHandler.java:82)\n\tat software.amazon.awssdk.core.internal.handler.BaseSyncClientHandler.measureApiCallSuccess(BaseSyncClientHandler.java:179)\n\tat software.amazon.awssdk.core.internal.handler.BaseSyncClientHandler.execute(BaseSyncClientHandler.java:76)\n\tat software.amazon.awssdk.core.client.handler.SdkSyncClientHandler.execute(SdkSyncClientHandler.java:45)\n\tat software.amazon.awssdk.awscore.client.handler.AwsSyncClientHandler.execute(AwsSyncClientHandler.java:56)\n\tat software.amazon.awssdk.services.dynamodb.DefaultDynamoDbClient.scan(DefaultDynamoDbClient.java:4876)\n\tat software.amazon.awssdk.services.dynamodb.paginators.ScanIterable$ScanResponseFetcher.nextPage(ScanIterable.java:134)\n\tat software.amazon.awssdk.services.dynamodb.paginators.ScanIterable$ScanResponseFetcher.nextPage(ScanIterable.java:125)\n\tat software.amazon.awssdk.core.pagination.sync.PaginatedResponsesIterator.next(PaginatedResponsesIterator.java:58)\n\tat io.airbyte.integrations.source.dynamodb.DynamodbOperations.scanTable(DynamodbOperations.java:170)\n\tat io.airbyte.integrations.source.dynamodb.DynamodbSource.scanFullRefresh(DynamodbSource.java:185)\n\tat io.airbyte.integrations.source.dynamodb.DynamodbSource.lambda$read$1(DynamodbSource.java:111)\n\tat java.base/java.util.stream.ReferencePipeline$3$1.accept(ReferencePipeline.java:197)\n\tat java.base/java.util.ArrayList$ArrayListSpliterator.forEachRemaining(ArrayList.java:1625)\n\tat java.base/java.util.stream.AbstractPipeline.copyInto(AbstractPipeline.java:509)\n\tat java.base/java.util.stream.AbstractPipeline.wrapAndCopyInto(AbstractPipeline.java:499)\n\tat java.base/java.util.stream.AbstractPipeline.evaluate(AbstractPipeline.java:575)\n\tat java.base/java.util.stream.AbstractPipeline.evaluateToArrayNode(AbstractPipeline.java:260)\n\tat java.base/java.util.stream.ReferencePipeline.toArray(ReferencePipeline.java:616)\n\tat java.base/java.util.stream.ReferencePipeline.toArray(ReferencePipeline.java:622)\n\tat java.base/java.util.stream.ReferencePipeline.toList(ReferencePipeline.java:627)\n\tat io.airbyte.integrations.source.dynamodb.DynamodbSource.read(DynamodbSource.java:113)\n\tat io.airbyte.integrations.base.IntegrationRunner.runInternal(IntegrationRunner.java:138)\n\tat io.airbyte.integrations.base.IntegrationRunner.run(IntegrationRunner.java:98)\n\tat io.airbyte.integrations.source.dynamodb.DynamodbSource.main(DynamodbSource.java:50)\n",

amit-evermile avatar May 30 '23 13:05 amit-evermile