localstack
InvalidParameterValue when using a SQS FIFO queue as a event bridge pipe target.
Is there an existing issue for this?
- [X] I have searched the existing issues
Current Behavior
Create an event bridge pipe with the following:
- Source: DynamoDB
- Target: SQS FIFO Queue
Create a simple Lambda to consume the message.
Resources:
OutBoxSqsQueue: Type: AWS::SQS::Queue Properties: VisibilityTimeout: 600 QueueName: !Sub ${OutboxFifoQueue}.fifo FifoQueue: true ContentBasedDeduplication: true
OrdersTable: Type: AWS::DynamoDB::Table Properties: BillingMode: PAY_PER_REQUEST AttributeDefinitions: - AttributeName: id AttributeType: S - AttributeName: order_id AttributeType: S KeySchema: - AttributeName: id KeyType: HASH - AttributeName: order_id KeyType: RANGE TableName: !Sub ${OrdersTableName} StreamSpecification: StreamViewType: NEW_AND_OLD_IMAGES
EventBridgePipe: Type: AWS::Pipes::Pipe Properties: Name: dynamodb-pipe Description: "Pipe to connect DynamoDB Stream to AWS SQS FIFO Queue" RoleArn: !GetAtt PipeRole.Arn Source: !GetAtt OrdersTable.StreamArn SourceParameters: DynamoDBStreamParameters: StartingPosition: LATEST BatchSize: 1 Target: !GetAtt OutBoxSqsQueue.Arn TargetParameters: SqsQueueParameters: MessageGroupId: $.dynamodb.NewImage.id.S
Provisioned Pipe:
Arn: arn:aws:pipes:eu-west-2:000000000000:pipe/dynamodb-pipe CreationTime: '2024-05-30T11:10:50.433439+02:00' CurrentState: RUNNING DesiredState: RUNNING EnrichmentParameters: {} LastModifiedTime: '2024-05-30T11:10:50.433439+02:00' Name: dynamodb-pipe RoleArn: arn:aws:iam::000000000000:role/local-cf-stack-PipeRole-f5161bbc Source: arn:aws:dynamodb:eu-west-2:000000000000:table/orders-table/stream/2024-05-30T09:10:36.723 SourceParameters: DynamoDBStreamParameters: BatchSize: 1 StartingPosition: LATEST StateReason: USER_INITIATED Tags: {} Target: arn:aws:sqs:eu-west-2:000000000000:outbox-fifo-queue.fifo TargetParameters: SqsQueueParameters: MessageGroupId: $.dynamodb.NewImage.id.S
Insert data into the table.
ERROR:
File "/opt/code/localstack/.venv/lib/python3.11/site-packages/botocore/client.py", line 565, in _api_call localstack_file_processor_main | return self._make_api_call(operation_name, kwargs) localstack_file_processor_main | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ localstack_file_processor_main | File "/opt/code/localstack/.venv/lib/python3.11/site-packages/botocore/client.py", line 1021, in _make_api_call localstack_file_processor_main | raise error_class(parsed_response, operation_name) localstack_file_processor_main | botocore.exceptions.ClientError: An error occurred (InvalidParameterValue) when calling the SendMessageBatch operation: The request must contain the parameter MessageGroupId. localstack_file_processor_main | 2024-05-30T09:15:38.268 ERROR --- [functhread93] l.s.pipes.pipe_worker : Error while handling pipe events [{'eventID': '6851506d', 'eventName': 'INSERT', 'dynamodb': {'ApproximateCreationDateTime': 1717060537, 'Keys': {'id': {'S': '223ab415-7c77-4d3b-90fc-1dbaa094b341'}, 'order_id': {'S': '6ed42a80-4496-42fd-93a0-3c862bfc5040'}}, 'NewImage': {'id': {'S': '223ab415-7c77-4d3b-90fc-1dbaa094b341'}, 'order_id': {'S': '6ed42a80-4496-42fd-93a0-3c862bfc5040'}, 'accountNumber': {'S': '5100000005'}, 'entryNumber': {'S': '4'}, 'type': {'S': 'DEBIT'}}, 'SequenceNumber': '49652524873082293128637712948189075213397080096779534338', 'SizeBytes': 187, 'StreamViewType': 'NEW_AND_OLD_IMAGES'}, 'eventSourceARN': 'arn:aws:dynamodb:eu-west-2:000000000000:table/orders-table/stream/2024-05-30T09:10:36.723', 'eventSource': 'aws:dynamodb', 'awsRegion': 'eu-west-2', 'eventVersion': '1.1'}] for pipe with execution id 868f88eb-f89d-49c5-b6ae-1e0d4afd73ec localstack_file_processor_main | Traceback (most recent call last): localstack_file_processor_main | File "/opt/code/localstack/.venv/lib/python3.11/site-packages/localstack_ext/services/pipes/pipe_worker.py.enc", line 53, in process_events_batch localstack_file_processor_main | A.process_target_stage(C);A.logger.log(messageType='ExecutionSucceeded',logLevel=_D);return True localstack_file_processor_main | ^^^^^^^^^^^^^^^^^^^^^^^^^ localstack_file_processor_main | File "/opt/code/localstack/.venv/lib/python3.11/site-packages/localstack_ext/services/pipes/pipe_worker.py.enc", line 72, in process_target_stage localstack_file_processor_main | A.logger.log(messageType='TargetInvocationStarted',logLevel=_A);C=A.sender.send_events(D) localstack_file_processor_main | ^^^^^^^^^^^^^^^^^^^^^^^ localstack_file_processor_main | File "/opt/code/localstack/.venv/lib/python3.11/site-packages/localstack_ext/services/pipes/senders/sqs_sender.py.enc", line 9, in send_events localstack_file_processor_main | for B in batched(events,MAX_MESSAGES_PER_BATCH):C=[{'Id':str(B),'MessageBody':A if isinstance(A,str)else json.dumps(A)}for(B,A)in enumerate(B)];A.target_client.send_message_batch(QueueUrl=A.target_queue_url,Entries=C)
Expected Behavior
The lambda should consume the message successfully. If I provision this stack to AWS, it works 100%.
How are you starting LocalStack?
With a docker-compose file
Steps To Reproduce
How are you starting localstack (e.g., bin/localstack command, arguments, or docker-compose.yml)
docker-compose up
version: '3.8'
services: localstack: container_name: "${LOCALSTACK_DOCKER_NAME-localstack_file_processor_main}" # image: localstack/localstack:latest image: localstack/localstack-pro networks: - localstack-net ports: - "4566:4566" - "4571:4571" - "${PORT_WEB_UI-8080}:${PORT_WEB_UI-8080}" environment: - DEBUG=1 - DOCKER_HOST=unix:///var/run/docker.sock - LAMBDA_DOCKER_NETWORK=localstack-net - SQS_ENDPOINT_STRATEGY=domain - LOCALSTACK_AUTH_TOKEN="ls-" - LAMBDA_RUNTIME_ENVIRONMENT_TIMEOUT=200 - LAMBDA_EXECUTOR=${LAMBDA_EXECUTOR- docker-reuse} tmpfs: - /var/lib/localstackcl:exec,mode=600 volumes: - "${LOCALSTACK_VOLUME_DIR:-./volume}:/var/lib/localstack" - "/var/run/docker.sock:/var/run/docker.sock" networks: localstack-net: external: false driver: bridge name: localstack-net
Client commands (e.g., AWS SDK code snippet, or sequence of "awslocal" commands)
Resources provision via CF.
OutBoxSqsQueue: Type: AWS::SQS::Queue Properties: VisibilityTimeout: 600 QueueName: !Sub ${OutboxFifoQueue}.fifo FifoQueue: true ContentBasedDeduplication: true
OrdersTable: Type: AWS::DynamoDB::Table Properties: BillingMode: PAY_PER_REQUEST AttributeDefinitions: - AttributeName: id AttributeType: S - AttributeName: order_id AttributeType: S KeySchema: - AttributeName: id KeyType: HASH - AttributeName: order_id KeyType: RANGE TableName: !Sub ${OrdersTableName} StreamSpecification: StreamViewType: NEW_AND_OLD_IMAGES
EventBridgePipe: Type: AWS::Pipes::Pipe Properties: Name: dynamodb-pipe Description: "Pipe to connect DynamoDB Stream to AWS SQS FIFO Queue" RoleArn: !GetAtt PipeRole.Arn Source: !GetAtt OrdersTable.StreamArn SourceParameters: DynamoDBStreamParameters: StartingPosition: LATEST BatchSize: 1 Target: !GetAtt OutBoxSqsQueue.Arn TargetParameters: SqsQueueParameters: MessageGroupId: $.dynamodb.NewImage.id.S
Environment
- OS:
ProductName: macOS
ProductVersion: 14.4.1
BuildVersion: 23E224
- LocalStack:
LocalStack version: localstack/localstack-pro
Anything else?
No response
Welcome to LocalStack! Thanks for reporting your first issue and our team will be working towards fixing the issue for you or reach out for more background information. We recommend joining our Slack Community for real-time help and drop a message to LocalStack Pro Support if you are a Pro user! If you are willing to contribute towards fixing this issue, please have a look at our contributing guidelines and our contributing guide.
Hello 👋! It looks like this issue hasn’t been active in longer than five months. We encourage you to check if this is still an issue in the latest release. In the absence of more information, we will be closing this issue soon. If you find that this is still a problem, please feel free to provide a comment or upvote with a reaction on the initial post to prevent automatic closure. If the issue is already closed, please feel free to open a new one.