bug: `list index out of range` when creating an ECS service through terraform

[alexlokshin-czi]

Is there an existing issue for this?

• [X] I have searched the existing issues

Current Behavior

I'm running localstack pro in Docker, trying to create an ECS service:

resource aws_ecs_service service {
  cluster         = var.cluster
  desired_count   = 1
  task_definition = aws_ecs_task_definition.task_definition.id
  launch_type     = "EC2"
  name            = var.name
  load_balancer {
    container_name   = "web"
    container_port   = var.service_port
    target_group_arn = aws_lb_target_group.target_group.id
  }
  network_configuration {
    security_groups  = var.security_groups
    subnets          = var.subnets
    assign_public_ip = false
  }

  wait_for_steady_state = var.wait_for_steady_state
}

Terraform keeps trying to create the service, but debug logs are filled with errors like these:

2022-08-09T21:16:08.480 ERROR --- [   asgi_gw_5] l.aws.handlers.logging     : exception during call chain
Traceback (most recent call last):
  File "/opt/code/localstack/localstack/aws/chain.py", line 57, in handle
    handler(self, self.context, response)
  File "/opt/code/localstack/localstack/aws/handlers/service.py", line 121, in __call__
    handler(chain, context, response)
  File "/opt/code/localstack/localstack/aws/handlers/service.py", line 91, in __call__
    skeleton_response = self.skeleton.invoke(context)
  File "/opt/code/localstack/localstack/aws/skeleton.py", line 153, in invoke
    return self.dispatch_request(context, instance)
  File "/opt/code/localstack/localstack/aws/skeleton.py", line 165, in dispatch_request
    result = handler(context, instance) or {}
  File "/opt/code/localstack/localstack/aws/forwarder.py", line 55, in _call
    return handler(context, req)
  File "/opt/code/localstack/localstack/aws/skeleton.py", line 117, in __call__
    return self.fn(*args, **kwargs)
  File "/opt/code/localstack/localstack/aws/api/core.py", line 115, in operation_marker
    return fn(*args, **kwargs)
  File "/opt/code/localstack/.venv/lib/python3.10/site-packages/localstack_ext/services/ecs/provider.py.enc", line 83, in create_service
  File "/opt/code/localstack/.venv/lib/python3.10/site-packages/botocore/client.py", line 508, in _api_call
    return self._make_api_call(operation_name, kwargs)
  File "/opt/code/localstack/.venv/lib/python3.10/site-packages/botocore/client.py", line 915, in _make_api_call
    raise error_class(parsed_response, operation_name)
botocore.exceptions.ClientError: An error occurred (InternalError) when calling the RunTask operation (reached max retries: 4): exception while calling ecs.RunTask: Traceback (most recent call last):
  File "/opt/code/localstack/localstack/aws/chain.py", line 57, in handle
    handler(self, self.context, response)
  File "/opt/code/localstack/localstack/aws/handlers/service.py", line 121, in __call__
    handler(chain, context, response)
  File "/opt/code/localstack/localstack/aws/handlers/service.py", line 91, in __call__
    skeleton_response = self.skeleton.invoke(context)
  File "/opt/code/localstack/localstack/aws/skeleton.py", line 153, in invoke
    return self.dispatch_request(context, instance)
  File "/opt/code/localstack/localstack/aws/skeleton.py", line 165, in dispatch_request
    result = handler(context, instance) or {}
  File "/opt/code/localstack/localstack/aws/forwarder.py", line 55, in _call
    return handler(context, req)
  File "/opt/code/localstack/localstack/aws/skeleton.py", line 117, in __call__
    return self.fn(*args, **kwargs)
  File "/opt/code/localstack/localstack/aws/api/core.py", line 115, in operation_marker
    return fn(*args, **kwargs)
  File "/opt/code/localstack/.venv/lib/python3.10/site-packages/localstack_ext/services/ecs/provider.py.enc", line 73, in run_task
IndexError: list index out of range
2022-08-09T21:16:08.482  INFO --- [   asgi_gw_5] localstack.request.aws     : AWS ecs.CreateService => 500 (InternalError)
2022-08-09T21:16:08.808 DEBUG --- [  asgi_gw_11] l.services.ecs.provider    : Running ECS task 'arn:aws:ecs:us-west-2:000000000000:task-definition/localstack-backend:1' through create service action
2022-08-09T21:16:08.816 ERROR --- [   asgi_gw_1] l.aws.handlers.logging     : exception during call chain
Traceback (most recent call last):
  File "/opt/code/localstack/localstack/aws/chain.py", line 57, in handle
    handler(self, self.context, response)
  File "/opt/code/localstack/localstack/aws/handlers/service.py", line 121, in __call__
    handler(chain, context, response)
  File "/opt/code/localstack/localstack/aws/handlers/service.py", line 91, in __call__
    skeleton_response = self.skeleton.invoke(context)
  File "/opt/code/localstack/localstack/aws/skeleton.py", line 153, in invoke
    return self.dispatch_request(context, instance)
  File "/opt/code/localstack/localstack/aws/skeleton.py", line 165, in dispatch_request
    result = handler(context, instance) or {}
  File "/opt/code/localstack/localstack/aws/forwarder.py", line 55, in _call
    return handler(context, req)
  File "/opt/code/localstack/localstack/aws/skeleton.py", line 117, in __call__
    return self.fn(*args, **kwargs)
  File "/opt/code/localstack/localstack/aws/api/core.py", line 115, in operation_marker
    return fn(*args, **kwargs)
  File "/opt/code/localstack/.venv/lib/python3.10/site-packages/localstack_ext/services/ecs/provider.py.enc", line 73, in run_task
IndexError: list index out of rang

Expected Behavior

ECS Service is successfully created.

How are you starting LocalStack?

With a docker run command

Steps To Reproduce

How are you starting localstack (e.g., bin/localstack command, arguments, or docker-compose.yml)

docker run --rm -it -p 4566:4566 -p 4510-4559:4510-4559 -p 443:443 --env LOCALSTACK_API_KEY=xxxxxxxx --env DEBUG=1 localstack/localstack:1.0.3-arm64

Client commands (e.g., AWS SDK code snippet, or sequence of "awslocal" commands)

terraform apply

Environment

- OS: MacOS X 12.5 (M1)
- LocalStack: 1.0.3-arm64

Anything else?

No response

[localstack-bot]

Welcome to LocalStack! Thanks for reporting your first issue and our team will be working towards fixing the issue for you or reach out for more background information. We recommend joining our Slack Community for real-time help and drop a message to LocalStack Pro Support if you are a Pro user! If you are willing to contribute towards fixing this issue, please have a look at our contributing guidelines and our developer guide.

[alexlokshin-czi]

Here's the offending request from the trace log:

AWS ecs.CreateService => 500 (InternalError); CreateServiceRequest({'cluster': 'arn:aws:ecs:us-west-2:000000000000:cluster/happy-rdev', 'serviceName': 'localstack3-backend', 'taskDefinition': 'dp-rdev-localstack3-backend', 'loadBalancers': [{'targetGroupArn': 'arn:aws:elasticloadbalancing:us-west-2:000000000000:targetgroup/tf-20220809224528343000000002/50dc6c495c0c9188', 'containerName': 'web', 'containerPort': 5000}], 'desiredCount': 1, 'clientToken': 'terraform-20220809224533911600000003', 'launchType': 'EC2', 'capacityProviderStrategy': [], 'deploymentConfiguration': {'maximumPercent': 200, 'minimumHealthyPercent': 100}, 'networkConfiguration': {'awsvpcConfiguration': {'subnets': ['subnet-67e63da1', 'subnet-2cb59776', 'subnet-cdfc0a08'], 'securityGroups': ['sg-0cb68ad56f8572ed9'], 'assignPublicIp': 'DISABLED'}}, 'schedulingStrategy': 'REPLICA', 'enableECSManagedTags': False, 'enableExecuteCommand': False}, headers={'Host': 'localhost:4566', 'User-Agent': 'APN/1.0 HashiCorp/1.0 Terraform/0.13.5 (+https://www.terraform.io) terraform-provider-aws/dev (+https://registry.terraform.io/providers/hashicorp/aws) aws-sdk-go/1.44.67 (go1.18.4; darwin; amd64)', 'Content-Length': '813', 'Authorization': 'AWS4-HMAC-SHA256 Credential=test/20220809/us-west-2/ecs/aws4_request, SignedHeaders=content-length;content-type;host;x-amz-date;x-amz-target, Signature=00f7a2fd6e3650cb21951e8bd945efaa038b58449872db0a201affed54f8273c', 'Content-Type': 'application/x-amz-json-1.1', 'X-Amz-Date': '20220809T224622Z', 'X-Amz-Target': 'AmazonEC2ContainerServiceV20141113.CreateService', 'Accept-Encoding': 'gzip', 'x-localstack-tgt-api': 'ecs'}); InternalError(exception while calling ecs.CreateService: Traceback (most recent call last):
  File "/opt/code/localstack/localstack/aws/chain.py", line 57, in handle
    handler(self, self.context, response)
  File "/opt/code/localstack/localstack/aws/handlers/service.py", line 121, in __call__
    handler(chain, context, response)
  File "/opt/code/localstack/localstack/aws/handlers/service.py", line 91, in __call__
    skeleton_response = self.skeleton.invoke(context)
  File "/opt/code/localstack/localstack/aws/skeleton.py", line 153, in invoke
    return self.dispatch_request(context, instance)
  File "/opt/code/localstack/localstack/aws/skeleton.py", line 165, in dispatch_request
    result = handler(context, instance) or {}
  File "/opt/code/localstack/localstack/aws/forwarder.py", line 55, in _call
    return handler(context, req)
  File "/opt/code/localstack/localstack/aws/skeleton.py", line 117, in __call__
    return self.fn(*args, **kwargs)
  File "/opt/code/localstack/localstack/aws/api/core.py", line 115, in operation_marker
    return fn(*args, **kwargs)
  File "/opt/code/localstack/.venv/lib/python3.10/site-packages/localstack_ext/services/ecs/provider.py.enc", line 83, in create_service
  File "/opt/code/localstack/.venv/lib/python3.10/site-packages/botocore/client.py", line 508, in _api_call
    return self._make_api_call(operation_name, kwargs)
  File "/opt/code/localstack/.venv/lib/python3.10/site-packages/botocore/client.py", line 915, in _make_api_call
    raise error_class(parsed_response, operation_name)

For visibility, here's the request:

{
	'cluster': 'arn:aws:ecs:us-west-2:000000000000:cluster/happy-rdev',
	'serviceName': 'localstack3-backend',
	'taskDefinition': 'dp-rdev-localstack3-backend',
	'loadBalancers': [{
		'targetGroupArn': 'arn:aws:elasticloadbalancing:us-west-2:000000000000:targetgroup/tf-20220809224528343000000002/50dc6c495c0c9188',
		'containerName': 'web',
		'containerPort': 5000
	}],
	'desiredCount': 1,
	'clientToken': 'terraform-20220809224533911600000003',
	'launchType': 'EC2',
	'capacityProviderStrategy': [],
	'deploymentConfiguration': {
		'maximumPercent': 200,
		'minimumHealthyPercent': 100
	},
	'networkConfiguration': {
		'awsvpcConfiguration': {
			'subnets': ['subnet-67e63da1', 'subnet-2cb59776', 'subnet-cdfc0a08'],
			'securityGroups': ['sg-0cb68ad56f8572ed9'],
			'assignPublicIp': 'DISABLED'
		}
	},
	'schedulingStrategy': 'REPLICA',
	'enableECSManagedTags': False,
	'enableExecuteCommand': False
} 
  
headers = {
	'Host': 'localhost:4566',
	'User-Agent': 'APN/1.0 HashiCorp/1.0 Terraform/0.13.5 (+https://www.terraform.io) terraform-provider-aws/dev (+https://registry.terraform.io/providers/hashicorp/aws) aws-sdk-go/1.44.67 (go1.18.4; darwin; amd64)',
	'Content-Length': '813',
	'Authorization': 'AWS4-HMAC-SHA256 Credential=test/20220809/us-west-2/ecs/aws4_request, SignedHeaders=content-length;content-type;host;x-amz-date;x-amz-target, Signature=00f7a2fd6e3650cb21951e8bd945efaa038b58449872db0a201affed54f8273c',
	'Content-Type': 'application/x-amz-json-1.1',
	'X-Amz-Date': '20220809T224622Z',
	'X-Amz-Target': 'AmazonEC2ContainerServiceV20141113.CreateService',
	'Accept-Encoding': 'gzip',
	'x-localstack-tgt-api': 'ecs'
}

Could it be related to an empty capacityProviderStrategy ?

[alexlokshin-czi]

Based on what I see, this can be related to the amount of memory requested through

resource aws_ecs_task_definition task_definition {
   ...
   memory = 8192
   ...
}

If I reduce it to 512, error doesn't manifest.

[whummer]

Hi @alexlokshin-czi , thanks for reporting, and apologies for the long delay. This is indeed related to the memory requested by the task - we have in the meantime pushed a few enhancements to have better error reporting in this case (i.e., should no longer raise the list index out of range error).

Can you please pull the latest Docker image and give it another try? Please keep us posted on how it goes - we'll be happy to look into it in more detail if the problem persists.. 👍 Thanks for your help!

[whummer]

Closing as resolved - please let us know if the problem persists.. Thanks!