create ECS service with service discovery in public dns namespace fails

[diablodale]

Summary

Unable to create ECS service with public dns namespace service discovery. Multiple failure scenarios.

Setup

• Ubuntu 18.04.3 x86_64
• ecs-cli version 1.17.0 (0301ac8) from the official AWS download
• go is not installed
• Create and have already working a public DNS zone in Route53. In the examples below, you will need to change this public DNS zone and its ID from the one I use below. Below I use the DNS zone electroglitch.com with the ID Z11DHWRTAZEXOD
• Create and have working your own task_role_arn and task_execution_role. Replace mine with your two roles in the below ecs-params.yml

~/.ecs/config

version: v1
default: frontend
clusters:
  frontend:
    cluster: frontend
    region: us-east-1
    default_launch_type: EC2

docker-compose.yml

version: '3'
services:
  coolthing:
    image: alpine:3.10
    command: tail -f /dev/null
    hostname: coolthing

ecs-params.yml

version: 1
task_definition:
  ecs_network_mode: bridge
  task_role_arn: ecs_task_frontend_hidale
  task_execution_role: ecs_agent_frontend_hidale
  task_size:
    mem_limit: 200
    cpu_limit: 512

run_params:
  task_placement:
    constraints:
      - type: distinctInstance

  service_discovery:
    container_name: coolthing
    container_port: 8123
    public_dns_namespace:
      name: electroglitch.com
      #name: electroglitch.com.
      #id: Z11DHWRTAZEXOD
    service_discovery_service:
      name: coolthing.internal
      description: cool thing created by the Amazon ECS CLI
      dns_config:
        type: SRV
        ttl: 60

Repo 1

1. Edit ecs-params.yml.
2. Replace the public_dns_namespace.name value with your Route53 DNS zone name without a trailing period
3. Run the following

   ecs-cli compose --file docker-compose.yml service up --cluster-config frontend --deployment-max-percent 100 --deployment-min-healthy-percent 0 --enable-service-discovery
   

Actual Result 1

INFO[0000] Using ECS task definition                     TaskDefinition="ecs-tutorial:63"
FATA[0001] Failed to find public namespace electroglitch.com

Expected Result 1

No error and a SRV entry to be inserted into my public DNS zone.

Repo 2

1. Edit ecs-params.yml.
2. Replace the public_dns_namespace.name value with your Route53 DNS zone name with a trailing period
3. Run the following

   ecs-cli compose --file docker-compose.yml service up --cluster-config frontend --deployment-max-percent 100 --deployment-min-healthy-percent 0 --enable-service-discovery
   

Actual Result 2

INFO[0000] Using ECS task definition                     TaskDefinition="ecs-tutorial:63"
FATA[0001] Failed to find public namespace electroglitch.com.

Expected Result 2

No error and a SRV entry to be inserted into my public DNS zone.

Repo 3

1. Edit ecs-params.yml.
2. Put a comment # at the start of the line public_dns_namespace.name line
3. Remove the leading # at the start of the line #id: Z11DHWRTAZEXOD
4. Replace this id with your Route53 zone id
5. Run the following

   ecs-cli compose --file docker-compose.yml service up --cluster-config frontend --deployment-max-percent 100 --deployment-min-healthy-percent 0 --enable-service-discovery
   

Actual Result 3

INFO[0000] Using ECS task definition                     TaskDefinition="ecs-tutorial:63"
INFO[0000] Using existing namespace Z11DHWRTAZEXOD      
INFO[0002] Waiting for the Service Discovery Service to be created... 
INFO[0002] Cloudformation stack status                   stackStatus=CREATE_IN_PROGRESS
ERRO[0033] Failure event                                 reason="Z11DHWRTAZEXOD (Service: AWSServiceDiscovery; Status Code: 400; Error Code: NamespaceNotFound; Request ID: 39b306de-a80f-48a1-92ad-f766672d8fcf)" resourceType="AWS::ServiceDiscovery::Service"
FATA[0033] Failed to find output ServiceDiscoveryServiceARN in stack amazon-ecs-cli-setup-service-discovery-service-frontend-ecs-tutorial

Expected Result 3

No error and a SRV entry to be inserted into my public DNS zone.

Workarounds

None known

[marcelkottmann]

Hello, is there any known workaround for this or any ecs cli version number that worked previously?

[diablodale]

Personally, I didn't find a workaround. I instead wrote my own SD engine w/ cloudwatch events and lambda that uses route53, EC2, and ECS apis. It fits my immediate needs (can't wait, gotta ship) while longer-term the AWS native SD solution can be sorted. I'm unclear if the problem is docs, lack of feature, bug in ecs-cli, or somewhere else. It was too open-ended for me to go on an unbounded bug hunt.

[marcelkottmann]

Thank you for your feedback. In the meantime I worked around this issue with a Route 53 ALIAS entry to an application loadbalancer and configured my ecs service as target group of this loadbalancer.

[aaron-splicer]

ok, I dug into this a bit and thought I'd share my findings. Hope this is the correct place to share this.

1. Got failure with $ecs-cli compose up --public-dns-namespace xyz.com. The trick was to manually create a namespace here: https://us-west-1.console.aws.amazon.com/cloudmap/home/namespaces?region=us-west-1. Once I did, then the public-dns-namespace switch worked.
2. Turns out that was a waste of time because per https://docs.aws.amazon.com/AmazonECS/latest/developerguide/service-discovery.html, "The DNS records created for a service discovery service always register with the private IP address for the task, rather than the public IP address, even when public namespaces are used."
3. So, I then successfully used (fired up outta-the-box) a lamba from here: https://github.com/foby/aws-ecs-public-dns, https://github.com/foby/aws-ecs-public-dns (thanks Andreas Pasch). Note this only works for hosted zones created manually and not auto-created by "Cloud Map"