How to integrate service discovery with AWS ECS

[saschathiergart]

I am trying to get gobetween running on AWS ECS. My ECS services run tasks in bridge mode using the service discovery feature provided by AWS. In my gobetween config, i have

[servers.tftp.discovery]
kind = "srv"
srv_lookup_server = "core-test:53"   # (required) "<host:port>"
srv_lookup_pattern = "tftp.core-test."   # (required) lookup service

Doing dig tftp.core-test SRV in the running container instance gives me:

 ; <<>> DiG 9.11.3-1ubuntu1.3-Ubuntu <<>> tftp.core-test SRV
;; global options: +cmd
;; Got answer:
;; ->>HEADER<<- opcode: QUERY, status: NOERROR, id: 26016
;; flags: qr rd ra; QUERY: 1, ANSWER: 1, AUTHORITY: 4, ADDITIONAL: 1

;; QUESTION SECTION:
;tftp.core-test.			IN	SRV

;; ANSWER SECTION:
tftp.core-test.		60	IN	SRV	1 1 69 66101ea210104b18a7c77a5e83307807.tftp.core-test.

;; AUTHORITY SECTION:
core-test.		172800	IN	NS	ns-1024.awsdns-00.org.
core-test.		172800	IN	NS	ns-1536.awsdns-00.co.uk.
core-test.		172800	IN	NS	ns-512.awsdns-00.net.
core-test.		172800	IN	NS	ns-0.awsdns-00.com.

;; ADDITIONAL SECTION:
66101ea210104b18a7c77a5e83307807.tftp.core-test. 60 IN A 172.33.6.168

;; Query time: 3 msec
;; SERVER: 172.33.0.2#53(172.33.0.2)
;; WHEN: Wed Feb 20 14:51:42 UTC 2019
;; MSG SIZE  rcvd: 286

However, in the log i can see:

2019-02-20 15:45:202019-02-20 14:45:20 [ERROR] (discovery): srv error dial udp: lookup core-test on 172.33.0.2:53: no such host retrying in 2s
2019-02-20 15:45:202019-02-20 14:45:20 [INFO ] (srvFetch): Fetching core-test:53 tftp.core-test.

Maybe i don't get how to properly set the srv_lookup_server and srv_lookup_pattern properties. Any idea what i am missing out here?

[nickdoikov]

@pritstift

1. try to delete srv_dns_protocol = "udp"

seems like you can`t connect to dns via udp

2. please check api for backend server pools , or enable logging verbosity and check does discovery was successful

config looks fine for me .

[saschathiergart]

@nickdoikov Thanks for your response.

Regarding 1.: I have the srv_dns_protocol ="udp" nowhere in my config. Regarding 2.: How do i enable verbose logging in gobetween. I could not find any info in the docs. The -h option doesn't tell me how either.

[saschathiergart]

@nickdoikov Apparently it works if i use the IP address of the DNS server instead of its domain name:

srv_lookup_server = "172.33.0.2:53". Is this behaviour intended? It seems like gobetween does not or cannot resolve the name of the DNS server.

[ekini]

I had that issue too and the workaround that worked for me was installing dnsmasq and setting srv_lookup_server = "127.0.0.1:53" in gobetween.

It's a bit weird on AWS, so that

ping 66101ea210104b18a7c77a5e83307807.tftp.core-test.

will say Name or service not known, but host and dig can resolve successfully.

[MatthewPattell]

Hello, I trying run gobetween in EC2 (via docker). I want forward requests to domains and ports (in SRV record).

[servers.sample]
bind = "localhost:3000"
protocol = "tcp"
balance = "weight"

  [servers.sample.discovery]
  failpolicy = "keeplast"
  kind = "srv"
  srv_lookup_server = "127.0.0.1:53"
  srv_lookup_pattern = "my-domain.local."

Log:

lb          | gobetween v0.8.0+snapshot
lb          | 2019-08-14 07:25:50 [INFO ] (api): API disabled
lb          | 2019-08-14 07:25:50 [INFO ] (manager): Initializing...
lb          | 2019-08-14 07:25:50 [INFO ] (metrics): Metrics disabled
lb          | 2019-08-14 07:25:50 [INFO ] (server): Creating 'sample': localhost:3000 weight srv none
lb          | 2019-08-14 07:25:50 [INFO ] (scheduler): Starting scheduler sample
lb          | 2019-08-14 07:25:50 [INFO ] (srvFetch): Fetching 127.0.0.1:53 my-domain.local.
lb          | 2019-08-14 07:25:50 [ERROR] (discovery): srv error read udp 127.0.0.1:51221->127.0.0.1:53: read: connection refused retrying in 2s
lb          | 2019-08-14 07:25:50 [INFO ] (discovery): Applying failpolicy keeplast
lb          | 2019-08-14 07:25:50 [INFO ] (manager): Initialized
lb          | 2019-08-14 07:25:52 [INFO ] (srvFetch): Fetching 127.0.0.1:53 my-domain.local.
lb          | 2019-08-14 07:25:52 [ERROR] (discovery): srv error read udp 127.0.0.1:56556->127.0.0.1:53: read: connection refused retrying i

What am I doing wrong?

[illarion]

@MatthewPattell Looks like your dns server is not listening on 127.0.0.1:53. Check the output of netstat: sudo netstat -lnu

[MatthewPattell]

@illarion thanks for the answer. Here is the result sudo netstat -lnu:

Active Internet connections (only servers)
Proto Recv-Q Send-Q Local Address               Foreign Address             State      
udp        0      0 10.0.5.134:123              0.0.0.0:*                               
udp        0      0 127.0.0.1:123               0.0.0.0:*                               
udp        0      0 0.0.0.0:123                 0.0.0.0:*                               
udp        0      0 0.0.0.0:68                  0.0.0.0:*                               
udp        0      0 fe80::1e:50ff:fe72:baa8:546 :::*                                    

Maybe I didn’t understand something correctly. SRV records creates in Route 53 via ECS Service Discovery. How to specify in srv_lookup_server Route 53 DNS?

I try use NS record srv_lookup_server = ns-***.awsdns-**.co.uk:53:

lb          | 2019-08-14 11:23:24 [INFO ] (srvFetch): Fetching ns-***.awsdns-**.co.uk:53 my-domain.local.
lb          | 2019-08-14 11:23:24 [INFO ] (manager): Initialized
lb          | 2019-08-14 11:23:24 [WARNI] (srvFetch): Empty response fromns-***.awsdns-**.co.uk:53my-domain.local.

Output nslookup nslookup -type=srv my-domain.local:

Server:		10.0.0.2
Address:	10.0.0.2#53

Non-authoritative answer:
my-domain.local	service = 1 1 32782 c763b65e-938e-410b-ab20-99e01e4bdfb0.my-domain.local.

Curl output curl 10.0.0.2:53:

curl: (52) Empty reply from server

Ok, I set srv_lookup_server = 10.0.0.2:53:

2019-08-14 11:54:20 [INFO ] (api): API disabled
2019-08-14 11:54:20 [INFO ] (metrics): Metrics disabled
2019-08-14 11:54:20 [INFO ] (manager): Initializing...
2019-08-14 11:54:20 [INFO ] (server): Creating 'sample': localhost:3000 weight srv none
2019-08-14 11:54:20 [INFO ] (scheduler): Starting scheduler sample
2019-08-14 11:54:20 [INFO ] (srvFetch): Fetching 10.0.0.2:53 my-domain.local.
2019-08-14 11:54:20 [INFO ] (manager): Initialized

What's next? My docker compose file:

version: '2'

services:
  dbgproxy:
    image: yyyar/gobetween
    container_name: lb
    hostname: lb
    ports:
     - 89:3000
    volumes:
     - /home/ec2-user/gobetween:/etc/gobetween/conf

And curl 127.0.0.1:89:

curl: (56) Recv failure: Connection reset by peer

[MatthewPattell]

I found my mistake. Need use 0.0.0.0 ip in config server, if you use docker.

[shantanugadgil]

@MatthewPattell I am curious as to what finally worked? (i.e. how does the config look?)

Also, was there a DNS server running on the host machine at all? The netstat output indicates that there is no DNS server listening

[MatthewPattell]

@shantanugadgil yes, it works.

1. Run EC2 instance optimized for ecs, with user data (for ECS cluster).
2. In my ECS cluster I create service with discovery. After start service, in my Route 53 created SRV record (.local domain zone)

Example (ssh to EC2):

mkdir -p /home/ec2-user/gobetween/conf
touch /home/ec2-user/gobetween/conf/gobetween.toml
touch /home/ec2-user/gobetween/docker-compose.yml

Find DNS private ip:

cat /etc/resolv.conf | grep -i '^nameserver' | head -n1 | cut -d ' ' -f2

In /home/ec2-user/gobetween/conf/gobetween.toml:

[servers.sample]
bind = "0.0.0.0:3000"
protocol = "tcp"
balance = "weight"

  [servers.sample.discovery]
  failpolicy = "keeplast"
  kind = "srv"
  srv_lookup_server = "10.0.0.2:53"
  srv_lookup_pattern = "my-domain.local."

In /home/ec2-user/gobetween/docker-compose.yml:

version: '2'

services:
  lb:
    image: yyyar/gobetween
    container_name: lb
    hostname: lb
    ports:
     - 89:3000
    volumes:
     - /home/ec2-user/gobetween/conf:/etc/gobetween/conf

4. Install docker-compose
5. Run docker-compose up -d
6. Test ``curl 127.0.0.1:89```

You can also run through ECS.