nginx: [emerg] host not found in upstream

[djravine]

Hi Guys,

I'm currently having an issue with ECS Service Discovery and Route53 Auto Naming. I have added the Service Registry to the service and all the Hosted Zones records are being populated automatically. But I cannot seem to work out how to resolve the DNS SRV records with NGINX + ngx_upstream_resolveMK.

# DNS RESOLVER
resolver ns-x.awsdns-xx.com valid=10s;

# UPSTREAMS
upstream kibana {
  resolveMK servicediscovery.ecs service=kibana;
}

# HOST - kibana.example.com
server {
  server_name kibana.example.com;
  listen 80;
  location / {
	proxy_pass https://kibana/;
	rewrite ^/(.*)$ /$1 break;
  }
}

ERROR: nginx: [emerg] host not found in upstream "servicediscovery.ecs" in /usr/local/nginx/sites-enabled/kibana.conf:3

Does ngx_upstream_resolveMK need an A record to resolve the DNS where the SRV records are stored? So it appears that there is a missing A record needed to resolve "servicediscovery.ecs" to the Route 53 Private Zone. Do I need to manually add this? or is there a way to dynamically add this A record? I think this is the cause of the problem, but I'm still learning and might be way off.

STACKOVERFLOW: https://stackoverflow.com/questions/53549991/nginx-reverse-proxy-ngx-upstream-resolvemk-trying-to-resolve-srv-from-ecs-se

[ebernard]

I haven't tested it myself yet, but is your namespace a private one? If so, maybe your resolver address is incorrect?

[djravine]

The namespace is a Private Namespace. I have triple checked the resolver address and it's 100% correct. I can correctly DIG/NSLOOKUP the SRV records from both the Host EC2 Instance and inside the Docker Container. But ResolveMK Cannot seem to resolve the same.

[djravine]

I have also attempted to use the VPC Resolver on the CIDR x.x.x.2 IP. This also doesn't appear to work. :'(

[DanielIvaylov]

Hi, @djravine what was the solution for you? This is what is going on my containers.

After i restarted Kibana Elastic and Logstash they are running but after restarting nginx i got this erros:

In the logs of nginx i get: nginx: [emerg] host not found in upstream "kibana" in /etc/nginx/nginx.conf:82

my nginx file:

    location /kibana {
        rewrite ^/kibana/(.*) /$1 break;
        proxy_pass              http://kibana:5601/;
        proxy_set_header        Host $http_host;
        proxy_set_header        X-Real-IP $remote_addr;
        proxy_set_header        X-Forwarded-For $proxy_add_x_forwarded_for;
        proxy_set_header        X-Forwarded-Proto $scheme;
   }

    location /logstash {
        proxy_http_version 1.1;
        proxy_set_header Upgrade $http_upgrade;
        proxy_set_header Connection 'upgrade';
        proxy_set_header Host $host;
        proxy_pass            http://logstash:8082/;
        proxy_read_timeout    90;
        proxy_connect_timeout 90;
    }

Thanks