Neil-Ni
Possible to support multiple services?
If I had another service, say helloworld2 - how difficult would it be to expand on this demo to support that?
Suppose you wanted lb:80/helloworld2/* to be routed to that app...
I briefly demonstrate how to do that near the end of this post: http://blog.neilni.com/2015/09/14/consul-and-consul-template-with-docker-compose/
How would you get the second service configured with nginx simultaneously?
consul-template automatically populates the correct nginx conf for you. do you see anything if you try this?
docker-compose run lb consul-template -consul consul:8500 -template "/etc/consul-templates/app.conf:/tmp/result:service nginx restart" -dry
@Neil-Ni Your question was not for me, but this is what I get:
ubuntu@ip-172-31-30-16:~/_registrator/consul-demo$ docker-compose run lb consul-template -consul consul:8500 -template "/etc/consul-templates/app.conf:/tmp/result:service nginx restart" -dry
[fix-attrs.d] applying owners & permissions fixes...
[fix-attrs.d] 00-runscripts: applying...
[fix-attrs.d] 00-runscripts: exited 0.
[fix-attrs.d] done.
[cont-init.d] executing container initialization scripts...
[cont-init.d] done.
[services.d] starting services
[services.d] done.
2016/04/11 22:41:18 [DEBUG] (config) loading configs from "/etc/consul-template/conf"
2016/04/11 22:41:18 [DEBUG] (config) merging with "/etc/consul-template/conf/app.conf"
Consul Template returned errors:
open /etc/consul-templates/app.conf: no such file or directoryconsul-template exited 14
Received interrupt, cleaning up...
[cont-finish.d] executing container finish scripts...
[cont-finish.d] done.
[s6-finish] syncing disks.
[s6-finish] sending all processes the TERM signal.
[s6-finish] sending all processes the KILL signal and exiting
My own question is: In my browser I get
504 Gateway Time-out
nginx/1.9.6
In my AWS EC2 Ubuntu t2.micro environment I get the nginx welcome page with curl localhost - not exactly hello world, is it? So I really don't understand what is happening here.
$ curl localhost:8500/v1/catalog/services
{"consul":[],"consul-53":["udp"],"consul-8400":[],"consul-8500":[],"consuldemo_lb-80":[],"helloworld":["production"]}
That is ok. But then what? I read your blog post but cannot reproduce your results. Any hint?
helloworld:
build: ./helloworld
ports:
- "3000"
environment:
SERVICE_NAME: helloworld
SERVICE_TAGS: production
consul:
command: -server -bootstrap
image: progrium/consul:latest
ports:
- "8400:8400"
- "8500:8500"
- "8600:53/udp"
- "172.17.0.1:53:53/udp" # docker address
registrator:
command: -ip=54.93.106.184 consul://consul:8500 # AWS address
image: gliderlabs/registrator:latest
volumes:
- "/var/run/docker.sock:/tmp/docker.sock"
links:
- consul
lb:
build: ./lb
links:
- consul
ports:
- "80:80"
environment:
CONSUL_URL: consul:8500
volumes:
- ./lb/consul-template:/etc/consul-template
Hey @kklepper,
It's been a while since I last updated. My bad! Here's some directions and hope you find it helpful:
Consul template file has been moved to another location:
From your error:
open /etc/consul-templates/app.conf: no such file or directoryconsul-template exited 14.
Instead, this should get you what you are looking for:
sudo docker-compose run lb consul-template -consul consul:8500 -template "/etc/consul-template/templates/app.ctmpl:/tmp/result:service nginx restart" -dry
Docker bridge
Your docker address looks right, but just in case you want to verify again, do:
> ifconfig docker0
docker0 Link encap:Ethernet HWaddr 02:42:2B:63:33:5E
inet addr:172.17.0.1 Bcast:0.0.0.0 Mask:255.255.0.0
...
where 172.17.0.1 is the docker address.
IP
It looks like you are using a 54* IP which is normally a public instance IP, and I think you should be using your private IP instead. I'm not an expert on networking, but this IP that we are passing to consul is for all the other services to look up. By default, the public IP can find private IP, but I don't think it goes the other way around. That being said, if you bind your consul to the public IP, the rest of your services in your internal network probably cannot find it. The other reason you might want to bind to your private IP is that you don't really want other people to look up your services from outside.
And now you should be able to do curl <Private-IP> to get helloworld response
hope this helps :)
@Neil-Ni Thanks a lot! This is what I get in dry run:
ubuntu@ip-172-31-30-16:~/_registrator/consul-demo$ sudo docker-compose run lb consul-template -consul consul:8500 -template "/etc/consul-template/templates/app.ctmpl:/tmp/result:service nginx restart" -dry
Creating consuldemo_consul_1
Building lb
Step 1 : FROM 1science/nginx:consul
---> 2c4f8f88cea7
Step 2 : COPY config/nginx.conf /etc/nginx/nginx.conf
---> 1243fb75ed58
Removing intermediate container 738e98e03b07
Successfully built 1243fb75ed58
[fix-attrs.d] applying owners & permissions fixes...
[fix-attrs.d] 00-runscripts: applying...
[fix-attrs.d] 00-runscripts: exited 0.
[fix-attrs.d] done.
[cont-init.d] executing container initialization scripts...
[cont-init.d] done.
[services.d] starting services
[services.d] done.
2016/04/12 10:35:12 [DEBUG] (config) loading configs from "/etc/consul-template/conf"
2016/04/12 10:35:12 [DEBUG] (config) merging with "/etc/consul-template/conf/app.conf"
2016/04/12 10:35:12 [ERR] (view) "services" catalog services: error fetching: Unexpected response code: 500 (No cluster leader)
2016/04/12 10:35:12 [ERR] (runner) watcher reported error: catalog services: error fetching: Unexpected response code: 500 (No cluster leader)
2016/04/12 10:35:12 [ERR] (view) "services" catalog services: error fetching: Unexpected response code: 500 (No cluster leader)
2016/04/12 10:35:12 [ERR] (runner) watcher reported error: catalog services: error fetching: Unexpected response code: 500 (No cluster leader)
> /tmp/result
server {
listen 65333;
location / {
types {
application/json json;
}
default_type "application/json";
return 501 '{
"success": false,
"deploy": false,
"status": 501,
"body": {
"message": "No available upstream servers at current route from consul"
}
}';
}
}
server {
listen 80;
location / {
proxy_pass http://helloworld;
}
}2016/04/12 10:35:17 [emerg] 91#0: host not found in upstream "helloworld" in /etc/nginx/conf.d/app.conf:29
nginx: [emerg] host not found in upstream "helloworld" in /etc/nginx/conf.d/app.conf:29
2016/04/12 10:35:17 [ERR] (runner) error running command: exit status 1
Consul Template returned errors:
1 error(s) occurred:
* exit status 12016/04/12 10:35:18 [DEBUG] (config) loading configs from "/etc/consul-template/conf"
2016/04/12 10:35:18 [DEBUG] (config) merging with "/etc/consul-template/conf/app.conf"
This is without:
ubuntu@ip-172-31-30-16:~/_registrator/consul-demo$ sudo docker-compose run lb consul-template -consul consul:8500 -template "/etc/consul-template/templates/app.ctmpl:/tmp/result:service nginx restart"
[fix-attrs.d] applying owners & permissions fixes...
[fix-attrs.d] 00-runscripts: applying...
[fix-attrs.d] 00-runscripts: exited 0.
[fix-attrs.d] done.
[cont-init.d] executing container initialization scripts...
[cont-init.d] done.
[services.d] starting services
[services.d] done.
2016/04/12 10:37:38 [DEBUG] (config) loading configs from "/etc/consul-template/conf"
2016/04/12 10:37:38 [DEBUG] (config) merging with "/etc/consul-template/conf/app.conf"
* service: service `nginx' does not exist
2016/04/12 10:37:38 [ERR] (runner) error running command: exit status 1
Consul Template returned errors:
1 error(s) occurred:
* exit status 12016/04/12 10:37:38 [emerg] 91#0: host not found in upstream "helloworld" in /etc/nginx/conf.d/app.conf:29
nginx: [emerg] host not found in upstream "helloworld" in /etc/nginx/conf.d/app.conf:29
2016/04/12 10:37:38 [ERR] (runner) error running command: exit status 1
Consul Template returned errors:
1 error(s) occurred:
* exit status 1consul-template exited 14
[cont-finish.d] executing container finish scripts...
[cont-finish.d] done.
[s6-finish] syncing disks.
[s6-finish] sending all processes the TERM signal.
[s6-finish] sending all processes the KILL signal and exiting.
As for your syntax:
sudo docker-compose run lb consul-template -consul consul:8500 -template "/etc/consul-template/templates/app.ctmpl:/tmp/result:service nginx restart" -dry
I don't know what is happening here. I don't understand the syntax and don't find any hint to dry. /etc/consul-template/templates/app.ctmpl should refer to my box, but this is not what docker-compose or docker can digest -- everything has to be in the development directory. That's what's given by
volumes:
- ./lb/consul-template:/etc/consul-template
Now consul shows something:
ubuntu@ip-172-31-30-16:~$ curl localhost:8500/v1/catalog/service/helloworld
[{"Node":"1b7a412aa302","Address":"172.17.0.2","ServiceID":"a49ea2ac188b:consuldemo_helloworld_1:3000","ServiceName":"helloworld","ServiceTags":["production"],"ServiceAddress":"54.93.106.184","ServicePort":32770}]
ubuntu@ip-172-31-30-16:~$ curl localhost:8500/v1/catalog/service/consuldemo_lb-80
[{"Node":"1b7a412aa302","Address":"172.17.0.2","ServiceID":"a49ea2ac188b:consuldemo_lb_1:80","ServiceName":"consuldemo_lb-80","ServiceTags":null,"ServiceAddress":"54.93.106.184","ServicePort":80}]
The Address and ServiceAddress is identical in both cases here. But what is 54.93.106.184? My public IP right now is 54.93.197.156
ubuntu@ip-172-31-30-16:~$ curl 172.17.0.2
curl: (7) Failed to connect to 172.17.0.2 port 80: Connection refused
ubuntu@ip-172-31-30-16:~$ curl localhost
<!DOCTYPE html>
<html>
<head>
<title>Welcome to nginx!</title>
<style>
body {
width: 35em;
margin: 0 auto;
font-family: Tahoma, Verdana, Arial, sans-serif;
}
</style>
</head>
<body>
<h1>Welcome to nginx!</h1>
<p>If you see this page, the nginx web server is successfully installed and
working. Further configuration is required.</p>
<p>For online documentation and support please refer to
<a href="http://nginx.org/">nginx.org</a>.<br/>
Commercial support is available at
<a href="http://nginx.com/">nginx.com</a>.</p>
<p><em>Thank you for using nginx.</em></p>
</body>
</html>
I'm afraid I don't understand the whole construction.
This is my understanding so far: lb should spread the load to all the containers registrator detects and publishes to consul which somehow is managed by the template which in turn speaks nginx. It doesn't even work with one container.
Or am I missing something?
I just found out that -dry refers to consul-template and am reading up to understand what's happening. Thank you. I may come back to you either way.
This indicates that you have a file in /etc/consul-template/templates/ which I don't have.
ubuntu@ip-172-31-30-16:~/_registrator/consul-demo/lb$ cat consul-template/conf/app.conf
template {
source = "/etc/consul-template/templates/app.ctmpl"
destination = "/etc/nginx/conf.d/app.conf"
command = "nginx -s reload"
}
why did you curl 172.17.0.2? your private IP is 172-31-30-16. Also, the consul template file is inside the container not in your host machine.
Ah, I see my mistake with template location. docker-compose.yml puts things in place inside the container, and then it is copied for nginx. I think I understand this now. Thank you.
As for my private IP:
ubuntu@ip-172-31-30-16:~/_registrator/consul-demo$ curl 172.31.30.16
<html>
<head><title>504 Gateway Time-out</title></head>
<body bgcolor="white">
<center><h1>504 Gateway Time-out</h1></center>
<hr><center>nginx/1.9.6</center>
</body>
</html>
Let me recap here: Consul is a key value store and does not do anything by itself, registrator watches docker containers come and go and pass this information to Consul, and Consul-template witnesses these changes and passes them to NGINX, restarting it to put these changes into effect. Did I understand the mechanism thus far?
helloworld has NGINX as well, published on port 3000:
ubuntu@ip-172-31-30-16:~/_registrator/consul-demo$ curl 172.17.0.5:3000
Hello World! from 483f46ce6c53
lb introduces NGINX as load balancer which should pass inquiries to one or any number of helloworld containers. In order to do this, NGINX needs a conf file. This file will be manipulated by consul-template as needed.
So in theory I should be able to see this hello world message not only with curl but also on my public IP in the browser. In the browser, I get a timeout, and with curl I get the standard NGINX message indicating that the inquiry has not been passed on to helloworld.
ubuntu@ip-172-31-30-16:~/_registrator/consul-demo$ curl localhost
<!DOCTYPE html>
<html>
<head>
<title>Welcome to nginx!</title>
<style>
body {
width: 35em;
margin: 0 auto;
font-family: Tahoma, Verdana, Arial, sans-serif;
}
</style>
</head>
<body>
<h1>Welcome to nginx!</h1>
<p>If you see this page, the nginx web server is successfully installed and
working. Further configuration is required.</p>
<p>For online documentation and support please refer to
<a href="http://nginx.org/">nginx.org</a>.<br/>
Commercial support is available at
<a href="http://nginx.com/">nginx.com</a>.</p>
<p><em>Thank you for using nginx.</em></p>
</body>
</html>
This is a nice surprise:
ubuntu@ip-172-31-30-16:~/_registrator/consul-demo$ sudo docker exec -it consuldemo_lb_1 bash
## .
## ## ## ==
## ## ## ## ===
/""""""""""""""""\___/ ===
~~~ {~~ ~~~~ ~~~ ~~~~ ~~ ~ / ===- ~~~
\______ o __/
\ \ __/
\____\______/
___ _ ____ __
< /_________(_)__ ____ ________ / __ \____ _____/ /_____ _____
/ / ___/ ___/ / _ \/ __ \/ ___/ _ \ / / / / __ \/ ___/ //_/ _ \/ ___/
/ (__ ) /__/ / __/ / / / /__/ __/ / /_/ / /_/ / /__/ ,< / __/ /
/_/____/\___/_/\___/_/ /_/\___/\___/ /_____/\____/\___/_/|_|\___/_/
Alpine Linux 3.1 image. (Linux 3.13.0-40-generic #69-Ubuntu SMP Thu Nov 13 17:53:56 UTC 2014)
- with nginx version: nginx/1.9.6
- with consul-template v0.11.0
Now I look at the container and see
root@/ > cat etc/nginx/conf.d/app.conf
upstream helloworld {
least_conn;
server 192.168.99.100:32804 max_fails=3 fail_timeout=60 weight=1;
}
After firing up some more helloworlds, I see
upstream helloworld {
least_conn;
server 192.168.99.100:32805 max_fails=3 fail_timeout=60 weight=1;
server 192.168.99.100:32806 max_fails=3 fail_timeout=60 weight=1;
server 192.168.99.100:32808 max_fails=3 fail_timeout=60 weight=1;
server 192.168.99.100:32809 max_fails=3 fail_timeout=60 weight=1;
server 192.168.99.100:32807 max_fails=3 fail_timeout=60 weight=1;
}
This looks really good, doesn't it? So the problem seems to be with NGINX. I have no idea yet. Any hint?
It's pretty obvious that you are using the wrong IP. why still 192.168.99.100?
Excuse me, I don't know. Which IP should I use then? The one I used before looks pretty much like a public IP I used a couple of days before.
Ahhhh! I used the private IP here now, and it works on the public IP!!!
registrator:
command: -ip=172.31.30.16 consul://consul:8500
curl localhost still shows the welcome message. I'll take some time to learn more.
Thanks a lot!