gobetween
gobetween copied to clipboard
yyyar
Support multiple address and port bindings
In my Docker Swarm, I have several services, each serves different API endpoints. Eg:
- auth.example.com
- registry.example.com
- admin.example.com
And of course each has some replicas to be load-balanced by Gobetween.
Currently, according to the limited examples on official documentation, I can only:
- bind to only one address
- filter with only one label and only one private port.
[servers.mydomain]
bind = "auth.example.com"
protocol = "tcp"
<something else>
[servers.mydomain.discovery]
kind = "docker"
docker_endpoint = "http://localhost:2375" # Docker / Swarm API
docker_container_label = "api=true" # only one label supported
docker_container_private_port = 80 # only one port supported
My question is: How can I bind Gobetween to multiple addresses? With each address, how can I filter with multiple labels and ports?
Thank you!
Hi @hirikarate,
Since gobetween is L4, to make it work with different "virtual" domains (that is information available in L7, actually) you can either:
Option (1) Create separate [server] section per your domain, if you have network interfaces (ip address) per each of them (3 in total for your case):
[servers.auth]
bind = "<auth_ip_address>:80"
protocol = "tcp"
[servers.auth.discovery]
kind = "docker"
docker_endpoint = "http://localhost:2375"
docker_container_label = "service=auth"
docker_container_private_port = 80
[servers.registry]
bind = "<registry_ip_address>:80"
protocol = "tcp"
[servers.registry.discovery]
kind = "docker"
docker_endpoint = "http://localhost:2375"
docker_container_label = "service=registry"
docker_container_private_port = 80
[servers.admin]
bind = "<admin_ip_address>:80"
protocol = "tcp"
[servers.admin.discovery]
kind = "docker"
docker_endpoint = "http://localhost:2375"
docker_container_label = "service=admin"
docker_container_private_port = 80
And run your containers with appropriate label "service=..." to differentiate them.
$ docker run --label service=admin myadminimage myprogram
$ docker run --label service=registry myregistryimage myprogram
$ docker run --label service=auth myauthimage myprogram
Option (2) Use SNI to balance based on provided domain name, if your backends use TLS/HTTPS. You need only 1 network interface / ip address in this case:
[servers.example]
bind = "0.0.0.0:443"
protocol = "tcp"
[servers.example.sni]
read_timeout = "2s"
hostname_matching_strategy = "exact"
unexpected_hostname_strategy = "reject"
[servers.example.discovery]
kind = "docker"
docker_endpoint = "http://localhost:2375"
docker_container_private_port = 443
And use "sni" label to specify container domain name while running it, for example:
$ docker run --label sni=admin.example.com myadminimage myprogram
$ docker run --label sni=registry.example.com myregistryimage myprogram
$ docker run --label sni=auth.example.com myauthimage myprogram
Let us know if you'll have any success with your configuration. Thanks!
This is my gobetween.toml (detail below) and the service keeps yelling: "(server.handle): No matching sni found, rejecting due to 'reject' unexpected hostname strategy"
- The upstream service is Portainer (a Golang webapp) with port mapping as 9000:9000 and label "sni=example.com" (label attached to container, not to the service itself). It is already accessible via "http://example.com:9000"
- I tried with both self-signed certificates "example.com" and "*.example.com".
- Not accessible via "https://example.com" although the firewall is allowing ports 9000, 80 and 443.
- I deploy "gobetween" as a Docker Swarm service, the "service create" command must include
-v //var/run/docker.sock:/var/run/docker.sockfor the discovery to work,http://localhost:2375just doesn't work for me. (Took me some time to figure it out. I recommend adding this to documentation)
[servers]
[servers.example]
bind = "0.0.0.0:443"
protocol = "tls"
balance = "leastconn"
[servers.example.tls]
cert_path = "/run/secrets/ssl_cert"
key_path = "/run/secrets/ssl_key"
min_version = "tls1"
max_version = "tls1.2"
ciphers = []
prefer_server_ciphers = false
session_tickets = true
[servers.example.sni]
read_timeout = "2s"
hostname_matching_strategy = "exact"
unexpected_hostname_strategy = "reject"
[servers.example.discovery]
kind = "docker"
docker_endpoint = "unix:///var/run/docker.sock"
docker_container_private_port = 9000
[servers.example.healthcheck]
fails = 1
passes = 1
interval = "2s"
timeout = "1s"
kind = "ping"
For ease of reproduction, this is my docker-compose.yml
version: '3.2'
networks:
gennet:
external: true
volumes:
gobetween-vol:
external: true
secrets:
ssl_cert:
external: true
ssl_key:
external: true
services:
portainer:
image: "portainer/portainer"
command: -H unix:///var/run/docker.sock
ports:
- 9000:9000
networks:
- gennet
labels:
- "sni=example.com"
volumes:
- type: bind
source: //var/run/docker.sock
target: /var/run/docker.sock
gobetween:
image: "yyyar/gobetween"
networks:
- gennet
ports:
- 80:80
- 443:443
secrets:
- ssl_cert
- ssl_key
volumes:
- type: bind
source: //var/run/docker.sock
target: /var/run/docker.sock
- type: volume
source: gobetween-vol
target: /etc/gobetween/conf
This is file deploy.sh:
#!/bin/bash
docker network create -d overlay gennet;
docker volume create --name=gobetween-vol;
docker secret create ssl_cert /path/to/ssl/server.crt
docker secret create ssl_key /path/to/ssl/server.key
cp ./gobetween.toml /var/lib/docker/volumes/gobetween-vol/_data
docker stack deploy -c docker-compose.yml "stackname"
sleep 3
echo "Services created successfully."
docker service ls
-
How to deploy:
- Make sure 3 files
gobetween.toml,docker-compose.ymlanddeploy.share in same location. - Make sure to change label
sni=example.com(indocker-compose.yml) and/path/to/ssl/(indeploy.sh). - Make sure Docker Swarm is initialized (
docker swarm init --advertise-addr <IP_ADDRESS>) - Exec
sudo bash ./deploy.sh(you may want tosudo chmod +x ./deploy.sh)
- Make sure 3 files
-
Testing: If you can access Portainer via "http://example.com:9000" (don't forget firewall port) or "http://<IP_ADDRESS>:9000" then you are good.
-
Target: How to make it accessible via "https://example.com"?
-
Some helpful commands:
- View gobetween log:
docker service logs stackname_gobetween - Edit gobetween conf:
sudo vi /var/lib/docker/volumes/gobetween-vol/_data/gobetween.toml - Restart gobetween service:
docker service scale stackname_gobetween=0thendocker service scale stackname_gobetween=0
- View gobetween log:
-
Clean up:
- Removes all services
docker service rm $(docker service ls -q) - Removes all containers
docker container rm $(docker container ls -q) - Remove volume
docker volume rm gobetween-vol - Remove network
docker network rm gennet
- Removes all services
Please give your thoughts. Thank you!
@hirikarate Thanks for the details. Seems everything should work good. It may take a while for us to setup everything you listed to reproduce locally (I just tried the same thing just without swarm and was unable to reproduce) , but I think It may allow us to debug faster if you could provide also the following info:
-
Output of your
$ docker ps --format "{{.ID}} {{.Ports}} {{.Labels}}"orinspecton any of yours Portainer containers to ensure they have sni label. -
Please enable REST API in your gobetween configuration, and provide output of GET
http://HOST:8888/servers/example/statsendpoint (you'll probably need to expose gobetween REST API 8888 by default port in your container). It will show if gobetween correctly discovered your containers and got correct sni label value.
Thank you!
-
Result of
docker ps --format...., the last 2 lines are assigned labels.406f23b44c20 9000/tcp com.docker.swarm.node.id=i7vqv52siv47zeko68rnuhnck, com.docker.swarm.service.id=pnpogwpcnjaexhrsoz2xaxzcx, com.docker.swarm.service.name=infras_portainer, com.docker.swarm.task=, com.docker.swarm.task.id=7dkbx2tlxcybr9fr3y07fyhfo, com.docker.stack.namespace=infras, com.docker.swarm.task.name=infras_portainer.1.7dkbx2tlxcybr9fr3y07fyhfo, gnv.container.portainer=1, sni=example.com -
Result of REST API
docker exec 3da curl http://localhost:8888/servers/example/stats(call from inside the running container):% Total % Received % Xferd Average Speed Time Time Time Current Dload Upload Total Spent Left Speed 100 594 100 594 0 0 594 0 0:00:01 --:--:-- 0:00:01 580k { "active_connections": 0, "rx_total": 0, "tx_total": 0, "rx_second": 0, "tx_second": 0, "backends": [ { "host": "", "port": "0", "priority": 1, "weight": 1, "sni": "firstidea.vn", "stats": { "live": false, "total_connections": 0, "active_connections": 0, "refused_connections": 0, "rx": 0, "tx": 0, "rx_second": 0, "tx_second": 0 } } ] } -
Result of calling
curl http://localhost:8888/servers/example/stats(from outside container through published port): Hangs forever
Yes
406f23b44c20 9000/tcp
docker swarm mode do not propogate it`s ip via api because in swarm mode network works this way :
https://docs.docker.com/engine/swarm/ingress/#configure-an-external-load-balancer
but this is not so flexible. probably it needs to dive into swarm mode api to find out better way
OK, never mind it. Maybe what I want to achieve is not what Gobetween is designed for, as you said Gobetween works on L4.
I come up with this design, in which everything are containers/services in a Docker Swarm Mode. An Nginx container stands upfront to accept public requests and redirect them to the loadbalancer of each internal containers.
As how Docker Swarm Mode works, each service has many replica containers, and current Docker version (17.09.0-ce) has only one loadbalancing strategy (roundrobin). This is why in the first place I came for Gobetween.
Each actor in below diagram is Docker container, this diagram doesn't care about physical host machines as it is Docker Swarm manager's job.
+———————+
| | - Domain-name-based routing
| NGINX | - Reversed proxy
| | - SSL termination
+———————+
|
+———————————————————————————————+
| |
+—————+ +—————+
(Gobetween | | | | (Gobetween
container | G.A | +——————————————————————+ | G.B | container
for svc A) | | | | for svc B)
+—————+ +—————+
| (Loadbalancing) | (Loadbalancing)
+———————————————+ +———————————————+
| | | |
+———————————+ +———————————+ +———————————+ +———————————+
| SVC A | | SVC A | | SVC B | | SVC B |
| | | | | | | |
| REPLICA 1 | | REPLICA 2 | | REPLICA 1 | | REPLICA 2 |
| | | | | | | |
+———————————+ +———————————+ +———————————+ +———————————+
Thank you for your support sofar.
@hirikarate I'm not very familiar with Docker Swarm, but looking at your above diagram, it looks very similar to what I came up with here: https://github.com/yyyar/gobetween/issues/77.
My idea was that instead of your nginx load balancer, gobetween would be used. gobetween would then communicate with the other "downstream" gobetween services. The upstream gobetween service would be automatically configured by querying the downstream gobetween services via API.
The upstream gobetween service requires no knowledge of the discovery plugin used in the downstream gobetween servers. So your G.A. and G.B. would be configured to query via docker/docker swarm and the upstream would query via the gobetween API.
If this sounds like what you're wanting to achieve, I have a proof-of-concept here: https://github.com/yyyar/gobetween/pull/95. I used LXD for my proof of concept, but any current discovery plugin should work with this.
@jtopjian In fact I copied your diagram to save some key strokes 😄
My problem is , as I stated at the beginning of this topic, Gobetween is not domain-name-aware, so I cannot redirect requests to related internal micro services. Eg:
- Requests to "auth.example.com" should be loadbalanced among replicas of
SVC A. - Requests to "admin.example.com" should be loadbalanced among replicas of
SVC B.
I still don't get how your 2 upstream Gobetween instances can handle this situation without a L7 reversed proxy server (eg. Nginx), which is quite common in reality. Except when each of your upstream instance occupies a domain-name, but in that case they are not in active/passive keepalived pattern anymore.
At the top, gobetween runs on two separate hosts in an active/passive keepalived configuration.
Hi, are the services connecting to your original proposed endpoint TLS based? The reason I ask is, if they are TLS based, then Gobetween can be used to "route" requests based on on TCP+SNI. I use a similar setup to route entries based on TCP+SNI. I don't use Docker Swarm, I use Nomad+Consul
ref: https://github.com/yyyar/gobetween/issues/53 ref: https://cloud.githubusercontent.com/assets/2508915/24521698/b6890080-15aa-11e7-9b50-bbd48102cad6.png
Also, in general, the idea is to bind to least number of ports possible (for security reasons) (different from the inhouse/onpremise thinking where the infrastructure is behind a firewall already)
HTH, Shantanu
@hirikarate
My propblem is , as I stated at the beginning of this topic, Gobetween is not domain-name-aware, so I cannot redirect requests to related internal micro services. Eg:
ahh - my mistake. I'll admit I've only been scanning this topic. It was your diagram that caught my eye :)
That's a completely understandable requirement. As @shantanugadgil mentioned, TCP+SNI might be a solution, but also understandable if that won't work, either.
Unfortunately, Docker Swarm is in our system design's requirement, and gobetween is just a dropped-in helper. I think I should accept the fact that I need L7 reversed proxy in front of Gobetween instances.
Anyway thank all of you for your support.
After some time of experimentation I can confirm Swarm Mode does not play nice with gobetween - that's why your setup does not work (you have 0 and "" in port/host in your discovered backend).
Currently gobetween get ports mapping from container api / metadata. It works with plain Docker and plain old Docker Swarm (Docker Swarm standalone: https://github.com/docker/swarm). but not with new Swarm Mode. In Swarm Mode port mappings are stored inside "service" entity, so it's not compatible with current "docker" discovery.
We need to do some extra work to add Docker Swarm Mode discovery functionality to gobetween, either as separate type of discovery (say kind="docker-swarmmode"), or add add more parameters to current 'docker' discovery.
So I think TCP+SNI+Docker discovery would work for you when we'll add swarm mode support.
It seems like a separate feature/issue we need to create to discuss further.
One remark, just discovered it may work right if you create your service with the following publish parameter: "--publish mode=host,target=80". We just need to try it and verify. More info: https://docs.docker.com/engine/swarm/services/#publish-a-services-ports-directly-on-the-swarm-node @nickdoikov
@hirikarate could you please try configure your portainer service setup with publishing service ports "directly on swarm mode" as described here - "--publish mode=host,target=80", instead of publishing using a swarm routing mesh (by default)? In case it's still actual for you :-)
I just tried it with docker swarm mode and it works fine. We still need to figure out how make it work with swam routing mesh, there is separate issue for that - #111