Web server doesn't bind to all addresses in docker

[dumbasPL]

Problem: when there is more than one network interface attached to the container, the web server binds only to one of them and only on IPv4. The SSH service binds correctly to all v4 and v6 addresses.

This is problematic with setups using a reverse proxy on a separate network or anything IPv6-related. These issues might exist on the non-docker version as well, haven't checked.

(simplified) example use case with traefik and apprise

services:
  borgwarehouse:
    image: borgwarehouse/borgwarehouse
    container_name: borgwarehouse
    restart: always
    user: '1001:1001'
    networks:
      - default
      - proxy
    ports:
      - '2222:22'
    environment:
      - NEXTAUTH_URL=https://borg.example.com
      # rest redacted
    volumes:
      # redacted
    labels:
      - "traefik.enable=true"
      - "traefik.http.routers.borgwarehouse.entrypoints=https"
      - "traefik.http.routers.borgwarehouse.rule=Host(`borg.example.com`)"
      - "traefik.http.routers.borgwarehouse.tls=true"
      - "traefik.http.routers.borgwarehouse.service=borgwarehouse"            
      - "traefik.http.services.borgwarehouse.loadbalancer.server.port=3000"
      - "traefik.docker.network=proxy"
  apprise:
    image: caronc/apprise
    container_name: apprise
    restart: always
    networks:
      - default
networks:
  proxy:
    external: true

In my instance, the web server decided to bind to the default network (aka the one created by default by docker-compose) instead of the proxy network so the reverse proxy can't reach it.

[Ravinou]

Hi @dumbasPL 😀 I specified an environment variable for ipv6 here in the doc, so it didn't work? Thanks

[dumbasPL]

I specified an environment variable for ipv6

Setting HOSTNAME to 0.0.0.0 or :: as an env variable for the container does indeed fix the issue. However, I would expect it to behave like any other service in a container and bind to all interfaces by default unless specified otherwise.

A few problems with binding to :: by default are:

• it binds ONLY to v6 (ssh will correctly bind to both separately). This normally isn't an issue unless the user has net.ipv6.bindv6only set or has disabled v6 on their system/kernel (pany still do this, for some reason). Afaik there is no easy way to make next (or node for that matter) bind to both. And in my opinion, the defaults should "just work".
• propper v6 in containers is rare because it's a pain in the ass to do right. I only have it on the reverse proxy and containers that can't be reverse proxied. This one has SSH, so it would fall under this category, but SSH already binds correctly to both and the HTTP port is reverse proxied anyway for TLS.

[Ravinou]

Is this a change that won't affect existing installations or new installations that want to use IPv4 ? Thanks. I'll be testing this soon.

[dumbasPL]

Existing installations shouldn't be affected.

Current behavior on bare metal:

• Bind to all IPv4 interfaces (default for next) when HOSTNAME is unset

New behavior on bare metal:

• unchanged

Current behavior on docker:

• docker sets HOSTNAME to the hostname of the container
• next resolves HOSTNAME to the first IP in /etc/hosts (also generated by docker) and binds to it

New suggested behavior on docker:

• Bind to all IPv4 interfaces

I don't consider this a breaking change since the new behavior includes the old one and will work without any changes assuming they aren't running a v6-only stack in docker (highly unlikely, massive pain, discouraged by docker themselves).

The only thing that needs to be considered is that if we ignore the HOSTNAME docker gives us, this will also prevent the user from specifying their own value to for example enable binding to v6. But since next doesn't support binding to both v4 and v6 at the same time, it's highly unlikely that somebody will ever want to do that. They will either use the NAT system that docker gives you by default (aka it will translate incoming v6 to v4) or they will use a reverse proxy with v6 support in front of it.

If for some reason we want to give the user a way to overwrite the HOSTNAME in the container we have to be able to differentiate between the docker daemon setting the default one vs the user specifying a custom one. One way to do this would be to compare the HOSTNAME variable with the actual hostname of the container and only overwrite it to 0.0.0.0 if they match. If they don't that means the user provided a custom value and we should use that instead. Another option would be to add a new variable with a different name but that would mean that the docker configuration wouldn't be compatible with the bare metal one.

[Ravinou]

I took the time to carry out some tests to understand your case properly.

Your BorgWarehouse container is on several networks. So I reconstructed this case to understand it better by adding the following to the end of my docker-compose.yml :

networks:
  default:
  secondary_network:
    driver: bridge  # bridge type for testing, we don't care

Next, I checked that my container was booting on two networks with : docker inspect borgwarehouse --format '{{json .NetworkSettings.Networks}}' |jq .

And finally, I ran curl tests on each of the networks to see if I could access the web interface :

docker run --rm --network borgwarehouse_secondary_network appropriate/curl -s http://borgwarehouse:3000
docker run --rm --network borgwarehouse_default appropriate/curl -s http://borgwarehouse:3000

I have noticed that without the environment variable HOSTNAME=0.0.0.0 or without adding environment=HOSTNAME=0.0.0.0 in the borgwarehouse section of supervisord.conf, the web interface is not accessible from the secondary network. So, I'm agree with you and now perfectly understand your case :)

Now I have a question. Anyone wishing to bind IPv6 would have to rebuild the container by modifying supervisord.conf. Which is more trouble than changing an environment variable.

That's what really makes me hesitate to accept the change there. Are you sure that a docker network in ipv6 is not recommended and never done? Also, the configuration in supervisord.conf seems to take precedence over the environment variable file. So it potentially breaks the installation of someone who had already bind ipv6 in .env, right?

[dumbasPL]

Did some testing and I think I have a solution that doesn't break existing setups and allows the user to change the value. Will make a pr soon so maybe wait with the release

[Ravinou]

It's merged, and it will be in the next release. Thank you @dumbasPL 😊