matrix-docker-ansible-deploy Configure DNS should make some notes re: /etc/hosts

https://github.com/spantaleev/matrix-docker-ansible-deploy/blob/master/docs/configuring-dns.md

I ran into an issue with the matrix_appservice_discord bot, because the bot was resolving the matrix.example.com to 127.0.1.1 because that's how the default ubuntu /etc/hosts config resolves it.

I had to fix up /etc/hosts so the FQDN would resolve to the external IP.

Jan 26 '21 04:01 skepticalwaves

My testing showed that /etc/hosts entries only seem to make it into the container's /etc/hosts if you launch the container with --net=host:

docker run --rm docker.io/alpine:3.13 /bin/sh -c 'cat /etc/hosts'. Has a pristine /etc/hosts file that looks like this:

127.0.0.1       localhost
::1     localhost ip6-localhost ip6-loopback
fe00::0 ip6-localnet
ff00::0 ip6-mcastprefix
ff02::1 ip6-allnodes
ff02::2 ip6-allrouters
172.17.0.2      4dcf9047f2f8

docker run --rm --net=host docker.io/alpine:3.13 /bin/sh -c 'cat /etc/hosts'. Has the host's /etc/hosts file merged into its own.

I also have matrix.DOMAIN pointed to 127.0.0.1 in my own /etc/hosts file, but it hasn't affected services negatively (yet).

I don't think we run containers with --net=host though, so I wonder how you came to run into this issue. Perhaps some other way? Or some different Docker version does things differently?

Jan 26 '21 08:01 spantaleev

This was deployed on a pristine Ubuntu 20.04.1 using only the ansible script, with ansible installed via pip3.

The only component I had issues with was the matrix_appservice_discord bot:

Jan 25 03:43:08 matrix systemd[1]: Started Matrix Appservice Discord bridge.
Jan 25 03:43:14 matrix matrix-appservice-discord[250626]: Jan-25 03:43:14.011 [DiscordStore] info: Starting DB Init
Jan 25 03:43:14 matrix matrix-appservice-discord[250626]: Jan-25 03:43:14.017 [DiscordStore] info: connString present in config, using postgres
Jan 25 03:43:14 matrix matrix-appservice-discord[250626]: Jan-25 03:43:14.019 [Postgres] info: Opening @matrix-postgres:5432/matrix_appservice_discord
Jan 25 03:43:14 matrix matrix-appservice-discord[250626]: Jan-25 03:43:14.120 [DiscordStore] info: Database schema version is 11, latest version is 11
Jan 25 03:43:14 matrix matrix-appservice-discord[250626]: Jan-25 03:43:14.121 [DiscordStore] info: Updated database to the latest schema
Jan 25 03:43:14 matrix matrix-appservice-discord[250626]: Jan-25 03:43:14.138 [bot-sdkMatrixLiteClient (REQ-1)] info: [ 'POST https://matrix.<redacted>.com/_matrix/client/r0/register' ]
Jan 25 03:43:14 matrix matrix-appservice-discord[250626]: Jan-25 03:43:14.181 [bot-sdkMatrixLiteClient (REQ-1)] error: [
Jan 25 03:43:14 matrix matrix-appservice-discord[250626]:   Error: connect ECONNREFUSED 127.0.1.1:443
Jan 25 03:43:14 matrix matrix-appservice-discord[250626]:       at TCPConnectWrap.afterConnect [as oncomplete] (net.js:1146:16) {
Jan 25 03:43:14 matrix matrix-appservice-discord[250626]:     errno: -111,
Jan 25 03:43:14 matrix matrix-appservice-discord[250626]:     code: 'ECONNREFUSED',
Jan 25 03:43:14 matrix matrix-appservice-discord[250626]:     syscall: 'connect',
Jan 25 03:43:14 matrix matrix-appservice-discord[250626]:     address: '127.0.1.1',
Jan 25 03:43:14 matrix matrix-appservice-discord[250626]:     port: 443
Jan 25 03:43:14 matrix matrix-appservice-discord[250626]:   }
Jan 25 03:43:14 matrix matrix-appservice-discord[250626]: ]
Jan 25 03:43:14 matrix matrix-appservice-discord[250626]: Jan-25 03:43:14.184 [bot-sdkAppservice] error: [ 'Encountered error registering user: ' ]
Jan 25 03:43:14 matrix matrix-appservice-discord[250626]: Jan-25 03:43:14.185 [bot-sdkAppservice] error: [
Jan 25 03:43:14 matrix matrix-appservice-discord[250626]:   Error: connect ECONNREFUSED 127.0.1.1:443
Jan 25 03:43:14 matrix matrix-appservice-discord[250626]:       at TCPConnectWrap.afterConnect [as oncomplete] (net.js:1146:16) {
Jan 25 03:43:14 matrix matrix-appservice-discord[250626]:     errno: -111,
Jan 25 03:43:14 matrix matrix-appservice-discord[250626]:     code: 'ECONNREFUSED',
Jan 25 03:43:14 matrix matrix-appservice-discord[250626]:     syscall: 'connect',
Jan 25 03:43:14 matrix matrix-appservice-discord[250626]:     address: '127.0.1.1',
Jan 25 03:43:14 matrix matrix-appservice-discord[250626]:     port: 443
Jan 25 03:43:14 matrix matrix-appservice-discord[250626]:   }
Jan 25 03:43:14 matrix matrix-appservice-discord[250626]: ]
Jan 25 03:43:14 matrix matrix-appservice-discord[250626]: Jan-25 03:43:14.190 [DiscordAS] error: A fatal error occurred during startup: Error: connect ECONNREFUSED 127.0.1.1:443
Jan 25 03:43:14 matrix matrix-appservice-discord[250626]:     at TCPConnectWrap.afterConnect [as oncomplete] (net.js:1146:16) {
Jan 25 03:43:14 matrix matrix-appservice-discord[250626]:   errno: -111,
Jan 25 03:43:14 matrix matrix-appservice-discord[250626]:   code: 'ECONNREFUSED',
Jan 25 03:43:14 matrix matrix-appservice-discord[250626]:   syscall: 'connect',
Jan 25 03:43:14 matrix matrix-appservice-discord[250626]:   address: '127.0.1.1',
Jan 25 03:43:14 matrix matrix-appservice-discord[250626]:   port: 443
Jan 25 03:43:14 matrix matrix-appservice-discord[250626]: }
Jan 25 03:43:14 matrix systemd[1]: matrix-appservice-discord.service: Main process exited, code=exited, status=1/FAILURE
Jan 25 03:43:14 matrix systemd[1]: matrix-appservice-discord.service: Failed with result 'exit-code'.

After adjusting /etc/hosts and restarting the bot, things started working.

Jan 26 '21 14:01 skepticalwaves

Interesting! What's your vars.yml file like (hiding secrets, of course)?

Jan 26 '21 14:01 spantaleev

Jan 26 '21 15:01 skepticalwaves

I don't see anything out of the ordinary that should trigger this.

Which Docker version are you on? docker version.

Jan 26 '21 15:01 spantaleev

root@matrix:~# docker version
Client: Docker Engine - Community
 Version:           20.10.2
 API version:       1.41
 Go version:        go1.13.15
 Git commit:        2291f61
 Built:             Mon Dec 28 16:17:43 2020
 OS/Arch:           linux/amd64
 Context:           default
 Experimental:      true

Server: Docker Engine - Community
 Engine:
  Version:          20.10.2
  API version:      1.41 (minimum version 1.12)
  Go version:       go1.13.15
  Git commit:       8891c58
  Built:            Mon Dec 28 16:15:19 2020
  OS/Arch:          linux/amd64
  Experimental:     false
 containerd:
  Version:          1.4.3
  GitCommit:        269548fa27e0089a8b8278fc4fc781d7f65a939b
 runc:
  Version:          1.0.0-rc92
  GitCommit:        ff819c7e9184c13b7c2607fe6c30ae19403a7aff
 docker-init:
  Version:          0.19.0
  GitCommit:        de40ad0

Jan 26 '21 15:01 skepticalwaves

Not sure why this happens.

I've tested on Ubuntu 20.04.1 LTS

With Docker 19.03.13, I can't reproduce it. docker run --network=some-custom-network .. also leads to the same (no /etc/hosts sharing). It's only with --net=host that the container's /etc/hosts file contains the entries from the host.

I have even upgraded that system to Docker 20.10.2, so it should be the same as yours. It's still the same result - custom /etc/hosts entries are only transferred when --net=host is used (which the playbook doesn't normally use).

Jan 27 '21 11:01 spantaleev