resolv.conf keeps getting overwritten by another process

[ctrlbrk42]

Running GCP VM

My /etc/resolv.conf keeps getting overwritten. I suspect its due to DHCP lease renewal from GCP.

The Docker container doesn't detect this, and leaves the resolv.conf broken (no name resolution to containers from the host).

Could you consider monitoring resolv.conf for changes and then reverting them back to the proper config? Would be nice in my type of case.

[royduin]

Copy the content of /etc/resolv.conf and then remove it with: sudo rm /etc/resolv.conf, next recreate the file with Nano or Vim: vi /etc/resolv.conf and paste the content. Then restart the container.

[mageddo]

@ctrlbrk42 Acutally DPS watches for resolv.conf changes, DPS updates it every 20 seconds by default, you can check this by enabling debug log level. Two things can be happening here:

1. your /etc/resolv.conf is not volumed to the docker container
2. As an external program is writing to /etc/resolv.conf docker is losing the reference and the file volume stop working, this is a known issue

You can easily fix it by following the solution at this issue or reading the getting started instructions

[mageddo]

Please let me know if it works

[m1cx]

On Ubuntu 20.04, with the Docker command given in "Getting started" in case of periodical overwrite, I get:

$ docker run --rm --hostname dns.mageddo --name dns-proxy-server -p 5380:5380   -v /opt/dns-proxy-server/conf:/app/conf   -v /var/run/docker.sock:/var/run/docker.sock   -v /etc:/host/etc   -e MG_RESOLVCONF=/host/etc/resolv.conf   defreitas/dns-proxy-server
2020/07/10 07:56:41.053007 DEBUG f=local.go:29 pkg=github.com/mageddo/dns-proxy-server/events/local m=LoadConfiguration status=loading, path=/app/conf/config.json
2020/07/10 07:56:41.053990 DEBUG f=local.go:29 pkg=github.com/mageddo/dns-proxy-server/events/local m=LoadConfiguration status=loading, path=/app/conf/config.json
2020/07/10 07:56:41.054158 WARNING f=logging.go:120 pkg=github.com/mageddo/dns-proxy-server/vendor/github.com/mageddo/go-logging m=SetLevel status=log-level-changed, log-level=6
2020/07/10 07:56:41.058643 INFO f=index.go:65 pkg=github.com/mageddo/dns-proxy-server/vendor/github.com/mageddo/go-httpmap m=handle status=registering, path=GET /v1/caches
2020/07/10 07:56:41.058671 INFO f=index.go:65 pkg=github.com/mageddo/dns-proxy-server/vendor/github.com/mageddo/go-httpmap m=handle status=registering, path=GET /v1/caches/size
2020/07/10 07:56:41.058683 INFO f=index.go:65 pkg=github.com/mageddo/dns-proxy-server/vendor/github.com/mageddo/go-httpmap m=handle status=registering, path=GET /env/active
2020/07/10 07:56:41.058692 INFO f=index.go:65 pkg=github.com/mageddo/dns-proxy-server/vendor/github.com/mageddo/go-httpmap m=handle status=registering, path=PUT /env/active
2020/07/10 07:56:41.058700 INFO f=index.go:65 pkg=github.com/mageddo/dns-proxy-server/vendor/github.com/mageddo/go-httpmap m=handle status=registering, path=GET /env/
2020/07/10 07:56:41.058713 INFO f=index.go:65 pkg=github.com/mageddo/dns-proxy-server/vendor/github.com/mageddo/go-httpmap m=handle status=registering, path=POST /env/
2020/07/10 07:56:41.058723 INFO f=index.go:65 pkg=github.com/mageddo/dns-proxy-server/vendor/github.com/mageddo/go-httpmap m=handle status=registering, path=DELETE /env/
2020/07/10 07:56:41.058731 INFO f=index.go:65 pkg=github.com/mageddo/dns-proxy-server/vendor/github.com/mageddo/go-httpmap m=handle status=registering, path=GET /hostname/
2020/07/10 07:56:41.058742 INFO f=index.go:65 pkg=github.com/mageddo/dns-proxy-server/vendor/github.com/mageddo/go-httpmap m=handle status=registering, path=GET /hostname/find/
2020/07/10 07:56:41.058761 INFO f=index.go:65 pkg=github.com/mageddo/dns-proxy-server/vendor/github.com/mageddo/go-httpmap m=handle status=registering, path=POST /hostname/
2020/07/10 07:56:41.058770 INFO f=index.go:65 pkg=github.com/mageddo/dns-proxy-server/vendor/github.com/mageddo/go-httpmap m=handle status=registering, path=PUT /hostname/
2020/07/10 07:56:41.058779 INFO f=index.go:65 pkg=github.com/mageddo/dns-proxy-server/vendor/github.com/mageddo/go-httpmap m=handle status=registering, path=DELETE /hostname/
2020/07/10 07:56:41.058788 INFO f=index.go:65 pkg=github.com/mageddo/dns-proxy-server/vendor/github.com/mageddo/go-httpmap m=handle status=registering, path=GET /
2020/07/10 07:56:41.058797 INFO f=index.go:65 pkg=github.com/mageddo/dns-proxy-server/vendor/github.com/mageddo/go-httpmap m=handle status=registering, path=GET /configuration/
2020/07/10 07:56:41.058812 INFO f=index.go:65 pkg=github.com/mageddo/dns-proxy-server/vendor/github.com/mageddo/go-httpmap m=handle status=registering, path=DELETE /network/disconnect-containers/
2020/07/10 07:56:41.059130 WARNING f=dns.go:160 pkg=main m=main server started
2020/07/10 07:56:41.059616 INFO f=asm_amd64.s:1338 pkg=runtime m=goexit uuid=0fcd581a11 status=setup-default-dns, setup-dns=true
2020/07/10 07:56:41.060005 ERROR f=asm_amd64.s:1338 pkg=runtime m=goexit  status=cant-turn-default-dns 
open /host/etc/resolv.conf: no such file or directory
goroutine 25 [running]:
runtime/debug.Stack(0xc000310900, 0xc0001d7b40, 0x35)
	/usr/local/go/src/runtime/debug/stack.go:24 +0x9d
github.com/mageddo/dns-proxy-server/vendor/github.com/mageddo/go-logging.transformErrorInStackTrace(0xc0003261e0, 0x2, 0x2, 0x0, 0x203000, 0xd, 0x2)
	/app/src/github.com/mageddo/dns-proxy-server/vendor/github.com/mageddo/go-logging/nativelogger.go:72 +0x10c
github.com/mageddo/dns-proxy-server/vendor/github.com/mageddo/go-logging.(*defaultLogger).print(0xc0000a06e0, 0xc0003261e0, 0x2, 0x2, 0x3)
	/app/src/github.com/mageddo/dns-proxy-server/vendor/github.com/mageddo/go-logging/nativelogger.go:92 +0x63
github.com/mageddo/dns-proxy-server/vendor/github.com/mageddo/go-logging.(*defaultLogger).Error(0xc0000a06e0, 0xc0003261e0, 0x2, 0x2)
	/app/src/github.com/mageddo/dns-proxy-server/vendor/github.com/mageddo/go-logging/nativelogger.go:49 +0x52
github.com/mageddo/dns-proxy-server/vendor/github.com/mageddo/go-logging.Error(0xc0003261e0, 0x2, 0x2)
	/app/src/github.com/mageddo/dns-proxy-server/vendor/github.com/mageddo/go-logging/logging.go:91 +0x7e
main.main.func2()
	/app/src/github.com/mageddo/dns-proxy-server/dns.go:151 +0x1af
created by main.main
	/app/src/github.com/mageddo/dns-proxy-server/dns.go:145 +0x1c0

2020/07/10 07:56:41.060020 ERROR f=exitcodes.go:19 pkg=github.com/mageddo/dns-proxy-server/utils/exitcodes m=Exit status=exiting, code=1
2020/07/10 07:56:41.060031 INFO f=exitcodes.go:21 pkg=github.com/mageddo/dns-proxy-server/utils/exitcodes m=Exit  status=msg-posted
2020/07/10 07:56:41.060050 WARNING f=dns.go:162 pkg=main m=main status=exiting ;) signal=hangup
2020/07/10 07:56:41.060066 INFO f=resolvconf.go:23 pkg=github.com/mageddo/dns-proxy-server/resolvconf m=RestoreResolvconfToDefault status=begin
2020/07/10 07:56:41.060141 INFO f=resolvconf.go:26 pkg=github.com/mageddo/dns-proxy-server/resolvconf m=RestoreResolvconfToDefault status=success, err=%!v(MISSING)
open /host/etc/resolv.conf: no such file or directory
goroutine 1 [running]:
runtime/debug.Stack(0xc000310960, 0xc0001d7ec0, 0x35)
	/usr/local/go/src/runtime/debug/stack.go:24 +0x9d
github.com/mageddo/dns-proxy-server/vendor/github.com/mageddo/go-logging.transformErrorInStackTrace(0xc00030c980, 0x1, 0x1, 0xc0001cfdd0, 0x0, 0x0, 0x0)
	/app/src/github.com/mageddo/dns-proxy-server/vendor/github.com/mageddo/go-logging/nativelogger.go:72 +0x10c
github.com/mageddo/dns-proxy-server/vendor/github.com/mageddo/go-logging.(*defaultLogger).fPrint(0xc0000a06e0, 0x875967, 0x16, 0xc00030c980, 0x1, 0x1, 0x6)
	/app/src/github.com/mageddo/dns-proxy-server/vendor/github.com/mageddo/go-logging/nativelogger.go:87 +0x144
github.com/mageddo/dns-proxy-server/vendor/github.com/mageddo/go-logging.(*defaultLogger).Infof(0xc0000a06e0, 0x875967, 0x16, 0xc00030c980, 0x1, 0x1)
	/app/src/github.com/mageddo/dns-proxy-server/vendor/github.com/mageddo/go-logging/nativelogger.go:37 +0x66
github.com/mageddo/dns-proxy-server/vendor/github.com/mageddo/go-logging.Infof(0x875967, 0x16, 0xc00030c980, 0x1, 0x1)
	/app/src/github.com/mageddo/dns-proxy-server/vendor/github.com/mageddo/go-logging/logging.go:73 +0x96
github.com/mageddo/dns-proxy-server/resolvconf.RestoreResolvconfToDefault(0x877ee2, 0x1b)
	/app/src/github.com/mageddo/dns-proxy-server/resolvconf/resolvconf.go:25 +0xf1
main.main()
	/app/src/github.com/mageddo/dns-proxy-server/dns.go:162 +0x29f

I have been using Mageddo DNS both as Docker service (standard command) and as a native service - both stop resolving after some time.

[unclesamwk]

+1

[oeuftete]

@m1cx To use the instructions at http://mageddo.github.io/dns-proxy-server/latest/en/1-getting-started/running-it/#on-docker for when resolv.conf stops getting updated, I had to create my own Dockerfile:

#  The following reproduces the original Dockerfile, removes the `VOLUME`
#  command, and adds the directory creation.
FROM debian:10-slim AS BUILDER
ENV TMP_NAME=/tmp/dns-proxy-server.tgz
RUN apt-get update && apt-get install -y curl &&\
    curl -L https://github.com/mageddo/dns-proxy-server/releases/download/2.19.0/dns-proxy-server-linux-amd64-2.19.0.tgz > $TMP_NAME && \
    mkdir /app && tar -xvf $TMP_NAME -C /app

FROM debian:10-slim
LABEL dps.container=true
WORKDIR /app
COPY --from=BUILDER /app /app
RUN mkdir -p /host/etc
ENTRYPOINT ["/app/dns-proxy-server"]

---

Something like the following looks appealing, but the VOLUME ["/var/run/docker.sock", "/var/run/docker.sock"] in the original prevents this; the resulting image ends up with a created /var/run/docker.sock directory which blocks the mount you need to run.

# FROM defreitas/dns-proxy-server:latest
# RUN mkdir -p /host/etc

I believe the changes in #195, among other things, would make this easier to do locally.

Hope this helps.

[m1cx]

@oeuftete I tried to apply your suggestion, but ended up with exactly the same error while running it. Are your instructions complete?

I'm also not sure I understand the second part of your post. Is it finally possible to mount whole /etc into container so recreated resolv.conf is picked up, or not - with your solution?

[cpedrero]

Thanks @royduin I removed resolv.conf and it works!

[mageddo]

1=> In the current version 3.14.x DPS supports systemd-resolved which can be one of the responsibles to cause this issue, in that case is better to use it, see the how to.

2=> Run DPS with -v /etc/:/host/etc can also fix the issue where resolv.conf stop of being updated, MG_RESOLVCONF env is now looking at the following paths: /host/etc/systemd/resolved.conf,/host/etc/resolv.conf,/etc/systemd/resolved.conf,/etc/resolv.conf

3=> DPS actually also updates resolv.conf every 20 seconds

• ref 1
• ref 2

[mageddo]

I think that's enough to consider this as fixed, right?