Error "listen tcp 127.0.0.1:2380: bind: address already in use" when running ETCD container

[dc7303]

What happened?

I tried to use the ETCD container on an arm MacBook, but I'm having the same problem as issue #14209. (listen tcp 127.0.0.1:2380: bind: address already in use) In my case, the same issue occurs even after rebooting the computer. Instead, restart the container and it will run normally.

What did you expect to happen?

The container was expected to run normally.

How can we reproduce it (as minimally and precisely as possible)?

Steps 1-3 below are the process I was talking about. (I tested it by referring to the 'Connecting to other containers' section in the README provided by the docker hub.) step 1 $ docker network create app-tier --driver bridge

step 2

$ docker run -d --name Etcd-server \
    --network app-tier \
    --publish 2379:2379 \
    --publish 2380:2380 \
    --env ALLOW_NONE_AUTHENTICATION=yes \
    --env ETCD_ADVERTISE_CLIENT_URLS=http://etcd-server:2379 \
    bitnami/etcd:latest

$ docker logs Etcd-server

...
{"level":"info","ts":"2022-08-06T09:44:27.439Z","caller":"embed/etcd.go:131","msg":"configuring peer listeners","listen-peer-urls":["http://localhost:2380"]}
{"level":"info","ts":"2022-08-06T09:44:27.449Z","caller":"embed/etcd.go:368","msg":"closing etcd server","name":"default","data-dir":"/bitnami/etcd/data","advertise-peer-urls":["http://localhost:2380"],"advertise-client-urls":["http://etcd-server:2379"]}
{"level":"info","ts":"2022-08-06T09:44:27.450Z","caller":"embed/etcd.go:370","msg":"closed etcd server","name":"default","data-dir":"/bitnami/etcd/data","advertise-peer-urls":["http://localhost:2380"],"advertise-client-urls":["http://etcd-server:2379"]}
{"level":"fatal","ts":"2022-08-06T09:44:27.450Z","caller":"etcdmain/etcd.go:204","msg":"discovery failed","error":"listen tcp 127.0.0.1:2380: bind: address already in use","stacktrace":"go.etcd.io/etcd/server/v3/etcdmain.startEtcdOrProxyV2\n\t/go/src/go.etcd.io/etcd/release/etcd/server/etcdmain/etcd.go:204\ngo.etcd.io/etcd/server/v3/etcdmain.Main\n\t/go/src/go.etcd.io/etcd/release/etcd/server/etcdmain/main.go:40\nmain.main\n\t/go/src/go.etcd.io/etcd/release/etcd/server/main.go:32\nruntime.main\n\t/go/gos/go1.16.15/src/runtime/proc.go:225"}

step 3

$ docker restart Etcd-server

$ docker ps

CONTAINER ID   IMAGE                 COMMAND                  CREATED          STATUS          PORTS                              NAMES
9dd6bbbd310e   bitnami/etcd:latest   "/opt/bitnami/script…"   22 minutes ago   Up 22 minutes   0.0.0.0:2379-2380->2379-2380/tcp   Etcd-server

Anything else we need to know?

No response

Etcd version (please run commands below)

$ etcd --version
etcd Version: 3.5.4
Git SHA: 08407ff76
Go Version: go1.16.15
Go OS/Arch: linux/amd64

$ etcdctl version
etcdctl version: 3.5.4
API version: 3.5

Etcd configuration (command line flags or environment variables)

ALLOW_NONE_AUTHENTICATION=yes ETCD_ADVERTISE_CLIENT_URLS=http://etcd-server:2379

Etcd debug information (please run commands blow, feel free to obfuscate the IP address or FQDN in the output)

$ etcdctl member list -w table
# paste output here

$ etcdctl --endpoints=<member list> endpoint status -w table
# paste output here

Relevant log output

etcd 09:44:22.78
etcd 09:44:22.80 Welcome to the Bitnami etcd container
etcd 09:44:22.82 Subscribe to project updates by watching https://github.com/bitnami/containers
etcd 09:44:22.84 Submit issues and feature requests at https://github.com/bitnami/containers/issues
etcd 09:44:22.86
etcd 09:44:22.88 INFO  ==> ** Starting etcd setup **
etcd 09:44:23.00 INFO  ==> Validating settings in ETCD_* env vars..
etcd 09:44:23.03 WARN  ==> You set the environment variable ALLOW_NONE_AUTHENTICATION=yes. For safety reasons, do not use this flag in a production environment.
etcd 09:44:23.07 INFO  ==> Initializing etcd
etcd 09:44:23.09 INFO  ==> Generating etcd config file using env variables
etcd 09:44:23.21 INFO  ==> There is no data from previous deployments
etcd 09:44:23.23 INFO  ==> Obtaining cluster member ID
etcd 09:44:23.28 INFO  ==> Starting etcd in background
etcd 09:44:26.99 INFO  ==> Stored member ID: 8e9e05c52164694d
etcd 09:44:27.08 INFO  ==> ** etcd setup finished! **

etcd 09:44:27.19 INFO  ==> ** Starting etcd **
{"level":"info","ts":"2022-08-06T09:44:27.426Z","caller":"flags/flag.go:113","msg":"recognized and used environment variable","variable-name":"ETCD_ADVERTISE_CLIENT_URLS","variable-value":"http://etcd-server:2379"}
{"level":"info","ts":"2022-08-06T09:44:27.429Z","caller":"flags/flag.go:113","msg":"recognized and used environment variable","variable-name":"ETCD_AUTO_TLS","variable-value":"false"}
{"level":"info","ts":"2022-08-06T09:44:27.430Z","caller":"flags/flag.go:113","msg":"recognized and used environment variable","variable-name":"ETCD_CLIENT_CERT_AUTH","variable-value":"false"}
{"level":"info","ts":"2022-08-06T09:44:27.430Z","caller":"flags/flag.go:113","msg":"recognized and used environment variable","variable-name":"ETCD_DATA_DIR","variable-value":"/bitnami/etcd/data"}
{"level":"info","ts":"2022-08-06T09:44:27.431Z","caller":"flags/flag.go:113","msg":"recognized and used environment variable","variable-name":"ETCD_LISTEN_CLIENT_URLS","variable-value":"http://0.0.0.0:2379"}
{"level":"info","ts":"2022-08-06T09:44:27.431Z","caller":"flags/flag.go:113","msg":"recognized and used environment variable","variable-name":"ETCD_LOG_LEVEL","variable-value":"info"}
{"level":"info","ts":"2022-08-06T09:44:27.431Z","caller":"flags/flag.go:113","msg":"recognized and used environment variable","variable-name":"ETCD_PEER_AUTO_TLS","variable-value":"false"}
{"level":"warn","ts":"2022-08-06T09:44:27.431Z","caller":"flags/flag.go:93","msg":"unrecognized environment variable","environment-variable":"ETCD_INITIAL_ADVERTISE_PEER_URLS="}
{"level":"warn","ts":"2022-08-06T09:44:27.432Z","caller":"flags/flag.go:93","msg":"unrecognized environment variable","environment-variable":"ETCD_DAEMON_USER=etcd"}
{"level":"warn","ts":"2022-08-06T09:44:27.432Z","caller":"flags/flag.go:93","msg":"unrecognized environment variable","environment-variable":"ETCD_NEW_MEMBERS_ENV_FILE=/bitnami/etcd/data/new_member_envs"}
{"level":"warn","ts":"2022-08-06T09:44:27.432Z","caller":"flags/flag.go:93","msg":"unrecognized environment variable","environment-variable":"ETCD_CERT_FILE="}
{"level":"warn","ts":"2022-08-06T09:44:27.432Z","caller":"flags/flag.go:93","msg":"unrecognized environment variable","environment-variable":"ETCD_INITIAL_CLUSTER="}
{"level":"warn","ts":"2022-08-06T09:44:27.432Z","caller":"flags/flag.go:93","msg":"unrecognized environment variable","environment-variable":"ETCD_BASE_DIR=/opt/bitnami/etcd"}
{"level":"warn","ts":"2022-08-06T09:44:27.432Z","caller":"flags/flag.go:93","msg":"unrecognized environment variable","environment-variable":"ETCD_INITIAL_CLUSTER_STATE="}
{"level":"warn","ts":"2022-08-06T09:44:27.432Z","caller":"flags/flag.go:93","msg":"unrecognized environment variable","environment-variable":"ETCD_TMP_DIR=/opt/bitnami/etcd/tmp"}
{"level":"warn","ts":"2022-08-06T09:44:27.432Z","caller":"flags/flag.go:93","msg":"unrecognized environment variable","environment-variable":"ETCD_DISABLE_PRESTOP=no"}
{"level":"warn","ts":"2022-08-06T09:44:27.432Z","caller":"flags/flag.go:93","msg":"unrecognized environment variable","environment-variable":"ETCD_INIT_SNAPSHOTS_DIR=/init-snapshot"}
{"level":"warn","ts":"2022-08-06T09:44:27.432Z","caller":"flags/flag.go:93","msg":"unrecognized environment variable","environment-variable":"ETCD_INIT_SNAPSHOT_FILENAME="}
{"level":"warn","ts":"2022-08-06T09:44:27.432Z","caller":"flags/flag.go:93","msg":"unrecognized environment variable","environment-variable":"ETCD_START_FROM_SNAPSHOT=no"}
{"level":"warn","ts":"2022-08-06T09:44:27.432Z","caller":"flags/flag.go:93","msg":"unrecognized environment variable","environment-variable":"ETCD_DAEMON_GROUP=etcd"}
{"level":"warn","ts":"2022-08-06T09:44:27.433Z","caller":"flags/flag.go:93","msg":"unrecognized environment variable","environment-variable":"ETCD_CONF_DIR=/opt/bitnami/etcd/conf"}
{"level":"warn","ts":"2022-08-06T09:44:27.433Z","caller":"flags/flag.go:93","msg":"unrecognized environment variable","environment-variable":"ETCD_KEY_FILE="}
{"level":"warn","ts":"2022-08-06T09:44:27.433Z","caller":"flags/flag.go:93","msg":"unrecognized environment variable","environment-variable":"ETCD_DISASTER_RECOVERY=no"}
{"level":"warn","ts":"2022-08-06T09:44:27.433Z","caller":"flags/flag.go:93","msg":"unrecognized environment variable","environment-variable":"ETCD_CLUSTER_DOMAIN="}
{"level":"warn","ts":"2022-08-06T09:44:27.433Z","caller":"flags/flag.go:93","msg":"unrecognized environment variable","environment-variable":"ETCD_ROOT_PASSWORD="}
{"level":"warn","ts":"2022-08-06T09:44:27.433Z","caller":"flags/flag.go:93","msg":"unrecognized environment variable","environment-variable":"ETCD_NAME="}
{"level":"warn","ts":"2022-08-06T09:44:27.433Z","caller":"flags/flag.go:93","msg":"unrecognized environment variable","environment-variable":"ETCD_INITIAL_CLUSTER_TOKEN="}
{"level":"warn","ts":"2022-08-06T09:44:27.433Z","caller":"flags/flag.go:93","msg":"unrecognized environment variable","environment-variable":"ETCD_VOLUME_DIR=/bitnami/etcd"}
{"level":"warn","ts":"2022-08-06T09:44:27.433Z","caller":"flags/flag.go:93","msg":"unrecognized environment variable","environment-variable":"ETCD_BIN_DIR=/opt/bitnami/etcd/bin"}
{"level":"warn","ts":"2022-08-06T09:44:27.433Z","caller":"flags/flag.go:93","msg":"unrecognized environment variable","environment-variable":"ETCD_SNAPSHOTS_DIR=/snapshots"}
{"level":"warn","ts":"2022-08-06T09:44:27.434Z","caller":"flags/flag.go:93","msg":"unrecognized environment variable","environment-variable":"ETCD_ON_K8S=no"}
{"level":"warn","ts":"2022-08-06T09:44:27.434Z","caller":"flags/flag.go:93","msg":"unrecognized environment variable","environment-variable":"ETCD_LISTEN_PEER_URLS="}
{"level":"warn","ts":"2022-08-06T09:44:27.434Z","caller":"flags/flag.go:93","msg":"unrecognized environment variable","environment-variable":"ETCD_CONF_FILE=/opt/bitnami/etcd/conf/etcd.yaml"}
{"level":"warn","ts":"2022-08-06T09:44:27.434Z","caller":"flags/flag.go:93","msg":"unrecognized environment variable","environment-variable":"ETCD_DISABLE_STORE_MEMBER_ID=no"}
{"level":"warn","ts":"2022-08-06T09:44:27.434Z","caller":"flags/flag.go:93","msg":"unrecognized environment variable","environment-variable":"ETCD_TRUSTED_CA_FILE="}
{"level":"info","ts":"2022-08-06T09:44:27.436Z","caller":"etcdmain/etcd.go:73","msg":"Running: ","args":["etcd"]}
{"level":"warn","ts":"2022-08-06T09:44:27.438Z","caller":"etcdmain/etcd.go:446","msg":"found invalid file under data directory","filename":"member_id","data-dir":"/bitnami/etcd/data"}
{"level":"info","ts":"2022-08-06T09:44:27.438Z","caller":"etcdmain/etcd.go:116","msg":"server has been already initialized","data-dir":"/bitnami/etcd/data","dir-type":"member"}
{"level":"info","ts":"2022-08-06T09:44:27.439Z","caller":"embed/etcd.go:131","msg":"configuring peer listeners","listen-peer-urls":["http://localhost:2380"]}
{"level":"info","ts":"2022-08-06T09:44:27.449Z","caller":"embed/etcd.go:368","msg":"closing etcd server","name":"default","data-dir":"/bitnami/etcd/data","advertise-peer-urls":["http://localhost:2380"],"advertise-client-urls":["http://etcd-server:2379"]}
{"level":"info","ts":"2022-08-06T09:44:27.450Z","caller":"embed/etcd.go:370","msg":"closed etcd server","name":"default","data-dir":"/bitnami/etcd/data","advertise-peer-urls":["http://localhost:2380"],"advertise-client-urls":["http://etcd-server:2379"]}
{"level":"fatal","ts":"2022-08-06T09:44:27.450Z","caller":"etcdmain/etcd.go:204","msg":"discovery failed","error":"listen tcp 127.0.0.1:2380: bind: address already in use","stacktrace":"go.etcd.io/etcd/server/v3/etcdmain.startEtcdOrProxyV2\n\t/go/src/go.etcd.io/etcd/release/etcd/server/etcdmain/etcd.go:204\ngo.etcd.io/etcd/server/v3/etcdmain.Main\n\t/go/src/go.etcd.io/etcd/release/etcd/server/etcdmain/main.go:40\nmain.main\n\t/go/src/go.etcd.io/etcd/release/etcd/server/main.go:32\nruntime.main\n\t/go/gos/go1.16.15/src/runtime/proc.go:225"}

[rafi]

This seems to be an issue with https://github.com/bitnami/containers/tree/main/bitnami/etcd

[ahrtr]

I have the same feeling as @rafi , this shouldn't be an issue of etcd, you need to make sure there is no port conflict.

[yujintang]

if using bitnami/etcd single mode ,you shoule remove ETCD_ADVERTISE_CLIENT_URLS and resolve this problems.

[huanggze]

if using bitnami/etcd single mode ,you shoule remove ETCD_ADVERTISE_CLIENT_URLS and resolve this problems.

unfortunately, not working for my docker-compose.yaml :(

[steve-solun]

@huanggze @yujintang @serathius @ahrtr @dc7303 anybody found a solution? I observe the same behavior.

[li-jin-gou]

use quay.io/coreos/etcd instead of bitnami/etcd @steve-solun and i suspect it is a problem with m1/arm64.

[serathius]

Good point by @li-jin-gou, etcd project didn't create bitnami/etcd image thus we are not aware what changes were made and we cannot support it.

[Leslie-Wong-H]

Encountered the same problem when running milvus-etcd: Error response from daemon: driver failed programming external connectivity on endpoint milvus-etcd (d395b72fa9d3793e6f871be42d145a2daceca0c64e91a64a69e4652f09a0e1ca): listen tcp4 0.0.0.0:2379: bind: address already in use The docker-compose yaml:

  etcd:
    container_name: milvus-etcd
    image: quay.io/coreos/etcd:v3.5.0
    restart: unless-stopped
    environment:
      - ETCD_AUTO_COMPACTION_MODE=revision
      - ETCD_AUTO_COMPACTION_RETENTION=1000
      - ETCD_QUOTA_BACKEND_BYTES=4294967296
    volumes:
      - ${DOCKER_VOLUME_DIRECTORY:-.}/volumes/etcd:/etcd
    command: etcd -advertise-client-urls=http://127.0.0.1:2379 -listen-client-urls http://0.0.0.0:2379 --data-dir /etcd
    ports:
      - "2379:2379"

Resolved by removing the ports declaration.

  etcd:
    container_name: milvus-etcd
    image: quay.io/coreos/etcd:v3.5.0
    restart: unless-stopped
    environment:
      - ETCD_AUTO_COMPACTION_MODE=revision
      - ETCD_AUTO_COMPACTION_RETENTION=1000
      - ETCD_QUOTA_BACKEND_BYTES=4294967296
      - ETCD_SNAPSHOT_COUNT=50000
    volumes:
      - ${DOCKER_VOLUME_DIRECTORY:-.}/volumes/etcd:/etcd
    command: etcd -advertise-client-urls=http://127.0.0.1:2379 -listen-client-urls http://0.0.0.0:2379 --data-dir /etcd

[Akkarine]

If you are using docker desktop on windows and enabled kubernetes, it could be tricky. K8s uses 2379-2380 ports: https://kubernetes.io/docs/reference/networking/ports-and-protocols/ But looks like you can't see it in bounded processes (resolved on docker desctop level).

So you need to choose another ports.