Etcd v3 api: Initialization client error

[happlins]

What happened?

etcd client v3: v3.5.4

I get an error when I initialize the client.

When I change RejectOldCluster to false everything works fine.

	cfg := clientv3.Config{
		Endpoints:            []string{"192.168.3.41:22479", "192.168.3.41:22379", "192.168.3.41:22579"},
		Username:             "root",
		Password:             "123456",
		RejectOldCluster:     true,
		DialTimeout:          5 * time.Second,
		DialKeepAliveTime:    5 * time.Second,
		DialKeepAliveTimeout: 5 * time.Second,
		AutoSyncInterval:     time.Minute,
	}

{"level":"warn","ts":"2022-08-08T16:05:33.632+0800","logger":"etcd-client","caller":"[email protected]/retry_interceptor.go:62","msg":"retrying of unary invoker failed","target":"etcd-endpoints://0xc0001fb500/192.168.3.41:22479","attempt":0,"error":"rpc error: code
 = Canceled desc = context canceled"}
{"level":"warn","ts":"2022-08-08T16:05:33.639+0800","logger":"etcd-client","caller":"[email protected]/retry_interceptor.go:62","msg":"retrying of unary invoker failed","target":"etcd-endpoints://0xc0001fb500/192.168.3.41:22479","attempt":0,"error":"rpc error: code
 = Canceled desc = context canceled"}

There is a problem at line 512 of client.go where it should not break when there are no err. If other goroutines are slow, context canceled will occur. But this problem does not affect normal use.

What did you expect to happen?

No error output.

How can we reproduce it (as minimally and precisely as possible)?

step 1

docker-compose up -d

docker-compose.yaml:

version: "3.0"
services:
  etcd1:
    image: quay.io/coreos/etcd:v3.5.4
    container_name: "etcd1"
    networks:
      internal:
        ipv4_address: 172.25.1.3
    ports:
      - "22379-22380:22379-22380"
    volumes:
      - ./etcd1:/var/etcd/etcd-data
    environment:
      - ETCD_NAME=etcd1
      - ETCD_DATA_DIR=/var/etcd/etcd-data
      - ETCD_LISTEN_CLIENT_URLS=http://0.0.0.0:22379
      - ETCD_LISTEN_PEER_URLS=http://0.0.0.0:22380
      - ETCD_ADVERTISE_CLIENT_URLS=http://192.168.3.41:22379
      - ETCD_INITIAL_ADVERTISE_PEER_URLS=http://172.25.1.3:22380
      - ETCD_INITIAL_CLUSTER_TOKEN=etcd-cluster
      - ETCD_INITIAL_CLUSTER=etcd1=http://172.25.1.3:22380,etcd2=http://172.25.1.4:22480,etcd3=http://172.25.1.5:22580
      - ETCD_INITIAL_CLUSTER_STATE=new      
  etcd2:
    image: quay.io/coreos/etcd:v3.5.4
    container_name: "etcd2"
    networks:
      internal:
        ipv4_address: 172.25.1.4
    ports:
      - "22479-22480:22479-22480"
    volumes:
      - ./etcd2:/var/etcd/etcd-data
    environment:
      - ETCD_NAME=etcd2
      - ETCD_DATA_DIR=/var/etcd/etcd-data
      - ETCD_LISTEN_CLIENT_URLS=http://0.0.0.0:22479
      - ETCD_LISTEN_PEER_URLS=http://0.0.0.0:22480
      - ETCD_ADVERTISE_CLIENT_URLS=http://192.168.3.41:22479
      - ETCD_INITIAL_ADVERTISE_PEER_URLS=http://172.25.1.4:22480
      - ETCD_INITIAL_CLUSTER_TOKEN=etcd-cluster
      - ETCD_INITIAL_CLUSTER=etcd1=http://172.25.1.3:22380,etcd2=http://172.25.1.4:22480,etcd3=http://172.25.1.5:22580
      - ETCD_INITIAL_CLUSTER_STATE=new      
  etcd3:
    image: quay.io/coreos/etcd:v3.5.4
    container_name: "etcd3"
    networks:
      internal:
        ipv4_address: 172.25.1.5
    ports:
      - "22579-22580:22579-22580"
    volumes:
      - ./etcd3:/var/etcd/etcd-data
    environment:
      - ETCD_NAME=etcd3
      - ETCD_DATA_DIR=/var/etcd/etcd-data
      - ETCD_LISTEN_CLIENT_URLS=http://0.0.0.0:22579
      - ETCD_LISTEN_PEER_URLS=http://0.0.0.0:22580
      - ETCD_ADVERTISE_CLIENT_URLS=http://192.168.3.41:22579
      - ETCD_INITIAL_ADVERTISE_PEER_URLS=http://172.25.1.5:22580
      - ETCD_INITIAL_CLUSTER_TOKEN=etcd-cluster
      - ETCD_INITIAL_CLUSTER=etcd1=http://172.25.1.3:22380,etcd2=http://172.25.1.4:22480,etcd3=http://172.25.1.5:22580
      - ETCD_INITIAL_CLUSTER_STATE=new       
networks:
  internal:
    driver: bridge
    ipam:
      config:
        - subnet: 172.25.1.0/16

step 2

Initialize client.

	cfg := clientv3.Config{
		Endpoints:            []string{"192.168.3.41:22479", "192.168.3.41:22379", "192.168.3.41:22579"},
		RejectOldCluster:     true,
		DialTimeout:          5 * time.Second,
		DialKeepAliveTime:    5 * time.Second,
		DialKeepAliveTimeout: 5 * time.Second,
		AutoSyncInterval:     time.Minute,
	}
       client, err := clientv3.New(cfg)

Anything else we need to know?

No response

Etcd version (please run commands below)

$ etcd --version
# paste output here
etcd Version: 3.5.4
Git SHA: 08407ff76
Go Version: go1.16.15
Go OS/Arch: linux/amd64

$ etcdctl version
# paste output here
etcdctl version: 3.5.4
API version: 3.5

Etcd configuration (command line flags or environment variables)

paste your configuration here

Etcd debug information (please run commands blow, feel free to obfuscate the IP address or FQDN in the output)

$ etcdctl member list -w table
# paste output here

+------------------+---------+-------+--------------------------+----------------------------+------------+
|        ID        | STATUS  | NAME  |        PEER ADDRS        |        CLIENT ADDRS        | IS LEARNER |
+------------------+---------+-------+--------------------------+----------------------------+------------+
|  4aea8f1af490926 | started | etcd1 | https://172.25.1.3:22380 | https://192.168.3.41:22379 |      false |
| 43038eadf883f7e2 | started | etcd3 | https://172.25.1.5:22580 | https://192.168.3.41:22579 |      false |
| 7191a6e6cc023b1a | started | etcd2 | https://172.25.1.4:22480 | https://192.168.3.41:22479 |      false |
+------------------+---------+-------+--------------------------+----------------------------+------------+

$ etcdctl --endpoints=<member list> endpoint status -w table
# paste output here
+----------------------------+------------------+---------+---------+-----------+------------+-----------+------------+--------------------+--------+
|          ENDPOINT          |        ID        | VERSION | DB SIZE | IS LEADER | IS LEARNER | RAFT TERM | RAFT INDEX | RAFT APPLIED INDEX | ERRORS |
+----------------------------+------------------+---------+---------+-----------+------------+-----------+------------+--------------------+--------+
| https://192.168.3.41:22379 |  4aea8f1af490926 |   3.5.4 |   20 kB |     false |      false |         3 |         42 |                 42 |        |
| https://192.168.3.41:22479 | 7191a6e6cc023b1a |   3.5.4 |   20 kB |     false |      false |         3 |         42 |                 42 |        |
| https://192.168.3.41:22579 | 43038eadf883f7e2 |   3.5.4 |   20 kB |      true |      false |         3 |         42 |                 42 |        |
+----------------------------+------------------+---------+---------+-----------+------------+-----------+------------+--------------------+--------+

Relevant log output

{"level":"warn","ts":"2022-08-08T16:05:33.632+0800","logger":"etcd-client","caller":"[email protected]/retry_interceptor.go:62","msg":"retrying of unary invoker failed","target":"etcd-endpoints://0xc0001fb500/192.168.3.41:22479","attempt":0,"error":"rpc error: code
 = Canceled desc = context canceled"}
{"level":"warn","ts":"2022-08-08T16:05:33.639+0800","logger":"etcd-client","caller":"[email protected]/retry_interceptor.go:62","msg":"retrying of unary invoker failed","target":"etcd-endpoints://0xc0001fb500/192.168.3.41:22479","attempt":0,"error":"rpc error: code
 = Canceled desc = context canceled"}

[ahrtr]

Thanks @happlins for raising this issue. It looks like a real issue to me. The loop you pointed out should only break when err != nil. It means there is no need to wait for the result of other endpoints, because it will return an error anyway.

Please feel free to deliver a PR for this.

[halegreen]

Hi, is this issue revoled?