help request: apisix connet etcd ipv6 address fail
Description
stateful-set etcd cluster with ipv6 address, when apisix pod init, it failed because connect to etcd domain url(headless service) fail. It shows
request etcd endpoint 'http://etcd-0.etcd-headless.apisix.svc.cluster.local:2379/version' error, host or service not provided, or not known
Warning! Request etcd endpoint 'http://etcd-0.etcd-headless.apisix.svc.cluster.local:2379/version' error, host or service not provided, or not known, retry time=1
curl the etcd url in apisix pod is success. and I change the etcd url from string to ip-address(in the configmap), it shows
/usr/local/openresty/luajit/bin/luajit ./apisix/cli/apisix.lua init
/usr/local/openresty/luajit/bin/luajit ./apisix/cli/apisix.lua init_etcd
request etcd endpoint 'http://[fd00:c5a6::13:61c]:2379/version' error, Address family for hostname not supported
request etcd endpoint 'http://[fd00:c5a6::1b:7478]:2379/version' error, Address family for hostname not supported
request etcd endpoint 'http://[fd00:c5a6::27:3dfb]:2379/version' error, Address family for hostname not supported
all etcd nodes are unavailable
Warning! Request etcd endpoint 'http://[fd00:c5a6::13:61c]:2379/version' error, Address family for hostname not supported, retry time=1
and the openresty version is with ipv6:
[root@5927f3188b04 resty]# /usr/local/openresty/bin/openresty -p /usr/local/apisix -V
nginx version: openresty/1.21.4.1
built by gcc 9.3.1 20200408 (Red Hat 9.3.1-2) (GCC)
built with OpenSSL 1.1.1s 1 Nov 2022
TLS SNI support enabled
configure arguments: --prefix=/usr/local/openresty/nginx --with-cc-opt='-O2 -DAPISIX_BASE_VER=1.21.4.1.7 -DNGX_GRPC_CLI_ENGINE_PATH=/usr/
local/openresty/libgrpc_engine.so -DNGX_HTTP_GRPC_CLI_ENGINE_PATH=/usr/local/openresty/libgrpc_engine.so -DNGX_LUA_ABORT_AT_PANIC -I/usr/
local/openresty/zlib/include -I/usr/local/openresty/pcre/include -I/usr/local/openresty/openssl111/include' --add-module=../ngx_devel_kit
-0.3.1 --add-module=../echo-nginx-module-0.62 --add-module=../xss-nginx-module-0.06 --add-module=../ngx_coolkit-0.2 --add-module=../set-m
isc-nginx-module-0.33 --add-module=../form-input-nginx-module-0.12 --add-module=../encrypted-session-nginx-module-0.09 --add-module=../sr
cache-nginx-module-0.32 --add-module=../ngx_lua-0.10.21 --add-module=../ngx_lua_upstream-0.07 --add-module=../headers-more-nginx-module-0
.33 --add-module=../array-var-nginx-module-0.05 --add-module=../memc-nginx-module-0.19 --add-module=../redis2-nginx-module-0.15 --add-mod
ule=../redis-nginx-module-0.3.9 --add-module=../ngx_stream_lua-0.0.11 --with-ld-opt='-Wl,-rpath,/usr/local/openresty/luajit/lib -Wl,-rpat
h,/usr/local/openresty/wasmtime-c-api/lib -L/usr/local/openresty/zlib/lib -L/usr/local/openresty/pcre/lib -L/usr/local/openresty/openssl1
11/lib -Wl,-rpath,/usr/local/openresty/zlib/lib:/usr/local/openresty/pcre/lib:/usr/local/openresty/openssl111/lib' --add-module=/tmp/tmp.
OdVMn6DSl0/openresty-1.21.4.1/../mod_dubbo-1.0.2 --add-module=/tmp/tmp.OdVMn6DSl0/openresty-1.21.4.1/../ngx_multi_upstream_module-1.1.1 -
-add-module=/tmp/tmp.OdVMn6DSl0/openresty-1.21.4.1/../apisix-nginx-module-1.12.0 --add-module=/tmp/tmp.OdVMn6DSl0/openresty-1.21.4.1/../a
pisix-nginx-module-1.12.0/src/stream --add-module=/tmp/tmp.OdVMn6DSl0/openresty-1.21.4.1/../apisix-nginx-module-1.12.0/src/meta --add-mod
ule=/tmp/tmp.OdVMn6DSl0/openresty-1.21.4.1/../wasm-nginx-module-0.6.4 --add-module=/tmp/tmp.OdVMn6DSl0/openresty-1.21.4.1/../lua-var-ngin
x-module-v0.5.3 --add-module=/tmp/tmp.OdVMn6DSl0/openresty-1.21.4.1/../grpc-client-nginx-module-v0.4.2 --with-poll_module --with-pcre-jit
--with-stream --with-stream_ssl_module --with-stream_ssl_preread_module --with-http_v2_module --without-mail_pop3_module --without-mail_
imap_module --without-mail_smtp_module --with-http_stub_status_module --with-http_realip_module --with-http_addition_module --with-http_a
uth_request_module --with-http_secure_link_module --with-http_random_index_module --with-http_gzip_static_module --with-http_sub_module -
-with-http_dav_module --with-http_flv_module --with-http_mp4_module --with-http_gunzip_module --with-threads --with-compat --with-ipv6 --
with-stream --with-http_ssl_module
now i am confused
- init-etcd seems that it just use socket http lib send request(https://github.com/apache/apisix/blob/master/apisix/cli/etcd.lua#L135)
- i found some issue which with upstream ipv6 and etcd ipv6 ,but close them already(https://github.com/apache/apisix/issues/7100), i do not get the right solution. in my version(2.15.3), it just NOT work.
etcd yaml:
apiVersion: v1
data:
jwt-token.pem: LS0tLS1CRUdJTiBSU0EgUFJJVkFURSBLRVktLS0tLQpNSUlKS2dJQkFBS0NBZ0VBN1d6LzVNb3AzVHF0QWNMeFVMU2p5Mm9SbmhJSXBvTndBV2pLb2Uxb08reDFlKzMzCjc1dG1BTU91SmhxS0c3TGhIdFVxWENiQXZlYWRyZ09QTnJ4MGVLS0J2Ni9sY0FBbUh2Y2FVdyt3d2lYNFFxWXgKSWE0NTVGODkyL3pBT2YwUm5UMVVjUXJQbDR3SDZWWndvY3QrNzhZL3lDaWRQcGpoWTNra0IveWszUmlxUVFMMwpqR3NHMXZLYURNUEl3bFgxc0xkRUdzTjhjN1pvd0x5SWtCc1BiSWlYSHdPeWE3Mkp2cVRZc2dydWw5Ri8yRFl1ClBWZEhaMzREdTBpd1NVNCt1N1lQS0lFYUp6RmdSd3c2czJXYlZBQXVEN1h6LzU2anMvTm5uZEFNWHlYT1RReGsKTnQwZlZVS2R5SWNPdGJGK2w0blFnNG1RVXBDS3MvdGhHYzFzY1hMWHpqWVJaclo3NjFkVFRYSm15UWRWbWhOYgpaemZ2SzJEZ3ppYkZBcStPQ29PaWE2TlByYUhTZEFsejlWcS92MFB4T0RlbXZDR2Q1clg2VHF4eUdqcGtGcWxOCmdNbVpWR2pHdVBEWXN5TVpjUDZOWEVoUFFhWmRWNzY5Q1kyY0xlSWVsWk1LNC9TUDNOL0RiU3E2cEdZaDhOV1kKV3pyV2k0UWtyUVI0UnBlOGhGL1FRbmo5RnpVTUxlR05zUG91L0JnTGJCbDIyN2E3NmdlUEsrMmp2VW5UMXlCVApLbzJmWFlnYTgxVEhpYzhOR2VCSDlrU1VQMThXVUVTcGI2cXVJTVQ5dHAzTmx6SDdYc3E4TnNqQlEwMVJqb2VCCm9Tc1hYcDZteWhJdU90QjZINkFLbTJwTitCU0lrRW9XL0RUYlJwTllkdE5FSjRpT3BWZFNYaFE1d1dzQ0F3RUEKQVFLQ0FnRUE0R3VHV2NGSjN3WkdZQUxmVExUaW1qU3hZR09WYjZlUHFKVmY2NzFYS0VVeks2aGRFa3dtRjVUZgpPTDhmU2wxRDgzdjBaVlJ3SFZBdVBUeTZFemdCTThJUHVKdlAySlViYzJ1SW8zaVNVSUhKYjE1ZDFJTTF6NHJECkV2eU95b0ZPTS8yem9vSmdoTlBrNm9jOVFZSHpnenMyTU1GTEdNOHpSRzVPUTRwb1gyQ2dGWUZ6Y25QNzdCTTIKejlnRUhNVlp6ZWI4NWZ0ZjJKUlQ0ZEtUbHZzdEVoN3VFVy9TSjRycDk1MGtBdTdZWmdLdHdEWStjTmJkOUpJbApmdEtQWjBzaGV4YjByd21SS2ZpL0U0a1l0MWZVZ1lzRi91MnpMQ3pNZWgwYUZRNkJrekpPRllBZi9jZjNoUVRKCjFiRHJrNHlwcHgycVRac3JJdy9qTzRyaVZweFk0ZlVjOW5MeVJPeG1CaC9uTHlFNFVEQWhZSGMwZjZhUWxYSVYKcGtpZ3d4ZENrditRNm9uaUZLUFU0RDF3VlhQOVVxZ2ltaFJ5V2tPcmxXUGZWa09laWIwek03Z3NvVXBtRGJ4TAoyOXAweHB6WVBPQ3REbXVXTEZxZWZmSUFFM0g5UFowWHVRUlJPQTV3cHlBdUVqK1U1cjJPMGQ5T0ZLQVA0akpXCmM3aHEwK29pZHhHNmZMK3VJZ0d1T1hPZW9uYU5jNFQ2NVZPbG9EQWkrdkZycUlXVktSbm9qMjFLdytyYzg0aHoKK2tPTktnbHVjZ2JjbUgrb1AvanU3dklTdW8raXpRUFViZnZ2alhwRThxc1l0eW1UY1J2UW5pd2I2c3h0ekNBVgpuZVE5ZElZMGYzT2JvQitucHBQMUFIZWszZlJLN1hZZ2c5eTJ4eE1IdFpyRitvUXNpaGtDZ2dFQkFPNHNzVnVLCmhLcWNJcmdkZUc2OEg0TGNtT09GWHRyZlRMNTVzTS83STI5UWhjMmczc3pYYXZoM1BndEo1ekxJeDMwdWdKdWYKN1h6Snh2Tjl2RGs1SUlVTXc3SnRJYXZKNGVQWks2L0lHZUVkOXhWTUF2NTIrVE5rOFJSbW5oeW5ua0FPNlkxcwpGM21GTURmblVxbFhzODRCVlM1N1Y3Z3RZMXV2TWpOMHRSdmdjNkxFeStpd1NQYTRnUXM2aFY4R3dMSVk3ekQvCmJHWU93bVgxNjhwbVBhbXFxOXIyUU1ubnNiWFhJb3R5aVZ1YzFwY3B5aFdqOHI1MWw4RFR6eVpEVUZWcTlXcC8KOFdkYXZ3OURwS0srckIxcEM5dFJadXF2ZjNQTkJ2UjdXdnZNS3FXTCtGZi9LdXhTMk9yenRCSXBSeVJJQTNwNgowTnBDbVVNenJObDRBRGNDZ2dFQkFQOHg5Y255dlVXbHl4b01xKzVLaTVpS3ppVzBBb1loWlZVajFLN3BRNktPCkwxeEs1UHg0WS9aTmNlNGs0c3F4azBkZmpCT3J1eFdYSk1oMGhYSHhPVzhBUDBSYU1waWc5YVh4MWkwVXp6anIKbEZLZWtoNzNrQWpFcHBYN0FGcVRHOHF2NUJ2azArRWI1cGhPNFM4ZFlSQngxdzdSKzYvVisxS3JuYm1LOTE1RQoxbVBWTWtzQ2dlNnFwUGs4aHhWeHVhcStrOWR3eVVVSXhQR2w1NW5BU0hyN1JYaldYa3pkallIZk8vZHFxSGwwCkc3MWx2QmN5dUMwOERsMW5LVE5KQlAvY0pEeGVNVlVha25NM00yN2FNeitPSmRnTUE2YVV1V3pYS3NIWHdBeHkKeWM0ZWJwZEdKcVkrRGRhNGNLckV6VTQ4T2hTM1l6cGxZcXlyemtqeXBtMENnZ0VCQU5PSFUvbEdHNjlldmlNZwo0VjRQckRqUDdPVmRCVEtFVWFkMWNvZzB0bkxkWkFpTlVITkMrdGt6SmlKbWYzU0dCMDV4WjMxUDIxOFI0YVZOClRVYlJLc1dmNFlwdERCT0RXQ0RCTnVDR2FoMmFQR0Jvd3R2M0ZEb0Y4MnEzL01MY1Ixc0RJNEFidVBtUFJaVFEKMklSWHhQUTRFTXdZZFg2NHFONjd5VzBUd09uQ1BWRVpVVFFXcEthaWxORkJKMFNQUGNLdkRIaEswWjRPaTY1OAphKzYxUVJ5WDNNdTZHeTUyblVrdmlabXBucjZEbTc1ak9meGdRRzNSQ0hjVVpveklDZWZpOTVXZmplbktHWmZYCjdyY3Zlck1nSVl5bVRpNHVzWlpLU0Z3SDFuVjlEWDg3d2ROOVowUytDN01Yd0J6RkZrN3R1bEVrRkt0VHJNU1gKZDNNbzlaOENnZ0VCQUptaFhCVHRtMkI0aFNMemlmWDB6c0ZRbnZZM3ZtVTlhYUd0NW5ZK1c4ZGF6Y3hFRWtLagp2NW9oMlhyQ25mM2tsWU9jVTlucndyOG10TEF3NWIxSXVZakhuMDdvOWhqSW5kbi9FeThrbmZQb2J1eW1KZFdhCnVBMXZSZEo0dnlmSDlDMDdZcG9nVWlYdEJBK2hQUk4rSGxjbUVaQU1mZmJIWFh2UmNTeW9LbStJQllFb2NoU1MKTitLVXZLaUc5ZFBGR1Q5ZlorV0ZNc1hxbDVYYWlPa1l4d215aDRwTms0dTJ2Vi80SWtPNjVCM3A5bUU1QTlqNwpZY0ZwckVReXZLenhRcDg5eWxyRllmR1lBNTUxRnZPZlRNYndMbnc0RDJLVG1tV1p5MVhUS3Z6VGhnWWViL21GCjgvS1UzUVB0R1hiVTc3d3ZxYythNzVQU1FXc3VLd3ZqZUtrQ2dnRUFWMHlBbG9YQ2hicnQrYml3MTVsek50dmwKUktWbFlyeXNoWlJ4VTNoRU9yTllPY042WkwxazZLWnUySVhWckJBVjlQNlVPbmdaK1VxUVpibHluSGVZcTZteApDWU1HSjVaNVZkYzJEYXVVaUNkczRybkd5U0pKWGFpUC82WVlxbEJvWkMvdnlHK3dBTy9ybThtdFlrR1hhdEc5CkRqMkJqd2l5Z0ROQUlKZ2l0OHZYTWM0RFlvK3Z5WkJUZVRNb21RNkFITDJiNUc2NE81ZWxHU08yZzd3SnJ6QysKdjhCZUIyTCs3U0x0amNVTEsydktoWFNnVE9EZTUwWWU4UVFBVTRYM1lIQStPc3pGa1pLWVNGaEtnV3hzNDdzUApIRGU1ZS92NjFNRDFnbkdaVjBrS1hIRHhwVWExNkpTKzlya3Z2UEF6aVQwNzltUnJ4YTliUnhmd2lZS3FQUT09Ci0tLS0tRU5EIFJTQSBQUklWQVRFIEtFWS0tLS0tCg==
kind: Secret
metadata:
labels:
app.kubernetes.io/instance: etcd
app.kubernetes.io/name: etcd
name: etcd-jwt-token
namespace: apisix
type: Opaque
---
apiVersion: apps/v1
kind: StatefulSet
metadata:
generation: 1
labels:
app.kubernetes.io/instance: etcd
app.kubernetes.io/name: etcd
name: etcd
namespace: apisix
spec:
podManagementPolicy: Parallel
replicas: 3
revisionHistoryLimit: 10
selector:
matchLabels:
app.kubernetes.io/instance: etcd
app.kubernetes.io/name: etcd
serviceName: etcd-headless
template:
metadata:
annotations:
checksum/token-secret: 60156acd10c1cc700c72beb227921dd1c20bf596700362f42d7ab98b32f67a44
prometheus.io/port: "2379"
prometheus.io/scrape: "true"
labels:
app.kubernetes.io/instance: etcd
app.kubernetes.io/name: etcd
spec:
affinity:
podAntiAffinity:
preferredDuringSchedulingIgnoredDuringExecution:
- podAffinityTerm:
labelSelector:
matchLabels:
app.kubernetes.io/instance: etcd
app.kubernetes.io/name: etcd
namespaces:
- apisix
topologyKey: kubernetes.io/hostname
weight: 1
containers:
- env:
- name: BITNAMI_DEBUG
value: "false"
- name: MY_POD_IP
valueFrom:
fieldRef:
apiVersion: v1
fieldPath: status.podIP
- name: MY_POD_NAME
valueFrom:
fieldRef:
apiVersion: v1
fieldPath: metadata.name
- name: MY_POD_NAMESPACE
valueFrom:
fieldRef:
apiVersion: v1
fieldPath: metadata.namespace
- name: ETCDCTL_API
value: "3"
- name: ETCD_ON_K8S
value: "yes"
- name: ETCD_START_FROM_SNAPSHOT
value: "no"
- name: ETCD_DISASTER_RECOVERY
value: "no"
- name: ETCD_NAME
value: $(MY_POD_NAME)
- name: ETCD_DATA_DIR
value: /bitnami/etcd/data
- name: ETCD_LOG_LEVEL
value: info
- name: ALLOW_NONE_AUTHENTICATION
value: "yes"
- name: ETCD_AUTH_TOKEN
value: jwt,priv-key=/opt/bitnami/etcd/certs/token/jwt-token.pem,sign-method=RS256,ttl=10m
- name: ETCD_ADVERTISE_CLIENT_URLS
value: http://$(MY_POD_NAME).etcd-headless.$(MY_POD_NAMESPACE).svc.cluster.local:2379
- name: ETCD_LISTEN_CLIENT_URLS
value: http://[::]:2379
- name: ETCD_INITIAL_ADVERTISE_PEER_URLS
value: http://$(MY_POD_NAME).etcd-headless.$(MY_POD_NAMESPACE).svc.cluster.local:2380
- name: ETCD_LISTEN_PEER_URLS
value: http://[::]:2380
- name: ETCD_INITIAL_CLUSTER_TOKEN
value: etcd-cluster-k8s
- name: ETCD_INITIAL_CLUSTER_STATE
value: new
- name: ETCD_INITIAL_CLUSTER
value: etcd-0=http://etcd-0.etcd-headless.$(MY_POD_NAMESPACE).svc.cluster.local:2380,etcd-1=http://etcd-1.etcd-headless.$(MY_POD_NAMESPACE).svc.cluster.local:2380,etcd-2=http://etcd-2.etcd-headless.$(MY_POD_NAMESPACE).svc.cluster.local:2380
- name: ETCD_CLUSTER_DOMAIN
value: etcd-headless.$(MY_POD_NAMESPACE).svc.cluster.local
- name: MY_STS_NAME
value: etcd
image: {{ .harbor_k_host }}/{{ .image_etcd }}
imagePullPolicy: IfNotPresent
lifecycle:
preStop:
exec:
command:
- /opt/bitnami/scripts/etcd/prestop.sh
livenessProbe:
failureThreshold: 5
initialDelaySeconds: 60
periodSeconds: 30
successThreshold: 1
tcpSocket:
port: 2379
timeoutSeconds: 5
name: etcd
ports:
- containerPort: 2379
name: client
protocol: TCP
- containerPort: 2380
name: peer
protocol: TCP
resources:
requests:
cpu: 1
memory: 1G
ephemeral-storage: 10Gi
limits:
cpu: 1
memory: 1G
ephemeral-storage: 10Gi
securityContext:
runAsNonRoot: true
runAsUser: 1001
terminationMessagePath: /dev/termination-log
terminationMessagePolicy: File
volumeMounts:
- mountPath: /bitnami/etcd
name: data
- mountPath: /opt/bitnami/etcd/certs/token/
name: etcd-jwt-token
readOnly: true
dnsPolicy: ClusterFirst
restartPolicy: Always
schedulerName: default-scheduler
securityContext:
fsGroup: 1001
serviceAccount: default
serviceAccountName: default
terminationGracePeriodSeconds: 30
volumes:
- name: etcd-jwt-token
secret:
defaultMode: 256
secretName: etcd-jwt-token
updateStrategy:
type: RollingUpdate
volumeClaimTemplates:
- apiVersion: v1
kind: PersistentVolumeClaim
metadata:
creationTimestamp: null
name: data
spec:
accessModes:
- ReadWriteOnce
resources:
requests:
storage: 8Gi
storageClassName: nfs-controller
volumeMode: Filesystem
status:
phase: Pending
---
apiVersion: v1
kind: Service
metadata:
annotations:
service.alpha.kubernetes.io/tolerate-unready-endpoints: "true"
labels:
app.kubernetes.io/instance: etcd
app.kubernetes.io/name: etcd
name: etcd-headless
namespace: apisix
spec:
clusterIP: None
clusterIPs:
- None
internalTrafficPolicy: Cluster
ipFamilies:
- IPv6
- IPv4
ipFamilyPolicy: RequireDualStack
ports:
- name: client
port: 2379
protocol: TCP
targetPort: client
- name: peer
port: 2380
protocol: TCP
targetPort: peer
selector:
app.kubernetes.io/instance: etcd
app.kubernetes.io/name: etcd
sessionAffinity: None
type: ClusterIP
Environment
- APISIX version (run
apisix version): - Operating system (run
uname -a): - OpenResty / Nginx version (run
openresty -Vornginx -V): - etcd version, if relevant (run
curl http://127.0.0.1:9090/v1/server_info): - APISIX Dashboard version, if relevant:
- Plugin runner version, for issues related to plugin runners:
- LuaRocks version, for installation issues (run
luarocks --version):
@stubbornTanzhe This fix/feat https://github.com/apache/apisix/issues/7100 is not available in 2.15.3. Here is a request to cherry pick this to 2.15.3 - https://github.com/apache/apisix/pull/8245#issuecomment-1504445848
@stubbornTanzhe This is available in 3.2.0
@stubbornTanzhe do you have time to backport this to 2.15.3?
@stubbornTanzhe This is available in 3.2.0
thank you so much
@stubbornTanzhe do you have time to backport this to 2.15.3?
really not, and I found that maybe there is some tricky thing with the backport, such as i just replace the patch.lua(just with the change code) and not work.
I will REALLY appreciate that if any of you backport this code(and update the rpm)
because 2.15.3 is a big version with no aggressive step and ipv6 feature is a big section in Independent and Controllable Nationalization ;)
can you do the backport thing? or can you give some advice about the patch.lua enable mechanism? @moonming @Revolyssup
This issue has been marked as stale due to 350 days of inactivity. It will be closed in 2 weeks if no further activity occurs. If this issue is still relevant, please simply write any comment. Even if closed, you can still revive the issue at any time or discuss it on the [email protected] list. Thank you for your contributions.
This issue has been closed due to lack of activity. If you think that is incorrect, or the issue requires additional review, you can revive the issue at any time.