ingress-nginx
ingress-nginx copied to clipboard
kubernetes
multiple ingress-nginx controllers chroot failed bind internal syslog
What happened:
I'm running two controllers on my EKS cluster running on calico CNI (this forces me to overwrite a number of ports due to how the AWS CNI works in hostNetwork: true).
When running in the chroot mode I'm unable to start all my pods, my guess is that a port or something similar gets occupied when running one public and one private controller on the same node.
When running ingress-nginx chroot 1.3.0 I get the following error:
➜ k logs ingress-nginx-private-controller-6dfbd7d4b8-55k7v
-------------------------------------------------------------------------------
NGINX Ingress controller
Release: v1.3.0
Build: 2b7b74854d90ad9b4b96a5011b9e8b67d20bfb8f
Repository: https://github.com/kubernetes/ingress-nginx
nginx version: nginx/1.19.10
-------------------------------------------------------------------------------
W0718 11:37:43.227986 7 client_config.go:617] Neither --kubeconfig nor --master was specified. Using the inClusterConfig. This might not work.
I0718 11:37:43.228153 7 main.go:230] "Creating API client" host="https://172.20.0.1:443"
I0718 11:37:43.235220 7 main.go:274] "Running in Kubernetes cluster" major="1" minor="22+" git="v1.22.9-eks-a64ea69" state="clean" commit="540410f9a2e24b7a2a870ebfacb3212744b5f878" platform="linux/amd64"
I0718 11:37:43.416411 7 main.go:104] "SSL fake certificate created" file="/etc/ingress-controller/ssl/default-fake-certificate.pem"
I0718 11:37:43.432264 7 ssl.go:531] "loading tls certificate" path="/usr/local/certificates/cert" key="/usr/local/certificates/key"
F0718 11:37:43.477058 7 logger.go:36] failed bind internal syslog: %!w(*net.OpError=&{listen udp <nil> 0xc0001e3d70 0xc0005b7720})
What you expected to happen:
The application should be able to start without any issues.
When I changed back to running without the chroot image it works perfectly.
I think some socket or similar is trying to bind the some port if you are running multiple versions of the controller on the same node when running hostNetwork: true
NGINX Ingress controller version (exec into the pod and run nginx-ingress-controller --version.):
NGINX Ingress controller
Release: v1.3.0
Build: 2b7b74854d90ad9b4b96a5011b9e8b67d20bfb8f
Repository: https://github.com/kubernetes/ingress-nginx
nginx version: nginx/1.19.10
Kubernetes version (use kubectl version):
Server Version: version.Info{Major:"1", Minor:"22+", GitVersion:"v1.22.9-eks-a64ea69", GitCommit:"540410f9a2e24b7a2a870ebfacb3212744b5f878", GitTreeState:"clean", BuildDate:"2022-05-12T19:15:31Z", GoVersion:"go1.16.15", Compiler:"gc", Platform:"linux/amd64"}
Environment:
-
Cloud provider or hardware configuration: EKS
-
OS (e.g. from /etc/os-release): Amazon Linux 2
-
Kernel (e.g.
uname -a): 5.4.190-107.353.amzn2.x86_64 -
Install tools:
Please mention how/where was the cluster created like kubeadm/kops/minikube/kind etc.: Helm chart through terraform
-
Basic cluster related info:
kubectl versionkubectl get nodes -o wide
-
How was the ingress-nginx-controller installed: helm through terraform
➜ helm get values ingress-nginx-private
USER-SUPPLIED VALUES:
controller:
addHeaders: null
admissionWebhooks:
port: 2443
affinity:
podAntiAffinity:
preferredDuringSchedulingIgnoredDuringExecution:
- podAffinityTerm:
labelSelector:
matchExpressions:
- key: app.kubernetes.io/name
operator: In
values:
- ingress-nginx
topologyKey: topology.kubernetes.io/zone
weight: 100
config:
allow-snippet-annotations: false
datadog-collector-host: $HOST_IP
enable-opentracing: "true"
server-tokens: "false"
containerPort:
http: 1080
https: 1443
dnsPolicy: ClusterFirstWithHostNet
electionID: ingress-controller-leader-nginx-private
extraArgs:
default-server-port: 8282
healthz-port: 10354
http-port: 1080
https-port: 1443
profiler-port: 10345
status-port: 10346
stream-port: 10347
extraEnvs:
- name: HOST_IP
valueFrom:
fieldRef:
apiVersion: v1
fieldPath: status.hostIP
hostNetwork: true
image:
chroot: true
ingressClass: nginx-private
ingressClassResource:
controllerValue: k8s.io/ingress-nginx-private
default: false
name: nginx-private
livenessProbe:
httpGet:
port: 10354
metrics:
enabled: true
port: 10354
service:
labels:
function: metrics
podAnnotations:
ad.datadoghq.com/controller.check_names: '["nginx", "nginx_ingress_controller"]'
ad.datadoghq.com/controller.init_configs: '[{},{}]'
ad.datadoghq.com/controller.instances: '[{"prometheus_url": "http://%%host%%:%%port_metrics%%/metrics"}]'
ad.datadoghq.com/controller.logs: '[{"service": "controller", "source": "nginx-ingress-controller"}]'
priorityClassName: platform-medium
readinessProbe:
httpGet:
port: 10354
replicaCount: 3
resources:
requests:
cpu: 100m
memory: 110Mi
service:
annotations:
service.beta.kubernetes.io/aws-load-balancer-internal: "true"
service.beta.kubernetes.io/aws-load-balancer-type: nlb
externalTrafficPolicy: Local
Public ingress-nginx controller
➜ helm get values ingress-nginx-public
USER-SUPPLIED VALUES:
controller:
addHeaders: null
affinity:
podAntiAffinity:
preferredDuringSchedulingIgnoredDuringExecution:
- podAffinityTerm:
labelSelector:
matchExpressions:
- key: app.kubernetes.io/name
operator: In
values:
- ingress-nginx
topologyKey: topology.kubernetes.io/zone
weight: 100
config:
allow-snippet-annotations: false
datadog-collector-host: $HOST_IP
enable-opentracing: "true"
server-tokens: "false"
dnsPolicy: ClusterFirstWithHostNet
electionID: ingress-controller-leader-nginx-public
extraArgs:
default-ssl-certificate: ingress-nginx/ingress-nginx
extraEnvs:
- name: HOST_IP
valueFrom:
fieldRef:
apiVersion: v1
fieldPath: status.hostIP
hostNetwork: true
image:
chroot: true
ingressClass: nginx-public
ingressClassResource:
controllerValue: k8s.io/ingress-nginx-public
default: true
name: nginx-public
metrics:
enabled: true
service:
labels:
function: metrics
podAnnotations:
ad.datadoghq.com/controller.check_names: '["nginx", "nginx_ingress_controller"]'
ad.datadoghq.com/controller.init_configs: '[{},{}]'
ad.datadoghq.com/controller.instances: '[{"prometheus_url": "http://%%host%%:%%port_metrics%%/metrics"}]'
ad.datadoghq.com/controller.logs: '[{"service": "controller", "source": "nginx-ingress-controller"}]'
priorityClassName: platform-medium
replicaCount: 3
resources:
requests:
cpu: 100m
memory: 110Mi
service:
annotations:
service.beta.kubernetes.io/aws-load-balancer-type: nlb
externalTrafficPolicy: Local
How to reproduce this issue:
My guess is that it should be enough to run two controllers, both with hostNetwork: true and
controller:
image:
chroot: true
and see that some of the pods start on the same node.
I also saved the log from when i run the controller with chroot but in v1.2.1. But it's more or less the same output but with the stack as well.
######## 1.2.1
➜ k logs ingress-nginx-public-controller-5ffd9c5cbf-bnvgc
-------------------------------------------------------------------------------
NGINX Ingress controller
Release: v1.2.1
Build: 08848d69e0c83992c89da18e70ea708752f21d7a
Repository: https://github.com/kubernetes/ingress-nginx
nginx version: nginx/1.19.10
-------------------------------------------------------------------------------
W0718 11:36:08.980738 7 client_config.go:617] Neither --kubeconfig nor --master was specified. Using the inClusterConfig. This might not work.
I0718 11:36:08.980919 7 main.go:230] "Creating API client" host="https://172.20.0.1:443"
I0718 11:36:08.994150 7 main.go:274] "Running in Kubernetes cluster" major="1" minor="22+" git="v1.22.9-eks-a64ea69" state="clean" commit="540410f9a2e24b7a2a870ebfacb3212744b5f878" platform="linux/amd64"
I0718 11:36:09.221205 7 main.go:104] "SSL fake certificate created" file="/etc/ingress-controller/ssl/default-fake-certificate.pem"
I0718 11:36:09.262684 7 ssl.go:531] "loading tls certificate" path="/usr/local/certificates/cert" key="/usr/local/certificates/key"
I0718 11:36:09.307437 7 nginx.go:256] "Starting NGINX Ingress controller"
F0718 11:36:09.307848 7 logger.go:36] failed bind internal syslog: %!w(*net.OpError=&{listen udp <nil> 0xc00012cff0 0xc0003c3580})
goroutine 75 [running]:
k8s.io/klog/v2.stacks(0x1)
k8s.io/klog/[email protected]/klog.go:860 +0x8a
k8s.io/klog/v2.(*loggingT).output(0x2859100, 0x3, 0x0, 0xc0004440e0, 0x1, {0x2013624?, 0x1?}, 0xc0005f5c00?, 0x0)
k8s.io/klog/[email protected]/klog.go:825 +0x686
k8s.io/klog/v2.(*loggingT).printfDepth(0x2859100, 0x5c65041c?, 0x0, {0x0, 0x0}, 0x405d80?, {0x19cc6fb, 0x1f}, {0xc000039fb0, 0x1, ...})
k8s.io/klog/[email protected]/klog.go:630 +0x1f2
k8s.io/klog/v2.(*loggingT).printf(...)
k8s.io/klog/[email protected]/klog.go:612
k8s.io/klog/v2.Fatalf(...)
k8s.io/klog/[email protected]/klog.go:1516
main.logger({0x19b5424, 0xf})
k8s.io/ingress-nginx/cmd/nginx/logger.go:36 +0x1d4
created by main.main
k8s.io/ingress-nginx/cmd/nginx/main.go:158 +0xf75
goroutine 1 [chan receive]:
main.handleSigterm(0x19acee2?, 0xa, 0x1a8e180)
k8s.io/ingress-nginx/cmd/nginx/main.go:175 +0x9c
main.main()
k8s.io/ingress-nginx/cmd/nginx/main.go:165 +0x1097
goroutine 35 [sleep]:
time.Sleep(0x12a05f200)
runtime/time.go:194 +0x12e
k8s.io/ingress-nginx/internal/ingress/metric.(*collector).Start.func1()
k8s.io/ingress-nginx/internal/ingress/metric/main.go:149 +0x2c
created by k8s.io/ingress-nginx/internal/ingress/metric.(*collector).Start
k8s.io/ingress-nginx/internal/ingress/metric/main.go:148 +0x31f
goroutine 32 [syscall]:
syscall.Syscall6(0xe8, 0x12, 0xc0006afc14, 0x7, 0xffffffffffffffff, 0x0, 0x0)
syscall/asm_linux_amd64.s:43 +0x5
golang.org/x/sys/unix.EpollWait(0x0?, {0xc0006afc14?, 0x0?, 0x0?}, 0x0?)
golang.org/x/[email protected]/unix/zsyscall_linux_amd64.go:56 +0x58
github.com/fsnotify/fsnotify.(*fdPoller).wait(0xc0003e2ea0)
github.com/fsnotify/[email protected]/inotify_poller.go:86 +0x7d
github.com/fsnotify/fsnotify.(*Watcher).readEvents(0xc000100140)
github.com/fsnotify/[email protected]/inotify.go:206 +0x26e
created by github.com/fsnotify/fsnotify.NewWatcher
github.com/fsnotify/[email protected]/inotify.go:60 +0x1c5
goroutine 30 [IO wait]:
internal/poll.runtime_pollWait(0x7fbea1ce7f78, 0x72)
runtime/netpoll.go:302 +0x89
internal/poll.(*pollDesc).wait(0xc0003cec80?, 0xc000500000?, 0x0)
internal/poll/fd_poll_runtime.go:83 +0x32
internal/poll.(*pollDesc).waitRead(...)
internal/poll/fd_poll_runtime.go:88
internal/poll.(*FD).Read(0xc0003cec80, {0xc000500000, 0x4116, 0x4116})
internal/poll/fd_unix.go:167 +0x25a
net.(*netFD).Read(0xc0003cec80, {0xc000500000?, 0xc0004291c0?, 0xc00050004b?})
net/fd_posix.go:55 +0x29
net.(*conn).Read(0xc000124018, {0xc000500000?, 0x0?, 0x1007fffe4?})
net/net.go:183 +0x45
crypto/tls.(*atLeastReader).Read(0xc00048f7a0, {0xc000500000?, 0x0?, 0x8?})
crypto/tls/conn.go:784 +0x3d
bytes.(*Buffer).ReadFrom(0xc0001ef3f8, {0x1c115e0, 0xc00048f7a0})
bytes/buffer.go:204 +0x98
crypto/tls.(*Conn).readFromUntil(0xc0001ef180, {0x1c13c60?, 0xc000124018}, 0x40d0?)
crypto/tls/conn.go:806 +0xe5
crypto/tls.(*Conn).readRecordOrCCS(0xc0001ef180, 0x0)
crypto/tls/conn.go:613 +0x116
crypto/tls.(*Conn).readRecord(...)
crypto/tls/conn.go:581
crypto/tls.(*Conn).Read(0xc0001ef180, {0xc000481000, 0x1000, 0x9e5aa0?})
crypto/tls/conn.go:1284 +0x16f
bufio.(*Reader).Read(0xc00044bec0, {0xc0002104a0, 0x9, 0x9f3d82?})
bufio/bufio.go:236 +0x1b4
io.ReadAtLeast({0x1c11440, 0xc00044bec0}, {0xc0002104a0, 0x9, 0x9}, 0x9)
io/io.go:331 +0x9a
io.ReadFull(...)
io/io.go:350
golang.org/x/net/http2.readFrameHeader({0xc0002104a0?, 0x9?, 0xc001326990?}, {0x1c11440?, 0xc00044bec0?})
golang.org/x/[email protected]/http2/frame.go:237 +0x6e
golang.org/x/net/http2.(*Framer).ReadFrame(0xc000210460)
golang.org/x/[email protected]/http2/frame.go:498 +0x95
golang.org/x/net/http2.(*clientConnReadLoop).run(0xc0004d5f98)
golang.org/x/[email protected]/http2/transport.go:2118 +0x130
golang.org/x/net/http2.(*ClientConn).readLoop(0xc00036e480)
golang.org/x/[email protected]/http2/transport.go:2014 +0x6f
created by golang.org/x/net/http2.(*Transport).newClientConn
golang.org/x/[email protected]/http2/transport.go:725 +0xa65
goroutine 36 [chan receive]:
k8s.io/ingress-nginx/internal/ingress/metric/collectors.namedProcess.Start(...)
k8s.io/ingress-nginx/internal/ingress/metric/collectors/process.go:185
created by k8s.io/ingress-nginx/internal/ingress/metric.(*collector).Start
k8s.io/ingress-nginx/internal/ingress/metric/main.go:152 +0x38a
goroutine 37 [IO wait]:
internal/poll.runtime_pollWait(0x7fbea1ce7e88, 0x72)
runtime/netpoll.go:302 +0x89
internal/poll.(*pollDesc).wait(0xc0001b5a80?, 0x0?, 0x0)
internal/poll/fd_poll_runtime.go:83 +0x32
internal/poll.(*pollDesc).waitRead(...)
internal/poll/fd_poll_runtime.go:88
internal/poll.(*FD).Accept(0xc0001b5a80)
internal/poll/fd_unix.go:614 +0x22c
net.(*netFD).accept(0xc0001b5a80)
net/fd_unix.go:172 +0x35
net.(*UnixListener).accept(0x0?)
net/unixsock_posix.go:166 +0x1c
net.(*UnixListener).Accept(0xc0001091a0)
net/unixsock.go:260 +0x3d
k8s.io/ingress-nginx/internal/ingress/metric/collectors.(*SocketCollector).Start(0xc00030ba40)
k8s.io/ingress-nginx/internal/ingress/metric/collectors/socket.go:354 +0x35
created by k8s.io/ingress-nginx/internal/ingress/metric.(*collector).Start
k8s.io/ingress-nginx/internal/ingress/metric/main.go:153 +0x3d8
goroutine 38 [IO wait]:
internal/poll.runtime_pollWait(0x7fbea1ce7d98, 0x72)
runtime/netpoll.go:302 +0x89
internal/poll.(*pollDesc).wait(0xc0003ce000?, 0x0?, 0x0)
internal/poll/fd_poll_runtime.go:83 +0x32
internal/poll.(*pollDesc).waitRead(...)
internal/poll/fd_poll_runtime.go:88
internal/poll.(*FD).Accept(0xc0003ce000)
internal/poll/fd_unix.go:614 +0x22c
net.(*netFD).accept(0xc0003ce000)
net/fd_unix.go:172 +0x35
net.(*TCPListener).accept(0xc0004f2000)
net/tcpsock_posix.go:139 +0x28
net.(*TCPListener).Accept(0xc0004f2000)
net/tcpsock.go:288 +0x3d
net/http.(*Server).Serve(0xc0002100e0, {0x1c29f10, 0xc0004f2000})
net/http/server.go:3039 +0x385
net/http.(*Server).ListenAndServe(0xc0002100e0)
net/http/server.go:2968 +0x7d
main.registerProfiler()
k8s.io/ingress-nginx/cmd/nginx/main.go:333 +0x24b
created by main.main
k8s.io/ingress-nginx/cmd/nginx/main.go:146 +0xdd3
goroutine 39 [chan receive]:
k8s.io/apimachinery/pkg/watch.(*Broadcaster).loop(0xc000128300)
k8s.io/[email protected]/pkg/watch/mux.go:247 +0x57
created by k8s.io/apimachinery/pkg/watch.NewLongQueueBroadcaster
k8s.io/[email protected]/pkg/watch/mux.go:89 +0x116
goroutine 40 [chan receive]:
k8s.io/client-go/tools/record.(*eventBroadcasterImpl).StartEventWatcher.func1()
k8s.io/[email protected]/tools/record/event.go:304 +0x73
created by k8s.io/client-go/tools/record.(*eventBroadcasterImpl).StartEventWatcher
k8s.io/[email protected]/tools/record/event.go:302 +0x8c
goroutine 41 [chan receive]:
k8s.io/client-go/tools/record.(*eventBroadcasterImpl).StartEventWatcher.func1()
k8s.io/[email protected]/tools/record/event.go:304 +0x73
created by k8s.io/client-go/tools/record.(*eventBroadcasterImpl).StartEventWatcher
k8s.io/[email protected]/tools/record/event.go:302 +0x8c
goroutine 42 [select]:
github.com/eapache/channels.(*RingChannel).ringBuffer(0xc0001812f0)
github.com/eapache/[email protected]/ring_channel.go:87 +0x16a
created by github.com/eapache/channels.NewRingChannel
github.com/eapache/[email protected]/ring_channel.go:32 +0x1d6
goroutine 43 [syscall]:
syscall.Syscall6(0xe8, 0x9, 0xc000569c14, 0x7, 0xffffffffffffffff, 0x0, 0x0)
syscall/asm_linux_amd64.s:43 +0x5
golang.org/x/sys/unix.EpollWait(0x0?, {0xc000569c14?, 0x0?, 0x0?}, 0x0?)
golang.org/x/[email protected]/unix/zsyscall_linux_amd64.go:56 +0x58
github.com/fsnotify/fsnotify.(*fdPoller).wait(0xc000139d40)
github.com/fsnotify/[email protected]/inotify_poller.go:86 +0x7d
github.com/fsnotify/fsnotify.(*Watcher).readEvents(0xc000100f50)
github.com/fsnotify/[email protected]/inotify.go:206 +0x26e
created by github.com/fsnotify/fsnotify.NewWatcher
github.com/fsnotify/[email protected]/inotify.go:60 +0x1c5
goroutine 44 [select]:
k8s.io/ingress-nginx/internal/watch.(*OSFileWatcher).watch.func1({0x7ffc14db9d63, 0x4})
k8s.io/ingress-nginx/internal/watch/file_watcher.go:73 +0xdf
created by k8s.io/ingress-nginx/internal/watch.(*OSFileWatcher).watch
k8s.io/ingress-nginx/internal/watch/file_watcher.go:71 +0x215
goroutine 45 [syscall]:
syscall.Syscall6(0xe8, 0xe, 0xc000589c14, 0x7, 0xffffffffffffffff, 0x0, 0x0)
syscall/asm_linux_amd64.s:43 +0x5
golang.org/x/sys/unix.EpollWait(0x0?, {0xc000589c14?, 0x0?, 0x0?}, 0x0?)
golang.org/x/[email protected]/unix/zsyscall_linux_amd64.go:56 +0x58
github.com/fsnotify/fsnotify.(*fdPoller).wait(0xc000139f60)
github.com/fsnotify/[email protected]/inotify_poller.go:86 +0x7d
github.com/fsnotify/fsnotify.(*Watcher).readEvents(0xc000100fa0)
github.com/fsnotify/[email protected]/inotify.go:206 +0x26e
created by github.com/fsnotify/fsnotify.NewWatcher
github.com/fsnotify/[email protected]/inotify.go:60 +0x1c5
goroutine 46 [select]:
k8s.io/ingress-nginx/internal/watch.(*OSFileWatcher).watch.func1({0x7ffc14db9d99, 0x3})
k8s.io/ingress-nginx/internal/watch/file_watcher.go:73 +0xdf
created by k8s.io/ingress-nginx/internal/watch.(*OSFileWatcher).watch
k8s.io/ingress-nginx/internal/watch/file_watcher.go:71 +0x215
goroutine 47 [chan receive]:
k8s.io/apimachinery/pkg/watch.(*Broadcaster).loop(0xc00031ac00)
k8s.io/[email protected]/pkg/watch/mux.go:247 +0x57
created by k8s.io/apimachinery/pkg/watch.NewLongQueueBroadcaster
k8s.io/[email protected]/pkg/watch/mux.go:89 +0x116
goroutine 48 [chan receive]:
k8s.io/client-go/tools/record.(*eventBroadcasterImpl).StartEventWatcher.func1()
k8s.io/[email protected]/tools/record/event.go:304 +0x73
created by k8s.io/client-go/tools/record.(*eventBroadcasterImpl).StartEventWatcher
k8s.io/[email protected]/tools/record/event.go:302 +0x8c
goroutine 49 [chan receive]:
k8s.io/client-go/tools/record.(*eventBroadcasterImpl).StartEventWatcher.func1()
k8s.io/[email protected]/tools/record/event.go:304 +0x73
created by k8s.io/client-go/tools/record.(*eventBroadcasterImpl).StartEventWatcher
k8s.io/[email protected]/tools/record/event.go:302 +0x8c
goroutine 51 [select]:
k8s.io/client-go/util/workqueue.(*delayingType).waitingLoop(0xc000082f60)
k8s.io/[email protected]/util/workqueue/delaying_queue.go:231 +0x305
created by k8s.io/client-go/util/workqueue.newDelayingQueue
k8s.io/[email protected]/util/workqueue/delaying_queue.go:68 +0x24f
goroutine 52 [select]:
k8s.io/client-go/util/workqueue.(*delayingType).waitingLoop(0xc000083140)
k8s.io/[email protected]/util/workqueue/delaying_queue.go:231 +0x305
created by k8s.io/client-go/util/workqueue.newDelayingQueue
k8s.io/[email protected]/util/workqueue/delaying_queue.go:68 +0x24f
goroutine 65 [select]:
k8s.io/ingress-nginx/internal/watch.(*OSFileWatcher).watch.func1({0x19ca7a3, 0xa})
k8s.io/ingress-nginx/internal/watch/file_watcher.go:73 +0xdf
created by k8s.io/ingress-nginx/internal/watch.(*OSFileWatcher).watch
k8s.io/ingress-nginx/internal/watch/file_watcher.go:71 +0x215
goroutine 66 [syscall]:
syscall.Syscall6(0xe8, 0x16, 0xc0006cfc14, 0x7, 0xffffffffffffffff, 0x0, 0x0)
syscall/asm_linux_amd64.s:43 +0x5
golang.org/x/sys/unix.EpollWait(0x0?, {0xc0006cfc14?, 0x0?, 0x0?}, 0x0?)
golang.org/x/[email protected]/unix/zsyscall_linux_amd64.go:56 +0x58
github.com/fsnotify/fsnotify.(*fdPoller).wait(0xc0003e3040)
github.com/fsnotify/[email protected]/inotify_poller.go:86 +0x7d
github.com/fsnotify/fsnotify.(*Watcher).readEvents(0xc000100230)
github.com/fsnotify/[email protected]/inotify.go:206 +0x26e
created by github.com/fsnotify/fsnotify.NewWatcher
github.com/fsnotify/[email protected]/inotify.go:60 +0x1c5
goroutine 67 [select]:
k8s.io/ingress-nginx/internal/watch.(*OSFileWatcher).watch.func1({0xc0003e2f71, 0x9})
k8s.io/ingress-nginx/internal/watch/file_watcher.go:73 +0xdf
created by k8s.io/ingress-nginx/internal/watch.(*OSFileWatcher).watch
k8s.io/ingress-nginx/internal/watch/file_watcher.go:71 +0x215
goroutine 68 [syscall]:
syscall.Syscall6(0xe8, 0x1a, 0xc00068fc14, 0x7, 0xffffffffffffffff, 0x0, 0x0)
syscall/asm_linux_amd64.s:43 +0x5
golang.org/x/sys/unix.EpollWait(0x0?, {0xc00068fc14?, 0x0?, 0x0?}, 0x0?)
golang.org/x/[email protected]/unix/zsyscall_linux_amd64.go:56 +0x58
github.com/fsnotify/fsnotify.(*fdPoller).wait(0xc0003e3060)
github.com/fsnotify/[email protected]/inotify_poller.go:86 +0x7d
github.com/fsnotify/fsnotify.(*Watcher).readEvents(0xc0001003c0)
github.com/fsnotify/[email protected]/inotify.go:206 +0x26e
created by github.com/fsnotify/fsnotify.NewWatcher
github.com/fsnotify/[email protected]/inotify.go:60 +0x1c5
goroutine 69 [runnable]:
k8s.io/ingress-nginx/internal/watch.(*OSFileWatcher).watch.func2()
k8s.io/ingress-nginx/internal/watch/file_watcher.go:71
runtime.goexit()
runtime/asm_amd64.s:1571 +0x1
created by k8s.io/ingress-nginx/internal/watch.(*OSFileWatcher).watch
k8s.io/ingress-nginx/internal/watch/file_watcher.go:71 +0x215
goroutine 70 [runnable]:
github.com/fsnotify/fsnotify.NewWatcher.func1()
github.com/fsnotify/[email protected]/inotify.go:60
runtime.goexit()
runtime/asm_amd64.s:1571 +0x1
created by github.com/fsnotify/fsnotify.NewWatcher
github.com/fsnotify/[email protected]/inotify.go:60 +0x1c5
goroutine 71 [runnable]:
k8s.io/ingress-nginx/internal/watch.(*OSFileWatcher).watch.func2()
k8s.io/ingress-nginx/internal/watch/file_watcher.go:71
runtime.goexit()
runtime/asm_amd64.s:1571 +0x1
created by k8s.io/ingress-nginx/internal/watch.(*OSFileWatcher).watch
k8s.io/ingress-nginx/internal/watch/file_watcher.go:71 +0x215
goroutine 76 [IO wait]:
internal/poll.runtime_pollWait(0x7fbea1ce7ca8, 0x72)
runtime/netpoll.go:302 +0x89
internal/poll.(*pollDesc).wait(0xc00061b100?, 0x0?, 0x0)
internal/poll/fd_poll_runtime.go:83 +0x32
internal/poll.(*pollDesc).waitRead(...)
internal/poll/fd_poll_runtime.go:88
internal/poll.(*FD).Accept(0xc00061b100)
internal/poll/fd_unix.go:614 +0x22c
net.(*netFD).accept(0xc00061b100)
net/fd_unix.go:172 +0x35
net.(*TCPListener).accept(0xc00048efd8)
net/tcpsock_posix.go:139 +0x28
net.(*TCPListener).Accept(0xc00048efd8)
net/tcpsock.go:288 +0x3d
net/http.(*Server).Serve(0xc0002101c0, {0x1c29f10, 0xc00048efd8})
net/http/server.go:3039 +0x385
net/http.(*Server).ListenAndServe(0xc0002101c0)
net/http/server.go:2968 +0x7d
main.startHTTPServer({0x0?, 0xbf08c41dfe570ed1?}, 0x43c7f65dc39ed705?, 0xc00064ca00)
k8s.io/ingress-nginx/cmd/nginx/main.go:345 +0x13b
created by main.main
k8s.io/ingress-nginx/cmd/nginx/main.go:162 +0x1031
goroutine 77 [select]:
k8s.io/apimachinery/pkg/util/wait.WaitForWithContext({0x1c2ac78, 0xc00064cb00}, 0xc000047410, 0xc72a0a?)
k8s.io/[email protected]/pkg/util/wait/wait.go:658 +0xe5
k8s.io/apimachinery/pkg/util/wait.poll({0x1c2ac78, 0xc00064cb00}, 0x98?, 0xc71885?, 0xc00063dd30?)
k8s.io/[email protected]/pkg/util/wait/wait.go:594 +0x9a
k8s.io/apimachinery/pkg/util/wait.PollImmediateUntilWithContext({0x1c2ac78, 0xc00064cb00}, 0x20?, 0xc00062b400?)
k8s.io/[email protected]/pkg/util/wait/wait.go:545 +0x49
k8s.io/apimachinery/pkg/util/wait.PollImmediateUntil(0x18?, 0x7fbec8b105b8?, 0x18?)
k8s.io/[email protected]/pkg/util/wait/wait.go:536 +0x7c
k8s.io/client-go/tools/cache.WaitForCacheSync(0x31?, {0xc000649560, 0x4, 0x4})
k8s.io/[email protected]/tools/cache/shared_informer.go:255 +0x97
k8s.io/ingress-nginx/internal/ingress/controller/store.(*Informer).Run(0xc000445c70, 0xc0001025a0)
k8s.io/ingress-nginx/internal/ingress/controller/store/store.go:170 +0x525
k8s.io/ingress-nginx/internal/ingress/controller/store.(*k8sStore).Run(0x0?, 0x0?)
k8s.io/ingress-nginx/internal/ingress/controller/store/store.go:1133 +0x25
k8s.io/ingress-nginx/internal/ingress/controller.(*NGINXController).Start(0xc0001b83c0)
k8s.io/ingress-nginx/internal/ingress/controller/nginx.go:258 +0x90
created by main.main
k8s.io/ingress-nginx/cmd/nginx/main.go:163 +0x1074
goroutine 79 [syscall]:
os/signal.signal_recv()
runtime/sigqueue.go:151 +0x2f
os/signal.loop()
os/signal/signal_unix.go:23 +0x19
created by os/signal.Notify.func1.1
os/signal/signal.go:151 +0x2a
goroutine 80 [sync.Cond.Wait]:
sync.runtime_notifyListWait(0xc0003920c8, 0x0)
runtime/sema.go:513 +0x13d
sync.(*Cond).Wait(0x16b0600?)
sync/cond.go:56 +0x8c
k8s.io/client-go/tools/cache.(*DeltaFIFO).Pop(0xc0003920a0, 0xc00063dd70)
k8s.io/[email protected]/tools/cache/delta_fifo.go:527 +0x22e
k8s.io/client-go/tools/cache.(*controller).processLoop(0xc000022120)
k8s.io/[email protected]/tools/cache/controller.go:183 +0x36
k8s.io/apimachinery/pkg/util/wait.BackoffUntil.func1(0x40d625?)
k8s.io/[email protected]/pkg/util/wait/wait.go:155 +0x3e
k8s.io/apimachinery/pkg/util/wait.BackoffUntil(0xc71385?, {0x1c13440, 0xc00064fc80}, 0x1, 0xc0001025a0)
k8s.io/[email protected]/pkg/util/wait/wait.go:156 +0xb6
k8s.io/apimachinery/pkg/util/wait.JitterUntil(0xc000022188?, 0x3b9aca00, 0x0, 0x0?, 0x7fbea1c2c380?)
k8s.io/[email protected]/pkg/util/wait/wait.go:133 +0x89
k8s.io/apimachinery/pkg/util/wait.Until(...)
k8s.io/[email protected]/pkg/util/wait/wait.go:90
k8s.io/client-go/tools/cache.(*controller).Run(0xc000022120, 0xc0001025a0)
k8s.io/[email protected]/tools/cache/controller.go:154 +0x2c5
k8s.io/client-go/tools/cache.(*sharedIndexInformer).Run(0xc0003937c0, 0xc0001b83c0?)
k8s.io/[email protected]/tools/cache/shared_informer.go:414 +0x47c
created by k8s.io/ingress-nginx/internal/ingress/controller/store.(*Informer).Run
k8s.io/ingress-nginx/internal/ingress/controller/store/store.go:160 +0xbb
goroutine 81 [sync.Cond.Wait]:
sync.runtime_notifyListWait(0xc000392168, 0x0)
runtime/sema.go:513 +0x13d
sync.(*Cond).Wait(0x16b0600?)
sync/cond.go:56 +0x8c
k8s.io/client-go/tools/cache.(*DeltaFIFO).Pop(0xc000392140, 0xc0001840e0)
k8s.io/[email protected]/tools/cache/delta_fifo.go:527 +0x22e
k8s.io/client-go/tools/cache.(*controller).processLoop(0xc0000226c0)
k8s.io/[email protected]/tools/cache/controller.go:183 +0x36
k8s.io/apimachinery/pkg/util/wait.BackoffUntil.func1(0x40d625?)
k8s.io/[email protected]/pkg/util/wait/wait.go:155 +0x3e
k8s.io/apimachinery/pkg/util/wait.BackoffUntil(0xc71385?, {0x1c13440, 0xc0001f6030}, 0x1, 0xc0001025a0)
k8s.io/[email protected]/pkg/util/wait/wait.go:156 +0xb6
k8s.io/apimachinery/pkg/util/wait.JitterUntil(0xc000022728?, 0x3b9aca00, 0x0, 0x0?, 0x7fbea1c2c380?)
k8s.io/[email protected]/pkg/util/wait/wait.go:133 +0x89
k8s.io/apimachinery/pkg/util/wait.Until(...)
k8s.io/[email protected]/pkg/util/wait/wait.go:90
k8s.io/client-go/tools/cache.(*controller).Run(0xc0000226c0, 0xc0001025a0)
k8s.io/[email protected]/tools/cache/controller.go:154 +0x2c5
k8s.io/client-go/tools/cache.(*sharedIndexInformer).Run(0xc000393680, 0xc00064ca00?)
k8s.io/[email protected]/tools/cache/shared_informer.go:414 +0x47c
created by k8s.io/ingress-nginx/internal/ingress/controller/store.(*Informer).Run
k8s.io/ingress-nginx/internal/ingress/controller/store/store.go:161 +0x145
goroutine 82 [runnable]:
k8s.io/ingress-nginx/internal/ingress/controller/store.(*Informer).Run.func3()
k8s.io/ingress-nginx/internal/ingress/controller/store/store.go:163
runtime.goexit()
runtime/asm_amd64.s:1571 +0x1
created by k8s.io/ingress-nginx/internal/ingress/controller/store.(*Informer).Run
k8s.io/ingress-nginx/internal/ingress/controller/store/store.go:163 +0x1db
goroutine 83 [runnable]:
k8s.io/ingress-nginx/internal/ingress/controller/store.(*Informer).Run.func4()
k8s.io/ingress-nginx/internal/ingress/controller/store/store.go:165
runtime.goexit()
runtime/asm_amd64.s:1571 +0x1
created by k8s.io/ingress-nginx/internal/ingress/controller/store.(*Informer).Run
k8s.io/ingress-nginx/internal/ingress/controller/store/store.go:165 +0x265
goroutine 84 [runnable]:
k8s.io/ingress-nginx/internal/ingress/controller/store.(*Informer).Run.func5()
k8s.io/ingress-nginx/internal/ingress/controller/store/store.go:166
runtime.goexit()
runtime/asm_amd64.s:1571 +0x1
created by k8s.io/ingress-nginx/internal/ingress/controller/store.(*Informer).Run
k8s.io/ingress-nginx/internal/ingress/controller/store/store.go:166 +0x2ed
goroutine 85 [runnable]:
k8s.io/apimachinery/pkg/util/wait.contextForChannel.func1()
k8s.io/[email protected]/pkg/util/wait/wait.go:300
created by k8s.io/apimachinery/pkg/util/wait.contextForChannel
k8s.io/[email protected]/pkg/util/wait/wait.go:300 +0xc5
goroutine 86 [select]:
k8s.io/apimachinery/pkg/util/wait.poller.func1.1()
k8s.io/[email protected]/pkg/util/wait/wait.go:708 +0x1c5
created by k8s.io/apimachinery/pkg/util/wait.poller.func1
k8s.io/[email protected]/pkg/util/wait/wait.go:691 +0xca
goroutine 87 [runnable]:
k8s.io/apimachinery/pkg/util/wait.(*Group).Start.func1()
k8s.io/[email protected]/pkg/util/wait/wait.go:71
created by k8s.io/apimachinery/pkg/util/wait.(*Group).Start
k8s.io/[email protected]/pkg/util/wait/wait.go:71 +0x85
goroutine 88 [runnable]:
k8s.io/apimachinery/pkg/util/wait.(*Group).Start.func1()
k8s.io/[email protected]/pkg/util/wait/wait.go:71
created by k8s.io/apimachinery/pkg/util/wait.(*Group).Start
k8s.io/[email protected]/pkg/util/wait/wait.go:71 +0x85
goroutine 89 [runnable]:
k8s.io/client-go/tools/cache.(*controller).Run.func1()
k8s.io/[email protected]/tools/cache/controller.go:129
created by k8s.io/client-go/tools/cache.(*controller).Run
k8s.io/[email protected]/tools/cache/controller.go:129 +0xbe
goroutine 90 [select]:
k8s.io/client-go/tools/cache.(*Reflector).ListAndWatch.func1(0xc0002102a0, 0xc0000221b0, 0xc0001025a0, 0xc0006e3d40)
k8s.io/[email protected]/tools/cache/reflector.go:315 +0x385
k8s.io/client-go/tools/cache.(*Reflector).ListAndWatch(0xc0002102a0, 0xc0001025a0)
k8s.io/[email protected]/tools/cache/reflector.go:361 +0x245
k8s.io/client-go/tools/cache.(*Reflector).Run.func1()
k8s.io/[email protected]/tools/cache/reflector.go:221 +0x26
k8s.io/apimachinery/pkg/util/wait.BackoffUntil.func1(0xc000657fb0?)
k8s.io/[email protected]/pkg/util/wait/wait.go:155 +0x3e
k8s.io/apimachinery/pkg/util/wait.BackoffUntil(0xc00042dd80?, {0x1c13420, 0xc000100730}, 0x1, 0xc0001025a0)
k8s.io/[email protected]/pkg/util/wait/wait.go:156 +0xb6
k8s.io/client-go/tools/cache.(*Reflector).Run(0xc0002102a0, 0xc0001025a0)
k8s.io/[email protected]/tools/cache/reflector.go:220 +0x1c6
k8s.io/apimachinery/pkg/util/wait.(*Group).StartWithChannel.func1()
k8s.io/[email protected]/pkg/util/wait/wait.go:56 +0x22
k8s.io/apimachinery/pkg/util/wait.(*Group).Start.func1()
k8s.io/[email protected]/pkg/util/wait/wait.go:73 +0x5a
created by k8s.io/apimachinery/pkg/util/wait.(*Group).Start
k8s.io/[email protected]/pkg/util/wait/wait.go:71 +0x85
goroutine 91 [select]:
golang.org/x/net/http2.(*ClientConn).RoundTrip(0xc00036e480, 0xc00031d600)
golang.org/x/[email protected]/http2/transport.go:1172 +0x451
golang.org/x/net/http2.(*Transport).RoundTripOpt(0xc0003ceb00, 0xc00031d600, {0xc0?})
golang.org/x/[email protected]/http2/transport.go:499 +0x1b7
golang.org/x/net/http2.(*Transport).RoundTrip(...)
golang.org/x/[email protected]/http2/transport.go:460
golang.org/x/net/http2.noDialH2RoundTripper.RoundTrip({0xc00042e140?}, 0xc00031d600?)
golang.org/x/[email protected]/http2/transport.go:2978 +0x1b
net/http.(*Transport).roundTrip(0xc00042e140, 0xc00031d600)
net/http/transport.go:539 +0x390
net/http.(*Transport).RoundTrip(0x1877d20?, 0xc00064ffb0?)
net/http/roundtrip.go:17 +0x19
k8s.io/client-go/transport.(*bearerAuthRoundTripper).RoundTrip(0xc0004488a0, 0xc00031d500)
k8s.io/[email protected]/transport/round_trippers.go:317 +0x3c5
k8s.io/client-go/transport.(*userAgentRoundTripper).RoundTrip(0xc000428500, 0xc00031d400)
k8s.io/[email protected]/transport/round_trippers.go:168 +0x322
net/http.send(0xc00031d400, {0x1c13a80, 0xc000428500}, {0x194daa0?, 0x9d9379fc00000001?, 0x0?})
net/http/client.go:252 +0x5d8
net/http.(*Client).send(0xc0004488d0, 0xc00031d400, {0x7fbea1d5cb30?, 0xf8?, 0x0?})
net/http/client.go:176 +0x9b
net/http.(*Client).do(0xc0004488d0, 0xc00031d400)
net/http/client.go:725 +0x8f5
net/http.(*Client).Do(...)
net/http/client.go:593
k8s.io/client-go/rest.(*Request).request(0xc00031d200, {0x1c2acb0, 0xc0001363f0}, 0x1?)
k8s.io/[email protected]/rest/request.go:980 +0x419
k8s.io/client-go/rest.(*Request).Do(0x7fbea1ce02a0?, {0x1c2acb0?, 0xc0001363f0?})
k8s.io/[email protected]/rest/request.go:1038 +0xc7
k8s.io/client-go/kubernetes/typed/core/v1.(*secrets).List(0xc000649660, {0x1c2acb0, 0xc0001363f0}, {{{0x0, 0x0}, {0x0, 0x0}}, {0x0, 0x0}, {0xc0003e30c0, ...}, ...})
k8s.io/[email protected]/kubernetes/typed/core/v1/secret.go:95 +0x185
k8s.io/client-go/informers/core/v1.NewFilteredSecretInformer.func1({{{0x0, 0x0}, {0x0, 0x0}}, {0x0, 0x0}, {0x0, 0x0}, 0x0, 0x0, ...})
k8s.io/[email protected]/informers/core/v1/secret.go:65 +0x182
k8s.io/client-go/tools/cache.(*ListWatch).List(0xc000657d50?, {{{0x0, 0x0}, {0x0, 0x0}}, {0x0, 0x0}, {0x0, 0x0}, 0x0, ...})
k8s.io/[email protected]/tools/cache/listwatch.go:106 +0x56
k8s.io/client-go/tools/cache.(*Reflector).ListAndWatch.func1.1.2({{{0x0, 0x0}, {0x0, 0x0}}, {0x0, 0x0}, {0x0, 0x0}, 0x0, 0x0, ...})
k8s.io/[email protected]/tools/cache/reflector.go:277 +0x62
k8s.io/client-go/tools/pager.SimplePageFunc.func1({0xc0003920a0?, 0x7fbea1ce05d0?}, {{{0x0, 0x0}, {0x0, 0x0}}, {0x0, 0x0}, {0x0, 0x0}, ...})
k8s.io/[email protected]/tools/pager/pager.go:40 +0x57
k8s.io/client-go/tools/pager.(*ListPager).List(0xc0004d7fb0, {0x1c2acb0, 0xc0001363e8}, {{{0x0, 0x0}, {0x0, 0x0}}, {0x0, 0x0}, {0x0, ...}, ...})
k8s.io/[email protected]/tools/pager/pager.go:92 +0x16b
k8s.io/client-go/tools/cache.(*Reflector).ListAndWatch.func1.1()
k8s.io/[email protected]/tools/cache/reflector.go:302 +0x207
created by k8s.io/client-go/tools/cache.(*Reflector).ListAndWatch.func1
k8s.io/[email protected]/tools/cache/reflector.go:268 +0x2fb
goroutine 92 [select]:
golang.org/x/net/http2.(*clientStream).writeRequest(0xc00036e780, 0xc00031d600)
golang.org/x/[email protected]/http2/transport.go:1340 +0x9c9
golang.org/x/net/http2.(*clientStream).doRequest(0xa00619a01?, 0xc000657fa0?)
golang.org/x/[email protected]/http2/transport.go:1202 +0x1e
created by golang.org/x/net/http2.(*ClientConn).RoundTrip
golang.org/x/[email protected]/http2/transport.go:1131 +0x30a
goroutine 93 [runnable]:
k8s.io/apimachinery/pkg/util/wait.(*Group).Start.func1()
k8s.io/[email protected]/pkg/util/wait/wait.go:71
created by k8s.io/apimachinery/pkg/util/wait.(*Group).Start
k8s.io/[email protected]/pkg/util/wait/wait.go:71 +0x85
goroutine 94 [runnable]:
k8s.io/apimachinery/pkg/util/wait.(*Group).Start.func1()
k8s.io/[email protected]/pkg/util/wait/wait.go:71
created by k8s.io/apimachinery/pkg/util/wait.(*Group).Start
k8s.io/[email protected]/pkg/util/wait/wait.go:71 +0x85
goroutine 95 [runnable]:
k8s.io/client-go/tools/cache.(*controller).Run.func1()
k8s.io/[email protected]/tools/cache/controller.go:129
created by k8s.io/client-go/tools/cache.(*controller).Run
k8s.io/[email protected]/tools/cache/controller.go:129 +0xbe
goroutine 96 [select]:
k8s.io/client-go/tools/cache.(*Reflector).ListAndWatch.func1(0xc000210540, 0xc000022750, 0xc0001025a0, 0xc0006f6d40)
k8s.io/[email protected]/tools/cache/reflector.go:315 +0x385
k8s.io/client-go/tools/cache.(*Reflector).ListAndWatch(0xc000210540, 0xc0001025a0)
k8s.io/[email protected]/tools/cache/reflector.go:361 +0x245
k8s.io/client-go/tools/cache.(*Reflector).Run.func1()
k8s.io/[email protected]/tools/cache/reflector.go:221 +0x26
k8s.io/apimachinery/pkg/util/wait.BackoffUntil.func1(0xc00032d7b0?)
k8s.io/[email protected]/pkg/util/wait/wait.go:155 +0x3e
k8s.io/apimachinery/pkg/util/wait.BackoffUntil(0xc00042de00?, {0x1c13420, 0xc000100870}, 0x1, 0xc0001025a0)
k8s.io/[email protected]/pkg/util/wait/wait.go:156 +0xb6
k8s.io/client-go/tools/cache.(*Reflector).Run(0xc000210540, 0xc0001025a0)
k8s.io/[email protected]/tools/cache/reflector.go:220 +0x1c6
k8s.io/apimachinery/pkg/util/wait.(*Group).StartWithChannel.func1()
k8s.io/[email protected]/pkg/util/wait/wait.go:56 +0x22
k8s.io/apimachinery/pkg/util/wait.(*Group).Start.func1()
k8s.io/[email protected]/pkg/util/wait/wait.go:73 +0x5a
created by k8s.io/apimachinery/pkg/util/wait.(*Group).Start
k8s.io/[email protected]/pkg/util/wait/wait.go:71 +0x85
goroutine 97 [runnable]:
k8s.io/client-go/tools/cache.(*Reflector).ListAndWatch.func1.1()
k8s.io/[email protected]/tools/cache/reflector.go:268
created by k8s.io/client-go/tools/cache.(*Reflector).ListAndWatch.func1
k8s.io/[email protected]/tools/cache/reflector.go:268 +0x2fb
Hey, just figured out. Can you try using the flag to configure right addressing?
https://github.com/kubernetes/ingress-nginx/blob/main/cmd/nginx/flags.go#L198
Because you are binding to hostNetwork, it will try to bind this port also on hostNetwork.
Also be aware that as soon as you are binding to hostNetwork this port will be exposed in your host and someone can push rogue messages to the logger :)
Ahh just as I thought there was some flag to define some address. Thanks allot for the quick response @rikatz!
I guess this possibility already exist when I don't run in chroot mode or isn't it public then? I think I will have time to test this out tomorrow, if It works I will create a PR to make this available to the helm chart.
Trust me I don't want to run hostNetwork but the AWS CNI don't give me much of an option....
Sorry for the delay @rikatz, I have verified and setting - --internal-logger-address=127.0.0.1:11515 solved the issue.
I think I should have time some time later next week to create a PR in the helm chart unless someone else is quicker then me.
FYI to other people that might be following the issue the flags file have moved: https://github.com/kubernetes/ingress-nginx/blob/0cc43d5e5295a769fa6e9428004a70378b632e9b/pkg/flags/flags.go#L202
@rikatz you wrote earlier "Also be aware that as soon as you are binding to hostNetwork this port will be exposed in your host and someone can push rogue messages to the logger ".
Would you say it's better to not run the chrooted image to not expose the syslog address? Sadly I can't remove the hostNetwork config.
Was planning to start to do a contribution in the helm chart but remembered that the extraArgs is already in place (I'm already using it).
I will create a documentation PR to make this value a bit easier to find without having to read the code.