Updating GeoIP2 data outputs many logs of "File changed detected. Reloading NGINX"

[atuk721]

What happened:

When I start ingress-nginx with the folloing settings and update GeoIP2 data, ingress-nginx outputs many logs of File changed detected. Reloading NGINX.

• Config
  • use-geoip: 'false
  • use-geoip2: 'true
• Mount emptyDir to /etc/ingress-controller/geoip
• Init container prepares GeoIP2 data and set emptyDir

What you expected to happen:

File change detection and nginx reloading should be once per a file change.

NGINX Ingress controller version (exec into the pod and run nginx-ingress-controller --version.):

-------------------------------------------------------------------------------
NGINX Ingress controller
  Release:       v1.10.0
  Build:         71f78d49f0a496c31d4c19f095469f3f23900f8a
  Repository:    https://github.com/kubernetes/ingress-nginx
  nginx version: nginx/1.25.3

-------------------------------------------------------------------------------

Kubernetes version (use kubectl version):

Client Version: v1.28.6
Kustomize Version: v5.0.4-0.20230601165947-6ce0bf390ce3
Server Version: v1.28.3

Environment:

• OS (e.g. from /etc/os-release): Ubuntu 22.04.4 LTS
• Kernel (e.g. uname -a): Linux blade 6.5.0-21-generic #21~22.04.1-Ubuntu SMP PREEMPT_DYNAMIC Fri Feb 9 13:32:52 UTC 2 x86_64 x86_64 x86_64 GNU/Linux
• Install tools: minikube and RKE

How to reproduce this issue:

You need to prepare GeoLite2-City.mmdb that need MaxMind account.

# 1. Download GeoLite2 data and extrat GeoLite2-City.mmdb to a specific directory.

# 2. Start minikube.
$ minikube start

# 3. Mount the specific directory to have GeoLite2-City.mmd to Minikube. 
$ minikube mount {specific folder}:/host

# 4. Install ingress-nginx.
$ cat << EOF > test.yaml
controller:
  config:
    use-geoip: 'false'
    use-geoip2: 'true'

  extraVolumeMounts:
    - name: geoip-data
      mountPath: /etc/ingress-controller/geoip

  extraVolumes:
    - name: geoip-data
      emptyDir: {}

    - name: host
      hostPath:
        path: /host
        type: Directory

  extraInitContainers:
    - name: init-geoip
      image: alpine
      args: ['cp', '/host/GeoLite2-City.mmdb', '/geoip/']

      volumeMounts:
        - name: geoip-data
          mountPath: /geoip

        - name: host
          mountPath: /host
EOF

$ helm install ingress-test ingress-nginx/ingress-nginx --version 4.10.0 -f test.yaml

# 5. Tail logs.
$ kubectl logs -f ingress-test-ingress-nginx-controller-{...}

# 6. Update GeoIP2 data file.
$ kubectl exec -it ingress-test-ingress-nginx-controller-{...} bash

$ cp /etc/ingress-controller/geoip/GeoLite2-City.mmdb /tmp/
$ cp /tmp/GeoLite2-City.mmdb /etc/ingress-controller/geoip/GeoLite2-City.mmdb

A part of the logs:

I0308 14:15:52.622881       7 nginx.go:207] "File changed detected. Reloading NGINX" path="/etc/ingress-controller/geoip/GeoLite2-City.mmdb"                                                                   
I0308 14:15:52.622926       7 nginx.go:207] "File changed detected. Reloading NGINX" path="/etc/ingress-controller/geoip/GeoLite2-City.mmdb"                                                                   
I0308 14:15:52.623825       7 nginx.go:207] "File changed detected. Reloading NGINX" path="/etc/ingress-controller/geoip/GeoLite2-City.mmdb"                                                                   
I0308 14:15:52.623906       7 nginx.go:207] "File changed detected. Reloading NGINX" path="/etc/ingress-controller/geoip/GeoLite2-City.mmdb"                                                                   
I0308 14:15:52.624024       7 nginx.go:207] "File changed detected. Reloading NGINX" path="/etc/ingress-controller/geoip/GeoLite2-City.mmdb"                                                                   
I0308 14:15:52.624078       7 nginx.go:207] "File changed detected. Reloading NGINX" path="/etc/ingress-controller/geoip/GeoLite2-City.mmdb"                                                                   
I0308 14:15:52.624147       7 nginx.go:207] "File changed detected. Reloading NGINX" path="/etc/ingress-controller/geoip/GeoLite2-City.mmdb"                                                                   
I0308 14:15:52.624227       7 nginx.go:207] "File changed detected. Reloading NGINX" path="/etc/ingress-controller/geoip/GeoLite2-City.mmdb"                                                                   
I0308 14:15:52.624322       7 nginx.go:207] "File changed detected. Reloading NGINX" path="/etc/ingress-controller/geoip/GeoLite2-City.mmdb"                                                                   
I0308 14:15:52.624435       7 nginx.go:207] "File changed detected. Reloading NGINX" path="/etc/ingress-controller/geoip/GeoLite2-City.mmdb"                                                                   
I0308 14:15:52.624562       7 nginx.go:207] "File changed detected. Reloading NGINX" path="/etc/ingress-controller/geoip/GeoLite2-City.mmdb"                                                                   
I0308 14:15:52.624663       7 nginx.go:207] "File changed detected. Reloading NGINX" path="/etc/ingress-controller/geoip/GeoLite2-City.mmdb"                                                                   
I0308 14:15:52.624761       7 nginx.go:207] "File changed detected. Reloading NGINX" path="/etc/ingress-controller/geoip/GeoLite2-City.mmdb"                                                                   
I0308 14:15:52.624843       7 nginx.go:207] "File changed detected. Reloading NGINX" path="/etc/ingress-controller/geoip/GeoLite2-City.mmdb"                                                                   
I0308 14:15:52.624964       7 nginx.go:207] "File changed detected. Reloading NGINX" path="/etc/ingress-controller/geoip/GeoLite2-City.mmdb"                                                                   
I0308 14:15:52.625056       7 nginx.go:207] "File changed detected. Reloading NGINX" path="/etc/ingress-controller/geoip/GeoLite2-City.mmdb"                                                                   
I0308 14:15:52.625141       7 nginx.go:207] "File changed detected. Reloading NGINX" path="/etc/ingress-controller/geoip/GeoLite2-City.mmdb"                                                                   
I0308 14:15:52.625234       7 nginx.go:207] "File changed detected. Reloading NGINX" path="/etc/ingress-controller/geoip/GeoLite2-City.mmdb"

If hostPath mounts directory to the Pod and update local mounted files using the folloing setting, there is no output in logs.

$ cat << EOF > test.yaml
controller:
  config:
    use-geoip: 'false'
    use-geoip2: 'true'

  extraVolumeMounts:
    - name: geoip-data
      mountPath: /etc/ingress-controller/geoip

  extraVolumes:
    - name: geoip-data
      hostPath:
        path: /host
        type: Directory
EOF

[k8s-ci-robot]

This issue is currently awaiting triage.

If Ingress contributors determines this is a relevant issue, they will accept it by applying the triage/accepted label and provide further guidance.

The triage/accepted label can be added by org members by writing /triage accepted in a comment.

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

[strongjz]

once the cp is finished does it still reload?

[atuk721]

No. It logs about 750 lines of "File changed detected. Reloading NGINX" in 100ms and stops to log.

Start line
I0329 09:16:03.853384       7 nginx.go:207] "File changed detected. Reloading NGINX" path="/etc/ingress-controller/geoip/GeoLite2-City.mmdb"

End line
I0329 09:16:03.946326       7 nginx.go:207] "File changed detected. Reloading NGINX" path="/etc/ingress-controller/geoip/GeoLite2-City.mmdb"

Total 757 lines.

[github-actions[bot]]

This is stale, but we won't close it automatically, just bare in mind the maintainers may be busy with other tasks and will reach your issue ASAP. If you have any question or request to prioritize this, please reach #ingress-nginx-dev on Kubernetes Slack.