context deadline exceeded (Client.Timeout exceeded while awaiting headers)

[igoooor]

Hello,

I am using owasp/modsecurity-crs:apache-alpine in my kubernetes cluster. The image is configured in a Deployment and I have the following liveness & readiness defined:

          readinessProbe:
            httpGet:
              path: /healthz
              port: http
              httpHeaders:
                - name: Host
                  value: healthz
            initialDelaySeconds: 5
            periodSeconds: 5
            failureThreshold: 3
            timeoutSeconds: 1
            successThreshold: 1
          livenessProbe:
            httpGet:
              path: /healthz
              port: http
              httpHeaders:
                - name: Host
                  value: healthz

BACKEND is pointing to a "dummy" nginx always returning 200 Everything seems to work well, but from time to time, "legit" requests as well as the two healthchecks above give the following error: context deadline exceeded (Client.Timeout exceeded while awaiting headers) That's all the information I have. Did anyone else encounter that issue? Do you have some best practice to follow? Thanks for your time and support

[fzipi]

Hi @igoooor !

Thanks for the report. Just to have additional context, can you add kubernetes version and runtimes you are using?

[igoooor]

Yes of course, I'm running my cluster on GKE, v1.23.8-gke.1900 on Container-optimised OS with containerd (cos_containerd)nodes

[theseion]

That message comes from the Go HTTP client, which must be the Kubernetes control plane. Since the container is configured as a proxy, it could be that your nginx container isn't responding. It could also be that the proxy itself doesn't respond. In both cases I would expect to see some issue on the cluster, e.g., container restarts.

If you don't see any issues, you could try stopping your nginx container vs. stopping the proxy and see whether you see different error messages. That could tell you which of the containers is causing the issue.

[igoooor]

Thank you for the hint, let me make a couple of tests in the next hours/day and I will update the issue accordingly

[fzipi]

Hi @igoooor! Did you had time to test this?