php-fpm-healthcheck icon indicating copy to clipboard operation
php-fpm-healthcheck copied to clipboard

Readiness probe error not handled due to php-fpm-healthcheck timeout

Open xriser opened this issue 4 years ago • 7 comments

When php-fpm stuck and didn't respond, php-fpm-healthckeck can't handle this, and timeouts to connect to the port.

root@deployment-php-5bddfd7964-7ctmf:/app# php-fpm-healthcheck -v Trying to connect to php-fpm via: localhost:9000/status ^C So, php-fpm-healthcheck doesn't return fail status in a reasonable time.

Since K8s doesn't handle timeouts healthcheck, it doesn't work. Because Kubernetes remove the pod from service only when heathcheck returning fail event. Readiness probe errored: rpc error: code = DeadlineExceeded desc = context deadline exceeded Readiness probe errored, but pod didn't remove from services.

Is there any way to setup a reasonable timeout to connect to the PHP port for the php-fpm-healthcheck?

xriser avatar Oct 19 '20 09:10 xriser

I'm getting the same - makes it a little useless when PHP-FPM stops responding

chrispage1 avatar Dec 16 '20 17:12 chrispage1

Just to update with this one for anyone else searching, I managed to resolve using BusyBox timeout command in my Kubernetes liveness probe...

livenessProbe:
  exec:
    command:
     - timeout
     - "1"
     - php-fpm-healthcheck
     - --listen-queue=15 # fails if there are more than 10 processes waiting in the fpm queue

This means that if php-fpm-healthcheck doesn't return success within 1 second it'll assume its dead.

chrispage1 avatar Dec 16 '20 18:12 chrispage1

Just to update with this one for anyone else searching, I managed to resolve using BusyBox timeout command in my Kubernetes liveness probe...

livenessProbe:
  exec:
    command:
     - timeout
     - "1"
     - php-fpm-healthcheck
     - --listen-queue=15 # fails if there are more than 10 processes waiting in the fpm queue

This means that if php-fpm-healthcheck doesn't return success within 1 second it'll assume its dead.

Did the same before. It works livenessProbe: exec: command: - /usr/bin/timeout - "3" - /usr/local/bin/php-fpm-healthcheck - --listen-queue=100 initialDelaySeconds: 300

xriser avatar Dec 17 '20 09:12 xriser

@xriser do you find PHP-FPM becomes unresponsive often? I'm having problems where the node will restart about 10 times a day due to PHP-FPM not responding

chrispage1 avatar Dec 20 '20 15:12 chrispage1

@xriser do you find PHP-FPM becomes unresponsive often? I'm having problems where the node will restart about 10 times a day due to PHP-FPM not responding

no, it depends on the application. If the app doesn't free php child they start to grow and running out of active child limits.

xriser avatar Dec 20 '20 18:12 xriser

If we implement a --timeout feature, we can also close #28 .

wandersonwhcr avatar Jan 14 '22 15:01 wandersonwhcr

Is this doc relevant?

https://kubernetes.io/docs/tasks/configure-pod-container/configure-liveness-readiness-startup-probes/#configure-probes

But yes, I think we must implement a --timeout feature, too.

image

wandersonwhcr avatar Jan 17 '22 19:01 wandersonwhcr