frankenphp Website becoming unresponsive (Frankenphp Worker Mode)

What happened?

For a while now we have been running Frankenphp worker mode (using symfony-docker) in production. Occasionally (once every week or two) the website will stop responding completely.

I have a lot of these errors in the log:

Uncaught PHP Exception Symfony\Component\ErrorHandler\Error\FatalError: "Error: Maximum execution time of 30 seconds exceeded" at BinaryFileResponse.php line 338

I am working on adding the solution at https://frankenphp.dev/docs/x-sendfile/ which should prevent these errors, and I don't know for sure if they are directly linked to the website going offline as they have been happening consistenly. I can't see anything obvious in the logs that would be causing it. I've recently discovered the additional Frankenphp metrics that can be added to the Caddy metrics and had a look at it this morning during one of the "crashes":

frankenphp_busy_threads 16 frankenphp_busy_workers{worker="/app/public/index.php"} 7535 frankenphp_queue_depth 0 frankenphp_ready_workers{worker="/app/public/index.php"} 16 frankenphp_total_threads 17 frankenphp_total_workers{worker="/app/public/index.php"} 16 frankenphp_worker_crashes{worker="/app/public/index.php"} 547 frankenphp_worker_queue_depth{worker="/app/public/index.php"} 7520 frankenphp_worker_request_count{worker="/app/public/index.php"} 435737 frankenphp_worker_request_time{worker="/app/public/index.php"} 155378.1959565056 frankenphp_worker_restarts{worker="/app/public/index.php"} 778 go_build_info{checksum="",path="github.com/dunglas/frankenphp/caddy",version="(devel)"} 1

The part that concerns me here is the 7535 busy workers and 7520 queue when I only have 16 workers. That sounds to me like the workers have all got stuck. I'm not sure that can be related to the file serving because there is a 30 second timeout which would cause the worker to crash and restart.

I'm not sure if this is a bug in Frankenphp or if not I'm hoping you can help me dig in to this a bit more and understand/investigate what I might have misconfigured?

Build Type

Docker (Debian Bookworm)

Worker Mode

Yes

Operating System

GNU/Linux

CPU Architecture

x86_64

PHP configuration

Relevant log output

Uncaught PHP Exception Symfony\Component\ErrorHandler\Error\FatalError: "Error: Maximum execution time of 30 seconds exceeded" at BinaryFileResponse.php line 338

Aug 08 '25 08:08 stevelewis33

Most likely this is a metrics reporting issue.

Note that if you are used to working in Linux, frankenphp's timeouts are based on wall clock time, not execution time. If you are using Linux with fpm/apache, it is based on execution time. Every other operating system is based on wall clock time. So, if your downloads are taking longer than 30s, it will timeout on your end.

Aug 08 '25 09:08 withinboredom

These errors indicate a timeout when writing a file. PHP currently has no stack trace for fatal errors so it's sometimes hard to figure out what caused the timeout (this might get fixed in 8.5, don't quote me on that though). Could you maybe see which files are causing these timeouts (are they very big files?)

Your server might get unresponsive if all threads are stuck reading/writing files and timing out for some reason.

There are a few.things you can do here,.for once increase max_threads to be able to serve more concurrent requests and track when these spikes happen. Another thing you could do is have these files served by Caddy (as you mentioned).

Aug 09 '25 22:08 AlliBalliBaba

Thanks for the feedback so far, earlier this week I deployed an update which passed off serving our large binary files to Caddy by using BinaryFileResponse::trustXSendfileTypeHeader(); in Symfony and the following additional Caddy config:

    request_header X-Sendfile-Type x-accel-redirect
    request_header X-Accel-Mapping /home/tlxapp/storage=/storage

    intercept {
        @accel header X-Accel-Redirect *
    	handle_response @accel {
            root * /home/tlxapp
    		rewrite * {resp.header.X-Accel-Redirect}
    		method * GET

    		# Remove the X-Accel-Redirect header set by PHP for increased security
    		header -X-Accel-Redirect

    		file_server
    	}
    }

This is definitely working, the files are no longer served by PHP and the "Maximum execution time of 30 seconds exceeded" errors have stopped.

Unfortunately this morning the site stopped responding again, the metrics once again show:

frankenphp_busy_threads 16
frankenphp_busy_workers{worker="/app/public/index.php"} 784
frankenphp_queue_depth 0
frankenphp_ready_workers{worker="/app/public/index.php"} 16
frankenphp_total_threads 17
frankenphp_total_workers{worker="/app/public/index.php"} 16
frankenphp_worker_crashes{worker="/app/public/index.php"} 5
frankenphp_worker_queue_depth{worker="/app/public/index.php"} 768
frankenphp_worker_request_count{worker="/app/public/index.php"} 31162
frankenphp_worker_request_time{worker="/app/public/index.php"} 13361.350223256197
frankenphp_worker_restarts{worker="/app/public/index.php"} 56

As you can see the quantity of crashes has been reduced but somehow we end up with all workers not responding.

We do make a large quantity of curl requests to access data from our PIM (Product Information Management) API which could cause threads to take a few seconds, but all curl requests are limited to 10 seconds before timing out and each request will make 1-2 curl requests at most. Do you think this could potentially be the cause that all 16 workers are just busy / held up by external API calls?

Aug 13 '25 10:08 stevelewis33

Hey yeah that's definitely possible, if you expect very slow IO, don't hesitate to increase max_threads. You can set it quite highly to better estimate how many request you are potentially handling concurrently

{
  frankenphp{
    num_threads 16
    max_threads 100
  }
}

max_threads spawns the threads at runtime only when they are needed and will stop them again if the are idle for too long

Aug 13 '25 12:08 AlliBalliBaba

I've been on holiday for 2 weeks, before I went away I configured the threads as follows:

frankenphp {
    {$FRANKENPHP_CONFIG}
    num_threads 32
    max_threads 128
}

Unfortunately our web site still became unresponsive on 2 occasions, the most recent being this morning, the metrics shows as follows again:

frankenphp_busy_threads 112 frankenphp_busy_workers{worker="/app/public/index.php"} 1027 frankenphp_queue_depth 0 frankenphp_ready_workers{worker="/app/public/index.php"} 112 frankenphp_total_threads 32 frankenphp_total_workers{worker="/app/public/index.php"} 112 frankenphp_worker_crashes{worker="/app/public/index.php"} 6 frankenphp_worker_queue_depth{worker="/app/public/index.php"} 915 frankenphp_worker_request_count{worker="/app/public/index.php"} 50802 frankenphp_worker_request_time{worker="/app/public/index.php"} 18632.786395583033 frankenphp_worker_restarts{worker="/app/public/index.php"} 106

Does anyone please have any ideas/suggestions before I seek some additional paid consultancy?

Sep 04 '25 15:09 stevelewis33

Yes looking at the metric I'm pretty sure this is due to all requests being queued up waiting for the PIM API responses (or some other slow endpoint). The api probably becomes slower as more requests are being made to it.

Something else you can also do is set a max_wait_time and max_execution_time in case requests are stalled for too long. It's better to reject than queue them if the server's overwhelmed. Being stricter with curl timeouts may help as well.

If you think the stalling may happen due to slow connection reading/writing, you can also set up read/write timeouts

some suggestions:

{
  frankenphp {
      {$FRANKENPHP_CONFIG}
      num_threads 32
      max_threads 256 # threads jacked up even more (make sure to watch much memory consumption)
      max_wait_time 15s # the maximum time requests can wait for threads before being rejected
  
      php_ini max_execution_time 30 # the maximum time a request may execute on the PHP side
  }
  
  # read/write timeouts (optional)
  servers {
      timeouts {
          idle 30s
          read_body 60s # make sure file uploads don't take longer than 60s
          read_header 30s
          write 30s
      }
  }
}

In the end, requests that take 20+ seconds is something you should ideally avoid altogether. There are a bunch of methods how you can prevent them from happening. Typically, you would identify these slow endpoints and process them in a queue or task. Once the task is finished, you can notify the user asynchronously via websocket/polling/etc (you might not even need to notify them at all).

If that is not an option, you can also process these slow requests with a dedicated 'slow' worker on specific request paths. This will prevent the slow endpoints from interfering with regular HTTP requests and has the additional benefit of being able to limit the concurrency to the external api, some example code:


# global config up here

https://example.com {
  php_server {
    worker {
      file index.php # regular worker entry point
      num 32 # maximum 32 concurrent requests to the slow api (to not overwhelm it)
      match /slow/request/path/* # all requests with this wildcard path go to the dedicated 'slow' threads
    }

    worker {
      file index2.php # must currently be a different entry file than the other worker (just copy index.php)
      num 32 # separate thread for all other requests
      match * # match al other request paths
    }
  }
}

Sep 04 '25 16:09 AlliBalliBaba