Document guideline for tweaking php-fpm pm settings with APC

[mikehaertl]

I had some hard time to figure out optimal settings for php-fpm with apc enabled. Over the years my rule of thumb was:

Look at the RSS column ps aux to find out the max size of a single php-fpm process. Then divide the RAM you want to spend for php-fpm by this value. This gives you the max. number of processes you can set as pm.max_children.

But with apcu it seems, this no longer is true. With a large APC memory setting RSS is way too high. Here's a real world example: I run php-fpm inside a docker container with apc.shm_size set to 1G. With apc.php I see that 400 MB are currently used (~600MB free). pm.max_children is set to 50 in php-fpm.

Now if i check with top I get something like this:

  PID USER      PR  NI    VIRT    RES    SHR S  %CPU %MEM     TIME+ COMMAND
14523 82        20   0 1339424 223832 211144 S   4,7  1,4   0:05.33 php-fpm7
14408 82        20   0 1341652 229964 213772 S   4,3  1,4   0:07.54 php-fpm7
14430 82        20   0 1339628 230712 216240 S   4,3  1,4   0:06.44 php-fpm7
...

Each process seems to use ~230MB. I currently have 32 children running, so this would sum up to 32 * 230MB = 7360 MB. But if I check the real memory used by the container with docker stats I get this:

CONTAINER           CPU %               MEM USAGE / LIMIT       MEM %               NET I/O             BLOCK I/O           PIDS
ffa979198f79        66.04%              809 MiB / 15.65 GiB     5.05%               432 MB / 3.24 GB    0 B / 0 B           0

The container only uses 809MiB. So it seems, that my calculation above was completely wrong.

Now I wonder what's a good way to get the real size of a single php-fpm process and the best setting for pm.max_children with APC.

I think something like this should be documented somewhere.

[mikehaertl]

@nikic According to #298 you seem to have a good understanding of APC's memory model. Could you maybe shed some light on the above questions?

I'm also monitoring APC with Munin and see, that the cache is frequently reset, even though the limit is not reached yet:

Any idea what could cause this?

The code to get the above numbers is basically this:

        $apc = apcu_cache_info(true);
        $mem = apcu_sma_info(true);
        $memTotal = $mem['num_seg'] * $mem['seg_size'];
        $memFree = $mem['avail_mem'];
        return $this->asJson([
            'memory_total' => $memTotal,
            'memory_used' => $memTotal - $memFree,
            'memory_vars' => $apc['mem_size'],
            'cache_entries' => $apc['num_entries'],
            'cache_hits' => $apc['num_hits'],
            'cache_misses' => $apc['num_misses'],
        ]);

[schtorch]

I would check if your webserver gets reloaded e.g. by yourself or from a cron like logrotate. A reload will reset the cache.

[mikehaertl]

@schtorch I also considered that. But as you see from the graph the reset happen at random times. And actually the process runs inside a docker container and logging happens on the host system. There's no restart of php-fpm or the like.

[nikic]

@mikehaertl top output is notoriously hard to interpret. RES is not a good measure of per-process memory usage, because is also includes (resident) shared memory. Using RES-SHR (with a fixed cost of SHR across all processes) is not correct either, because SHR is only potentially shared memory, not all of which is necessarily shared. For APCu in particular, the full size of the 1G segment you're using will be part of VIRT (as mmapped memory), but only a part of it will be resident.

Unfortunately, none of these metrics really tell you how the actual memory usage will look like (and it's not really possible anyway). I think that (RES-SHR)*N+SHR is probably the closest approximation to actual usage, e.g. in your example 216240+(230712-216240)*32 = 679344, which underestimates the actual usage, but does not give the misleading results of multiplying the resident size.

In the end, I would just recommend trying out different configurations and seeing which one shows the memory usage characteristics you would like to see.

As for the apcu restarts, I'm not sure why this is happening. It does seem rather suspicious that restarts are triggered at 75% load. The reason could be that a very large memory allocation is attempted, while the memory is too fragmented to find a single continuous block of memory with sufficient size. Though I feel like this would have to be quite a huge allocation in your example. Can you please check whether you have apc.smart set to a value other than 0? Also, do you have a cache.ttl set?

[mikehaertl]

@nikic Thanks a lot for the explanation. So there's no golden path and you always need some try & error to find the best settings. Good to know.

Regarding the restarts I don't know what's causing them. The confusing part is, that sometimes "Cache full count" increases but sometimes it reads 0 and it seems as if the process restarted (which it didn't). I have set pm.max_requests = 200 though. But I assume that APCu is managed by the master process so it should not be affected if child processes restart. But maybe I'm wrong.

Here's all the numbers from apc.php:

General Cache Information

APCu Version | 5.1.11
PHP Version | 7.1.17
Server Software | nginx/1.12.2
Shared Memory | 1 Segment(s) with 2.0 GBytes (mmap memory)
Start Time | 2018/06/25 11:10:10
Uptime | 19 hours and 23 minutes

Cache Information

Cached Variables | 160238 ( 1.1 GBytes)
Hits | 5868414
Misses | 441800
Request Rate (hits, misses) | 90.41 cache requests/second
Hit Rate | 84.08 cache requests/second
Miss Rate | 6.33 cache requests/second
Insert Rate | 7.14 cache requests/second
Cache full count | 0

Runtime Settings

apc.coredump_unmap | 0
apc.enable_cli | 0
apc.enabled | 1
apc.entries_hint | 131072
apc.gc_ttl | 3600
apc.mmap_file_mask |  
apc.preload_path |  
apc.serializer | php
apc.shm_segments | 1
apc.shm_size | 2G
apc.slam_defense | 1
apc.smart | 0
apc.ttl | 0
apc.use_request_time | 1
apc.writable | /tmp