Kubernetes configmap change leads to failed to open stream

[raffis]

Description

Is: <br /> <b>Warning</b>: file_get_contents(/usr/share/balloon/config/config.yaml): failed to open stream: No such file or directory in <b>/usr/share/balloon/vendor/hassankhan/config/src/FileParser/Yaml.php</b> on line <b>29</b><br /> <br /> <b>Fatal error</b>: Uncaught ErrorException: Cannot modify header information - headers already sent by (output started at /usr/share/balloon/vendor/hassankhan/config/src/FileParser/Yaml.php:29) in /usr/share/balloon/vendor/gyselroth/micro-http/src/Response.php:119 Stack trace: #0 [internal function]: Balloon\Bootstrap\AbstractBootstrap-&gt;Balloon\Bootstrap\{closure}(2, 'Cannot modify h...', '/usr/share/ball...', 119, Array) #1 /usr/share/balloon/vendor/gyselroth/micro-http/src/Response.php(119): header('Content-Type: a...') #2 /usr/share/balloon/vendor/gyselroth/micro-http/src/Response.php(183): Micro\Http\Response-&gt;sendHeaders() #3 /usr/share/balloon/src/lib/Bootstrap/Http.php(147): Micro\Http\Response-&gt;send() #4 [internal function]: Balloon\Bootstrap\Http-&gt;Balloon\Bootstrap\{closure}(Object(ErrorException)) #5 {main} thrown in <b>/usr/share/balloon/vendor/gyselroth/micro-http/src/Response.php</b> on line <b>119</b><br /> clearstatcache has been introduced in v2.0.9 but the error still persists.

Should: No error after change.

balloon server version

2.0.10

PHP version

Operating system

Webserver

[juckerf]

Seems still to be a problem with php and the realpath cache.

Analysis

Preparations

Patched src/lib/Bootstrap/ContainerBuilder.php for debug logging:

44	file_put_contents("/tmp/log",time() . " - " . getmypid() . " - " . realpath($path) . "
45	",FILE_APPEND);

Test

Config Change:

TIMESTAMP: 1532683121

    $ kubectl -n balloon-stage edit cm balloon-config && date +%s
    configmap "balloon-config" edited

Took some time to be reflected in container:

TIMESTAMP: 1532683121

    $ ls -la /etc/balloon/
    total 12
    drwxrwxrwx  3 root root 4096 Jul 27 09:15 .
    drwxr-xr-x 81 root root 4096 Jul 19 10:20 ..
    drwxr-xr-x  2 root root 4096 Jul 27 09:15 ..7987_27_07_11_15_22.350852074
    lrwxrwxrwx  1 root root   31 Jul 27 09:15 ..data -> ..7987_27_07_11_15_22.350852074
    lrwxrwxrwx  1 root root   18 Jul 19 10:18 config.yaml -> ..data/config.yaml

[...]

TIMESTAMP: 1532683167

    $ ls -la /etc/balloon/
    total 12
    drwxrwxrwx  3 root root 4096 Jul 27 09:19 .
    drwxr-xr-x 81 root root 4096 Jul 19 10:20 ..
    drwxr-xr-x  2 root root 4096 Jul 27 09:19 ..7987_27_07_11_19_25.387807297
    lrwxrwxrwx  1 root root   31 Jul 27 09:19 ..data -> ..7987_27_07_11_19_25.387807297
    lrwxrwxrwx  1 root root   18 Jul 19 10:18 config.yaml -> ..data/config.yaml

Took even longer to be reflected in realpath of all php processes (/tmp/log):

TIMESTAMP	-	PID	-	realpath
1532683164	-	8	-	/etc/balloon/..7987_27_07_11_15_22.350852074/config.yaml
1532683166	-	26	-	/etc/balloon/..7987_27_07_11_15_22.350852074/config.yaml
1532683169	-	7	-	/etc/balloon/..7987_27_07_11_19_25.387807297/config.yaml
1532683171	-	8	-	/etc/balloon/..7987_27_07_11_15_22.350852074/config.yaml
1532683174	-	26	-	/etc/balloon/..7987_27_07_11_15_22.350852074/config.yaml
1532683177	-	7	-	/etc/balloon/..7987_27_07_11_19_25.387807297/config.yaml
1532683179	-	8	-	/etc/balloon/..7987_27_07_11_15_22.350852074/config.yaml
1532683182	-	26	-	/etc/balloon/..7987_27_07_11_15_22.350852074/config.yaml
1532683184	-	7	-	/etc/balloon/..7987_27_07_11_19_25.387807297/config.yaml
1532683187	-	8	-	/etc/balloon/..7987_27_07_11_15_22.350852074/config.yaml
1532683189	-	26	-	/etc/balloon/..7987_27_07_11_15_22.350852074/config.yaml
1532683192	-	7	-	/etc/balloon/..7987_27_07_11_19_25.387807297/config.yaml
1532683194	-	8	-	/etc/balloon/..7987_27_07_11_15_22.350852074/config.yaml
1532683197	-	26	-	/etc/balloon/..7987_27_07_11_15_22.350852074/config.yaml
1532683199	-	7	-	/etc/balloon/..7987_27_07_11_19_25.387807297/config.yaml
1532683202	-	8	-	/etc/balloon/..7987_27_07_11_15_22.350852074/config.yaml
1532683204	-	26	-	/etc/balloon/..7987_27_07_11_15_22.350852074/config.yaml
1532683207	-	7	-	/etc/balloon/..7987_27_07_11_19_25.387807297/config.yaml
1532683209	-	8	-	/etc/balloon/..7987_27_07_11_15_22.350852074/config.yaml
1532683212	-	26	-	/etc/balloon/..7987_27_07_11_15_22.350852074/config.yaml
1532683214	-	7	-	/etc/balloon/..7987_27_07_11_19_25.387807297/config.yaml
1532683216	-	8	-	/etc/balloon/..7987_27_07_11_15_22.350852074/config.yaml
1532683219	-	26	-	/etc/balloon/..7987_27_07_11_15_22.350852074/config.yaml
1532683221	-	7	-	/etc/balloon/..7987_27_07_11_19_25.387807297/config.yaml
1532683224	-	8	-	/etc/balloon/..7987_27_07_11_15_22.350852074/config.yaml
1532683226	-	26	-	/etc/balloon/..7987_27_07_11_15_22.350852074/config.yaml
1532683229	-	7	-	/etc/balloon/..7987_27_07_11_19_25.387807297/config.yaml
1532683231	-	8	-	/etc/balloon/..7987_27_07_11_15_22.350852074/config.yaml
1532683234	-	26	-	/etc/balloon/..7987_27_07_11_15_22.350852074/config.yaml
1532683237	-	7	-	/etc/balloon/..7987_27_07_11_19_25.387807297/config.yaml
1532683239	-	8	-	/etc/balloon/..7987_27_07_11_15_22.350852074/config.yaml
1532683242	-	26	-	/etc/balloon/..7987_27_07_11_15_22.350852074/config.yaml
1532683244	-	7	-	/etc/balloon/..7987_27_07_11_19_25.387807297/config.yaml
1532683246	-	8	-	/etc/balloon/..7987_27_07_11_15_22.350852074/config.yaml
1532683249	-	26	-	/etc/balloon/..7987_27_07_11_15_22.350852074/config.yaml
1532683251	-	7	-	/etc/balloon/..7987_27_07_11_19_25.387807297/config.yaml
1532683254	-	8	-	/etc/balloon/..7987_27_07_11_19_25.387807297/config.yaml
1532683256	-	26	-	/etc/balloon/..7987_27_07_11_15_22.350852074/config.yaml
1532683259	-	7	-	/etc/balloon/..7987_27_07_11_19_25.387807297/config.yaml
1532683261	-	8	-	/etc/balloon/..7987_27_07_11_19_25.387807297/config.yaml
1532683264	-	26	-	/etc/balloon/..7987_27_07_11_15_22.350852074/config.yaml
1532683267	-	7	-	/etc/balloon/..7987_27_07_11_19_25.387807297/config.yaml
1532683269	-	8	-	/etc/balloon/..7987_27_07_11_19_25.387807297/config.yaml
1532683272	-	26	-	/etc/balloon/..7987_27_07_11_15_22.350852074/config.yaml
1532683274	-	7	-	/etc/balloon/..7987_27_07_11_19_25.387807297/config.yaml
1532683277	-	8	-	/etc/balloon/..7987_27_07_11_19_25.387807297/config.yaml
1532683279	-	26	-	/etc/balloon/..7987_27_07_11_15_22.350852074/config.yaml
1532683282	-	7	-	/etc/balloon/..7987_27_07_11_19_25.387807297/config.yaml
1532683284	-	8	-	/etc/balloon/..7987_27_07_11_19_25.387807297/config.yaml
1532683287	-	26	-	/etc/balloon/..7987_27_07_11_19_25.387807297/config.yaml

[juckerf]

seems like opcache joined the party... if i add a opcache_reset in src/lib/Bootstrap/ContainerBuilder.php (see below) the config change is reflected immediately (Note: an opcache_invalidate(__FILE__) does not suffice...)

    44	opcache_reset(); file_put_contents("/tmp/log",time() . " - " . getmypid() . " - " . realpath($path) . " - " . __FILE__ . "\n",FILE_APPEND);

[raffis]

running opcache_reset() each request overturns the usage of opcache. Maybe we need to store the config in apc and load it from there if the file can not be opened, but sounds fishy

[juckerf]

running opcache_reset() each request overturns the usage of opcache I'm aware of that... my last comment was merely to report my findings during debugging :wink:

[raffis]

https://pracucci.com/php-realpath-cache-and-kubernetes-secrets-configmap-updates.html

[raffis]

This will be solved with #338 since we will implement the http stack in balloon itself and moving away from a php-fpm/nginx construct.