fix: use xargs to speed up chown during init
init-code-server currently uses find -exec which accumulates all files before chowning them.
This is slow for users with a lot of files. Using xargs makes it much faster.
- [x] I have read the contributing guideline and understand that I have made the correct modifications
Description:
find -print0andxargs -0change everything to use null characters instead of whitespace between paths-n 1000is the number of files tochownper xarg execution- Performance capped once I got above ~20 and stayed there even up to 1 million, 1000 seemed a nice middle-ground between those which should maximize performance across different machine specs.
-P $(nproc)set the parallelism to the number of cores
Benefits of this PR and context:
Old
$ time find /config -path /config/workspace -prune -o -exec chown abc:abc {} +
real 12m19.069s
user 0m1.601s
sys 2m33.088s
New
$ time find /config -path /config/workspace -prune -o -print0 | xargs -0 -n 1000 -P $(nproc) chown abc:abc
real 0m46.416s
user 0m2.339s
sys 2m13.735s
How Has This Been Tested?
Manually by modifying the init script when building my image.
Source / References:
I am a bot, here are the test results for this PR: https://ci-tests.linuxserver.io/lspipepr/code-server/4.91.1-pkg-ab8f4cb9-dev-86f7517846cbf4b14dc7e686aad76a69d54162ae-pr-183/index.html https://ci-tests.linuxserver.io/lspipepr/code-server/4.91.1-pkg-ab8f4cb9-dev-86f7517846cbf4b14dc7e686aad76a69d54162ae-pr-183/shellcheck-result.xml
| Tag | Passed |
|---|---|
| amd64-4.91.1-pkg-ab8f4cb9-dev-86f7517846cbf4b14dc7e686aad76a69d54162ae-pr-183 | ✅ |
| arm64v8-4.91.1-pkg-ab8f4cb9-dev-86f7517846cbf4b14dc7e686aad76a69d54162ae-pr-183 | ✅ |
I am a bot, here are the test results for this PR: https://ci-tests.linuxserver.io/lspipepr/code-server/4.92.2-pkg-978f6c7a-dev-b9726974619d9578ec87f3f77b80da7a3c871cb3-pr-183/index.html https://ci-tests.linuxserver.io/lspipepr/code-server/4.92.2-pkg-978f6c7a-dev-b9726974619d9578ec87f3f77b80da7a3c871cb3-pr-183/shellcheck-result.xml
| Tag | Passed |
|---|---|
| amd64-4.92.2-pkg-978f6c7a-dev-b9726974619d9578ec87f3f77b80da7a3c871cb3-pr-183 | ✅ |
| arm64v8-4.92.2-pkg-978f6c7a-dev-b9726974619d9578ec87f3f77b80da7a3c871cb3-pr-183 | ✅ |
@aptalca any interest in this?
I'm sure there was a reason we didn't use xargs and I don't remember what it was, possibly a difference between the Alpine and Ubuntu implementations that broke something. Let me see if I can dig it up.
Bear in mind any change would impact every image we publish and not just code-server.
Bear in mind any change would impact every image we publish and not just code-server.
Could you explain what you mean by this? Isn't this change only in the init code for code-server? Or are you saying this line (or similar) exists in other images?
Oh sorry, still half asleep, didn't realise we had an errant hand-crafted chown still in the code-server init. Most of our chowns are handled by https://github.com/linuxserver/docker-mods/blob/mod-scripts/lsiown.v1, which does the same thing as the line you've modified.
Running this on alpine:latest and ubuntu:latest produces the expected results.
find / -path /etc/ -prune -o -print0 | xargs -0 -n 1000 -P $(nproc) echo chown root:root
And removing the echo seems to be doing what I'd expect too (mostly spitting out errors about read only). So I'm not sure what the issue encountered could have been.
I can make a similar change to lsiown.v1 to use xargs, but I don't know how to integrate usage of that script into this repo.
TL;DR we can't remember, but I think it was an issue with low-powered devices performing very badly with the xargs parallel chowns, especially those with slow storage.
One question I have is what have you got in your /config mount (outside of /config/workspace) that's taking 12 minutes to chown?
TL;DR we can't remember, but I think it was an issue with low-powered devices performing very badly with the xargs parallel chowns, especially those with slow storage.
Capping the parallelism at something like $((($(nproc) + 3)/4)) could potentially mitigate this.
One question I have is what have you got in your /config mount (outside of /config/workspace) that's taking 12 minutes to chown?
.rustup, .cargo, and .npm are the big ones, totaling to around 1 million files.
I am a bot, here are the test results for this PR: https://ci-tests.linuxserver.io/lspipepr/code-server/4.93.1-pkg-3d426902-dev-6d284b73b59bf76283aef06322ab5f2348f118b8-pr-183/index.html https://ci-tests.linuxserver.io/lspipepr/code-server/4.93.1-pkg-3d426902-dev-6d284b73b59bf76283aef06322ab5f2348f118b8-pr-183/shellcheck-result.xml
| Tag | Passed |
|---|---|
| amd64-4.93.1-pkg-3d426902-dev-6d284b73b59bf76283aef06322ab5f2348f118b8-pr-183 | ✅ |
| arm64v8-4.93.1-pkg-3d426902-dev-6d284b73b59bf76283aef06322ab5f2348f118b8-pr-183 | ✅ |
@Daniel-Aaron-Bloom can you give this test image a try?
ghcr.io/linuxserver/lspipepr-code-server:4.93.1-pkg-3d426902-dev-42aae8bde4a11d239abb91e5fd75f0e843ecc8a9-pr-188
It's the result of #188 , which should ignore the dev/cache folders during chown
scratch that, we decided to make chown conditional instead
