python
Clone per version (branch) to allow differential builds
Less differences on git updates should speed up the Sphinx builds basing on shared filesystem directory.
Testing strategy
Config simplified: two versions, two languages.
- run from main "first run", note the time
- run from main "second run", note the time
- run from branch "first run", note the time
- run from branch "second run", note the time
cpython / cpython_branches directories will be used so no clean-up is required between the runs.
Command: time python build_docs.py --build-root /tmp/docsbuild-test/build --www-root /tmp/docsbuild-test/www --skip-cache-invalidation --log-directory /tmp/docsbuild-test/logs --quick --group everyone.
Original script modified to always return should_rebuild() == True to allow testing the Sphinx part.
- main nth build:
597.85s user 35.97s system 180% cpu 5:51.44 total
- branch first build (long because of multiple git clones):
820.64s user 67.44s system 144% cpu 10:15.67 total
- branch second build:
574.04s user 32.99s system 171% cpu 5:54.70 total
No visible gain 😞 🤔 . Will try to continue the investigation soon.
Plan forward:
- expose Sphinx building logs in the script logs to get more debugging info
- compare the performance with single local builds
Beware, it was the case previously: b17bdc329d9c6c5461d9652d4983480fd8f64130
The commit messages tells:
Also having one clone per build takes 55GB on the server, and not reseting between builds cause bugs like https://bugs.python.org/issue44006.
Currently we have:
/dev/vda1 155G 17G 139G 11% /
