envoy-build-tools
envoy-build-tools copied to clipboard
envoyproxy
[WIP] Add minimal container for building docs
I attempted to do a local build of docs - and it turned out to be a lot more complex than simply running ./docs/build.sh from the root of the envoy repo
I then tried with the envoyproxy/envoy-build image used in ci. This worked but the image is 3.6GB and the average run to build the docs took around 15 mins.
To try and make this simpler and less resource-painful, i created this recipe which creates the environment required for building the docs only.
The image works by either mounting an envoy repo in the host to /source or by setting the ENVOY_REPO env var for the container, in which case it will fetch the repo at startup
The bare image for the env was around 670MB - but as it was fetching/building bazel and pip deps on each run it still took around 15 mins.
I therefore made the recipe run the docs build script once during image build. This makes the image slightly larger ~1.66GB but the runtime for building docs with the resultant image is reduced to around 3.5mins
I have left the travis script in the PR. If you are interested in taking this code in principle i will remove it and hook up azure pipelines.
Related to #90, something we can do is build a image without C++ toolchains (i.e. only bazel and python), for RBE build front end, for RBE jobs all C++ build/linking is happening remote so it work in all RBE CI. That will be similar to what I did for GetEnvoy:
gcr.io/getenvoy-package/bazel-linux-glibc 72a26368093ecda6721b548f0a04dfad2d0ea51d a30d7c2faad6 37 hours ago 960MB
gcr.io/getenvoy-package/rbe-linux-glibc 72a26368093ecda6721b548f0a04dfad2d0ea51d 8be68feaf9af 37 hours ago 2.68GB
I'm not a fan of warming bazel / pip caches in docker build phase for several reasons:
- Depends on master branch makes image build non-heremetic.
- Easy to invalidate all the caches once envoy changed requirements for python libs
- We're migrating python dependency handlings to bazel's
pip3_importthat makes pip3 cache not useful anyway (https://github.com/envoyproxy/envoy/pull/11108)
I'm not a fan of warming bazel / pip caches in docker build phase for several reasons:
the problem is that while removing the compiler makes the image smaller (as does using the official apt/debs from llvm), it doesnt make it any quicker to run build docs.
waiting 15 minutes to render small changes in docs is not a viable workflow for anyone focused on editing docs (3.5 mins is still pretty slow)
warming the caches afaict doesnt prevent newer versions being installed - certainly at least in the pip case - it just prevents you from downloading and recompiling deps that dont change very often, on every single build
the other option is to not warm the cache and then mount the caches in the image before running.
this has the advantage of keeping the image small, but it makes using it a load more complicated, and on first run - or lacking the proper mounts - very slow to use
the other option is to not warm the cache and then mount the caches in the image before running.
this has the advantage of keeping the image small, but it makes using it a load more complicated, and on first run - or lacking the proper mounts - very slow to use
We do mount in CI and default ci/run_envoy_docker.sh (by default it is /tmp/envoy-docker-build:/build, if doc build doesn't utilize that, then fix the script. All bazel caches are already in that mount path.
@phlax how did you run build docs? In my local box the after the second time the docs build is pretty quick:
ci/run_envoy_docker.sh 'ci/do_ci.sh docs' 0.10s user 0.10s system 0% cpu 1:14.94 total
A clean build in our CI (this doesn't include time to pull image, but include setting up bazel / pip cache) is about 5mins: https://app.circleci.com/pipelines/github/envoyproxy/envoy/31052/workflows/2c8a0668-0cba-4659-9609-d6f9acfd10a5/jobs/370762
We do mount in CI and default ci/run_envoy_docker.sh (by default it is /tmp/envoy-docker-build:/build, if doc build doesn't utilize that, then fix the script. All bazel caches are already in that mount path.
yep, as said, running the build inside that container works - its just a very large image
A clean build in our CI (this doesn't include time to pull image, but include setting up bazel / pip cache) is about 5mins: https://app.circleci.com/pipelines/github/envoyproxy/envoy/31052/workflows/2c8a0668-0cba-4659-9609-d6f9acfd10a5/jobs/370762
im afraid i cant access that. if its faster to compile i would guess that the build must be making use of the extra cores with circleci xlarge
here are timings on travis for 3 runs of building docs using the existing image and then using the image built in this pr
using ci/run_envoy_docker.sh
- download image 102.34s
- run1: 718.07s
- run2: 117.79s
- run3: 115.53s
using image from PR
- download image: 51.00s
- run1: 268.50s
- run2: 263.22s
- run3: 264.00s
the test run can be seen here https://travis-ci.org/github/phlax/envoy-docs/builds/716231495
not sure why this image is slower on subsequent builds - just caching is not speeding things up as much as i expected. Mounting the /build dir seems to be quicker than just warming caches - but i figure that can be done for this image too.
the point for me is that image size counts a lot. im working on several projects and regularly rmi images im not immediately working on. More often than not i avoid downloading/running anything locally and push tests to travis - so waiting for a large image to download is annoying if you are not using most of what is in it.
@phlax Right, so if you want to make a smaller image, don't warm bazel / pip cache because they are useless when /build is mounted. We don't really care travis perfomance because that's not where docs build are usually run.
Instead, if the goal is building a smaller image, do what I described in https://github.com/envoyproxy/envoy-build-tools/pull/92#issuecomment-670803877, and it can be used not only docs CI but all RBE based CI, i.e. release/gcc/asan/tsan, so the image is also tested. This helps your travis performance as well, why not?