Example Docker config improvements

[mikepurvis]

trafficstars

Hey— so, I've mucked about with things a bit in my own repo, and I just want to describe them a bit before I prepare a PR for it. The Dockerfile is somewhat shorter and simpler:

FROM ubuntu:trusty
MAINTAINER Mike Purvis

ENV DEBIAN_FRONTEND noninteractive
ENV BUILDBOT_CREATED nov_7_2014

# Install build stuff.
RUN apt-get update
RUN apt-get install -q -y python-virtualenv python-dev
RUN apt-get install -q -y reprepro cowbuilder debootstrap devscripts git git-buildpackage debhelper
RUN apt-get install -q -y debmirror

RUN virtualenv --no-site-packages /root/buildbot-env
RUN echo "export PATH=/root/buildbot-ros/scripts:${PATH}" >> /root/buildbot-env/bin/activate
RUN . /root/buildbot-env/bin/activate
RUN pip install rosdistro buildbot buildbot-slave

# Pick up the buildbot configuration
ADD . /root/buildbot-ros

# Create Buildbot.
RUN buildbot create-master /root/buildbot-ros
RUN buildslave create-slave /root/rosbuilder1 localhost:9989 rosbuilder1 [pw1]
RUN buildslave create-slave /root/rosbuilder2 localhost:9989 rosbuilder2 [pw2]

# Fix the file creation defaults.
RUN sed --in-place=.bak 's/umask = None/umask = 0022/' /root/buildbot-ros/buildbot.tac
RUN sed --in-place=.bak 's/umask = None/umask = 0022/' /root/rosbuilder1/buildbot.tac

EXPOSE 8010
CMD /root/buildbot-ros/run_server

And the build/run is now a two-parter, including the webserver for the resultant packages:

docker build -t buildbot-ros .
docker run -d -p 8000:80 -v /var/www:/usr/share/nginx/html:ro --name="packages" nginx
docker run -d --privileged -p 8010:8010 -v /var/www:/var/www:rw \
              --link=packages:packages --name="buildbot-ros" buildbot-ros

The major changes to watch out for here are:

• The buildbot-ros configuration files are being added in, rather than git-cloned in. This is better because docker can tell when you've changed the directory and it needs to rebuild. It also allows us to defer the key setup stuff until we're inside run_server.
• Because of this, the Dockerfile is in the root directory of the repo.
• The buildbot is able to access the built packages at http://packages/building; you can access them at http://[dockerd ip]:8000.

My proposal would be that the buildbot-ros repo (this one) gain an example signing key and Dockerfile in the root directory, with the idea that it can be deployed unmodified to inspect and play with, and that users deploying their own instances of buildbot-ros would clone this repo, create new keys, change the rosdistro url, and make any other changes needed for their use, and then build and deploy their own variant of the container.

For clarity, the run_server command (which now also handles the key stuff) would move to scripts/run_docker_container or similar.

Does this seem reasonable?

[jonbinney]

You can skip all the virtualenv/pip stuff on trusty, right? I'm not using docker, but on trusty i've got everything working using the apt-get versions. Makes the process slightly simpler.

[mikepurvis]

We can certainly skip the venv, since using docker provides containment. However, it's still worthwhile to get the most recent version of buildbot (0.8.8 in Trusty debs vs. 0.8.9 in pypi).

[mikepurvis]

I've ended up pursuing a more monolithic approach, with a single container holding the buildbot, slaves, nginx instance, and squid-deb-proxy for caching downloaded debs. Doing it this way, it comes down to an extremely simple one-liner to launch the whole thing.

This may not scale to a massive deployment, but I don't think buildbot-ros is really intended for that anyway. Having a single container to deploy saves a lot of complexity.

[mikepurvis]

@mikeferguson How open are you to merging a bunch of work which would basically make buildbot-ros a lot more convenient to deploy from Docker, but rather difficult/impossible to use any other way?

For example, the fix to #46 would be included, but that only works when the buildslave is executing as root within the container.

[mikepurvis]

No doc updates, but you can see the diff here:

https://github.com/mikepurvis/buildbot-ros/compare/mikeferguson:master...master

It's usable as-is directly from Docker hub: https://registry.hub.docker.com/u/mikepurvis/buildbot-ros/

Our in-house deployments are now set up to only very minimally extend this configuration— basically just adding in SSH and GPG keys, and swapping out the master.cfg file.

[mikeferguson]

@mikepurvis I'm not keen on making the mainline of buildbot-ros docker only (are you also assuming that the slaves always run inside the same docker instance as the master? that would actually not work for my new farm)

I've been a bit absent on the buildbot-ros stuff lately as I haven't had a good internet connection in the office. That is now fixed, and we're finally deploying buildbot-ros inhouse. I have also been talking with the folks at OSRF about the new buildfarm efforts they have been working on. Long term, we may try to refactor buildbot-ros to have a lot less code inside and leverage the existing scripts that the ROS farm uses. This has several benefits, including making sure that someone can replicate pretty much exactly what has been built on the OSRF farm.

[mikepurvis]

The batteries-included configuration runs the slaves and webserver all in the same container, but I don't think there's anything that demands that. My assumption is that a user wanting a more complicated multi-host setup would simply use the new OSRF Jenkins setup.

Going all-in on docker would have to be seen as focusing buildbot-ros on the use case where an individual or small team wants a setup with very, very minimal fuss (as can be seen by the fact that it's now literally a one liner to deploy the example container— because of that and how low-maintenance it is, we're actually running multiple of them in-house, targeting different clients/projects).

[mikepurvis]

Having thought about this some more, there's actually no reason this Docker config couldn't be used for a multi-container/multi-host deployment— in your overriding Dockerfile, you'd simply set ENV NUM_BUILDSLAVES 0, and then in your overriding master.cfg, manually specify the IP and ports of the slaves to connect to.

My Dockerfile for the overriding container is like so:

FROM mikepurvis/buildbot-ros

ADD master.cfg /buildbot-ros/master.cfg
ADD keys /buildbot-ros/keys

And then the fig.yml is:

buildbot:
  build: .
  ports:
  - 12002:80
  volumes:
  - /var/build/researchproducts/building:/building
  privileged: true

But as I say, it wouldn't be hard to create a more complicated deployment scheme (perhaps with Crane or Flocker), which launched the slaves on separate hosts— you'd still be reusing 99% of the "stock" container in this scenario.

[mikeferguson]

@mikepurvis I finally got a chance to review this in more detail. Here's my thoughts:

• Instead of just removing sudo -- we could make this a selected mode (perhaps --no-sudo or --use-sudo). That way it still works without docker.
• Will the env changes affect non-docker? I'm not sure
• I like pushing REPO_DIR into the environment -- we should probably do the same with the ROSDISTRO_INDEX (rather than hard coding it).
• If we don't delete scripts/aptrepo-create.sh, and make the sudo change above, I don't think this actually causes any major non-docker difficulties -- and so I'd be a big fan of a PR.

[mikepurvis]

The reason for wanting to remove sudo in the first place was so that the pbuilderrc setting would correctly persist through (see: https://github.com/mikeferguson/buildbot-ros/issues/46#issuecomment-65748501). We can make it optional, but by opting-in to sudo, you're opting out of having proper control over which OS is installed in the cow. Maybe there's some other way to deal with this, for example doing cmd = ['sudo', 'DIST=xyz', 'cowbuilder ...']. Having sudo doesn't harm docker apart from it erasing the environment going in.

One of the remaining fixes that's needed is separating the buildbot master configuration from the buildbot master state. I thought it was enough to set up the sqlite DB elsewhere (in the building repo), but I see now that most of the state is actually stored in the filesystem tree, not the DB. So to persist build logs and so on across container restarts, the entire buildbot-ros folder needs to be volumed-out.

The aptrepo-create.bash file is a bit of a mess. If you'd like it to be maintained, I'd suggest at least reimplementing it in python (as aptrepo-create) so that it uses the same empy template and basically just tacks on some argparse logic.