how to use coreos-assembler under docker with root

[vtolstov]

I'm have ci system that spawns docker containers for build. How can i build fcos image using coreos-assembler with my tree in this case?

[dustymabe]

You should be able to use the docker container runtime to run COSA and build artifacts. You'll need to adapt the documented podman invocation. One thing you'll definitely need is /dev/kvm for virtualization.

[mtalexan]

Reviving this because I just had to implement it for our system and found a big docker functionality caveat.

Mapping the podman command to docker is pretty straightforward. Here's the podman alias in a condensed form more relevant for CI builds:

cosa() {
   set -x
   podman run --rm -it --security-opt label=disable --privileged \
              --uidmap=1000:0:1 --uidmap=0:1:1000 --uidmap 1001:1001:64536 \
              -v ${PWD}:/srv/ \
	      --device /dev/kvm \
	      --device /dev/fuse \
              --tmpfs /tmp \
              -v /var/tmp:/var/tmp \
	      -v ${HOME}/.ssh:/home/builder/.ssh \
              quay.io/coreos-assembler/coreos-assembler:latest \
              "$@"
   rc=$?; set +x; return $rc
}

Note that if you have custom Root CAs for a proxy, your RPM repos, or your image registries, that's a whole other topic/issue you'll have to work around that's not covered here.

The equivalent of this in docker would basically be:

cosa() {
   set -x
   docker run --rm -it --security-opt label=disable --privileged \
              -v ${PWD}:/srv/ \
	      --device /dev/kvm \
	      --device /dev/fuse \
              --tmpfs /tmp \
              -v /var/tmp:/var/tmp \
	      -v ${HOME}/.ssh:/home/builder/.ssh \
              quay.io/coreos-assembler/coreos-assembler:latest \
              "$@"
   rc=$?; set +x; return $rc
}

Security Caveat

You will be running this container as a privileged rootful container, so it has full access to everything on your CI system with no restrictions as if it were running as root on your CI node's console.

Docker Functionality Caveat

The docker tool only supports running most privileged containers from the user with UID=1000. This is a limitation of the docker tool that there really is no workaround for.
Keep in mind that most container images are created with the internal user having UID=1000 and all the permissions within the container image are configured for that. So you have to start by assuming your UID inside the container has to match UID=1000 thru whatever process it takes. Non-privileged containers have no impact from the outside UIDs so it's no issue, but privileged ones map the apparent outside user UIDs to the container user UIDs directly. Unfortunately docker provides no option equivalent to podman's uidmap or userns commands that allow customization of this mapping between the calling environment and the container. So we have a transitive requirement that your docker privileged container calling environment has to somehow match the caller having UID=1000.
For rootless containers, the "calling environment" is actually a user-specific namespace defined in the /etc/subuid field. But the effect of it's application is that the command-line user appears to be UID=0 (root) within the user-specific namespace. However before we go down the path of trying to figure out how to customize the user namespace for docker calls, it's good to know how to get docker to run rootless containers in the first place. According to docker, you first have to uninstall/halt parts of your rootful docker. Then you can modify your rootful installation to instead be exclusively rootless. Already we can see that you're prohibited from running both rootless and rootful docker containers, which is obviously a non-starter for most users. And that's before we get into the technical complexities of trying to customize a user namespace properly to get that UID=1000 mapping.
TL;DR it's effectively only possible to directly map all existing user UIDs, or no user UIDs into docker containers, and no translation/remapping is supported.