[feature] Add chroot mapping to task config

[dkong]

Description

The exec driver gets it's chroot directories from either the default list or the list specified in the nomad client config.

For additional flexibility, it would be nice if we could specify the chroot mapping in the driver config of the job spec as well. The job spec chroot would have to be a subset of the nomad client chroot otherwise the driver will return an error.

Currently we're constrained to a single chroot mapping in the nomad client config that has to be appropriate for all types of jobs we want to run.

Example

nomad client config

client {
    chroot_env {
        "/bin/ls" = "/bin/ls"
        "/etc/ld.so.cache" = "/etc/ld.so.cache"
        "/etc/ld.so.conf" = "/etc/ld.so.conf"
        "/etc/ld.so.conf.d" = "/etc/ld.so.conf.d"
        "/lib" = "/lib"
        "/lib64" = "/lib64"
    }
}

valid job spec chroot

task "example" {
    driver = "exec"

    config {
        command = "/bin/sleep"
        args = ["1"]

        chroot_env {
            "/bin/ls" = "/bin/ls"
            "/lib/libiptc.so.0" = "/lib/libiptc.so.0"
        }
    }
}

invalid job spec chroot (/etc/sysctl.conf is not whitlisted in nomad client chroot)

task "example" {
    driver = "exec"

    config {
        command = "/bin/sleep"
        args = ["1"]

        chroot_env {
            "/bin/ls" = "/bin/ls"
            "/etc/sysctl.conf" = "/etc/sysctl.conf"
        }
    }
}

[dadgar]

Hey,

Would it be interesting to you if you could ship the rootfs that was used? So that each task could have exactly what it needed. We have been toying around with that idea

[jshaw86]

@dadgar could you elaborate on what you mean by rootfs?

So that each task could have exactly what it needed

Yes that's exactly what we want so if we're running a python application we only include python things, if it's a node application only node things etc...

[devendram]

@dadgar can you elaborate a bit on what you meant by shipping rootfs ? If I am getting it right, I guess you mean having the fs (packaged) for job included in job spec. Please provide more details.

[jshaw86]

@dadgar we are implementing the chroot feature but at the job spec level. Could you elaborate on what you mean by rootfs? Is this approach we are taking ok for now?

[dadgar]

@devendram Yeah the thought would be the job submitter could provide a tar of the file system them would like and we could extract it and run inside that chroot. We would of course mount the dev and proc filesystem and the other standard directories (alloc, secret, etc).

This could be very helpful for deploying non-static binaries.

[jshaw86]

@dadgar IMO this maybe blurring the lines between what nomad is responsible for and what terraform/packer is responsible for. E.g. We use packer/terraform to get all our execution environments on the respective agent, then we just describe to nomad what should exist in that chroot, nomad assumes packer and terraform has done there job.

By literally passing the filesystem to the nomad we now have to duplicate the work we've done in packer in terms of building our filesystem, and maintain those filesystem packages. I think most companies these days are polyglots so it would get unwieldy maintaining a filesystem to pass for python, node, java, C and so on.

[dadgar]

The way I am thinking about it is that it would be part of the CI/CD workflow and packer would output a tar with the FS and then Nomad would run it.

For static binaries you wouldn't need this. It becomes interesting when deploying things like Rails, python/ruby apps with gems/pip dependencies, dynamic linked apps etc. Shipping those dependencies isn't trivial currently and most people will just end up having to use Docker for those cases.

[jshaw86]

Hey @dadgar sorry I didn't get back on this one.

The way I am thinking about it is that it would be part of the CI/CD workflow and packer would output a tar with the FS and then Nomad would run it.

This seems a bit heavy, I would have to see it in practice to know how well it would work for our use case. I imagine having a whole chroot hosted remotely would have unnecessary IO overhead, local caching I guess could fix this. Even with our paired down chroot our size is still non-trivial.

Shipping those dependencies isn't trivial currently and most people will just end up having to use Docker for those cases.

We have been pretty successful with packaging virtualenv's for python and installing them via debian. This has the downside of the deb has to be installed on every nomad client.

The direction we are heading in is the remote FS(S3, gluster, a CDN) will host the applications, the applications are effectively tar's that are virtualenvs of just the application, nomad goes and pulls that application/tar, pulls in local resources pre installed on the nomad client (resolve.conf etc..) and executes the program.

The downside here is any changes to the local resources or new application enforced dependencies have to be added to the clients (causing a full feel drain and deploy) the upside is your only hosting the application remotely optimizing startup performance and file size. We don't view the downside as that bad in that we should probably be rolling the cluster semi-frequently anyway and it's a reasonable division between ops land and develop land.

[soupdiver]

Is there any update on this? I came across this issue when I was researching a situation I have: I want to specify a different chroot for a single task. I want to add a folder which contains keys that should not be mapped to every task but onlt this one.

[tgross]

Generally speaking we update issues when we're planning on working on them. I see that @Amier3 marked this as "help-wanted" but now that it's being surfaced to me I can't really see us implementing this under the security model that we have for Nomad (when this was originally opened, that was less figured-out).

If you've got an idea of how we might be able to work that problem, we're all ears though.