[ENHANCEMENT] support zfs jailed datasets

[dch]

Is your feature request related to a problem? Please describe.

Map (jail & mount) a generic zfs dataset into a bastille jail

I would like a largely disposable jail, where the data lives in a jailed zfs dataset, that is separate to the bastille jail tree. This enables rebuilding jails as needed, for fast updates, and only dropping in a small number of config files into the jail, leaving the large dataset outside the bastille tree, which simplifies backups and sysadmin work.

Given a single bastille jail (possibly built from a template), I would like to have:

• a zfs dataset (& possibly child datasets)
• that lives outside the bastille jail tree
• that is auto-mounted into the jail on startup, including any of its child datasets
• and un-mounted (unjailed) on shutdown
• and can be attached to a new bastille jail when it is (re)built
• optionally allows the jail user to issue appropriate delegated zfs commands

this enables using jails ephemerally, with the following structure:

• zroot/jailed/:jail/var/www/nginx -> inside the jail /var/www/nginx
• zroot/bastille/jails/:jail/root -> the main jail root filesystem

Describe the solution you'd like

user must prepare the following dataset:

# zfs create -o jailed=on -o canmount=off -o mountpoint=/var/www zroot/jailed/alcatraz
# zfs create zroot/jailed/alcatraz/nginx
# zfs list -o canmount,mounted,jailed,mountpoint,name,usedbydataset -r embiggen/jailed
CANMOUNT  MOUNTED  JAILED  MOUNTPOINT                       NAME                                                      USEDDS
off       no       off     none                             embiggen/jailed                                              96K
off       no       on      /var/www                         embiggen/jailed/alcatraz                                     96K
on        no       on      /var/www/nginx                   embiggen/jailed/alcatraz/nginx                               96K

when jail is created, bastille needs to:

• pull in a zfs_dataset or similar property from config, naming the root dataset zroot/jailed/alcatraz from above
• set bothallow.mount.zfs=1 enforce_statfs=1
• add zfs_enable=YES to jail's /etc/rc.conf or similar
• append something (perhaps to /usr/local/jail.conf ) to do the mounting, if the above zfs_enable won't do it already

This example, taken from iocell, shows the outcome:

root@f01 /u/h/dch# iocell list
JID  UUID                                  BOOT  STATE  TAG        TYPE       IP4          RELEASE
1    90a81ad1-64b9-11eb-b7d0-0cc47a16edea  on    up     eden       clonejail  100.64.0.10  12.2-RELEASE
2    92049a12-64b9-11eb-b7d0-0cc47a16edea  on    up     couchdb    clonejail  100.64.0.3   12.2-RELEASE
3    9367693e-64b9-11eb-b7d0-0cc47a16edea  on    up     mu         clonejail  100.64.0.8   12.2-RELEASE
4    94da7aa1-64b9-11eb-b7d0-0cc47a16edea  on    up     www        clonejail  100.64.0.6   12.2-RELEASE
5    9650ef84-64b9-11eb-b7d0-0cc47a16edea  on    up     seaweedfs  clonejail  100.64.0.2   12.2-RELEASE
6    97df0758-64b9-11eb-b7d0-0cc47a16edea  on    up     serenity   clonejail  100.64.0.9   12.2-RELEASE

root@f01 /u/h/dch# zfs list -r zroot/jailed
NAME                             USED  AVAIL  REFER  MOUNTPOINT
zroot/jailed                    6.74G   105G    88K  none
zroot/jailed/couchdb3            736M   105G   155M  /var/db/couchdb3
zroot/jailed/couchdb3/views     79.3M   105G  12.2M  /var/db/couchdb3/views
zroot/jailed/fdb                  88K   105G    88K  /var/db/fdb
zroot/jailed/mu                   88K   105G    88K  /var/mu
zroot/jailed/seaweedfs           744K   105G   264K  /var/db/seaweedfs
zroot/jailed/seaweedfs/dataset    96K   105G    96K  /var/db/seaweedfs/dataset
zroot/jailed/www                6.02G   105G  5.44G  /var/www
...

Describe alternatives you've considered

nullfs mounts. easy, but we want 2 things:

• leverage zfs so that the jail can do its own snapshots etc
• hide the dataset from the jail host, zfs jail .. does this for us

Additional context

Happy to get involved implementing this with some indication of where the various options/code would be best added.

[dch]

any interest in this?

[cedwards]

This looks interesting. Happy to discuss more; thank you for offering to implement.

Due to the changes required for rc.conf and jail.conf I'd recommend building into create.sh, possible triggered by a -Z option (for ZFS jailed). We have similar options in create.sh for -V (VNET) or -T ("thick"), etc.

I don't have time to implement myself but will gladly review a pull request if you come up with something.

[markjdb]

I have a use for this feature as well. It's mostly possible to get this working with config commands, at least when it's a single dataset (haven't thought much about the case of child datasets).

So far I did this:

• unhide /dev/zfs in the jail's devfs ruleset
• set allow.mount.zfs and enforce_statfs=1
• set exec.created such that it invokes zfs jail <name> <dataset>
  • this is annoying to set via bastille config since I can't use jail.conf variables without the shell expanding them
• add an fstab entry to mount the volume automatically when the jail is started
  • I can't use bastille mount for this since it only permits nullfs mounts for some reason.

This is similar to the steps listed above. Setting zfs_enable might be better than editing the jail fstab, I'm not sure.

Invoking zfs jail seems like it should become a jail(8) feature, but I guess we don't want to depend on new base system features.

Regarding the implementation, adding -Z <dataset> to the create command could be a way to go, but I wonder if it'd be better to let it be applied to an existing jail. (Also so you can "detach" a dataset without destroying the jail.) bastille config is strictly for setting jail parameters, but maybe it'd be good to support setting higher-level, Bastille-specific configuration parameters in some way.

[g-teley]

This is exactly what I was looking for

[patmaddox]

fwiw for anyone interested in delegating ZFS datasets to bastille jails, here are my additions to the bastille-generated jail.conf. This is for poudriere so there may be some extra stuff, but the basic line is exec.created += 'zfs jail pkg zroot/jdata/pkg'; and then tweaking the other params until it works.

pkg {
  # additions / changes
  allow.mount;
  allow.mount.devfs;
  allow.mount.zfs;
  allow.mount.nullfs;
  allow.mount.tmpfs;
  allow.mount.procfs;
  children.max = 10;
  devfs_ruleset = 4;
  enforce_statfs = 1;
  securelevel = 0; # defaults to 2, can use 2 with USE_TMPFS in poudriere
  exec.created += 'zfs jail pkg zroot/jdata/pkg';
  
  # bastille defaults
  exec.clean;
  exec.consolelog = /var/log/bastille/pkg_console.log;
  exec.start = '/bin/sh /etc/rc';
  exec.stop = '/bin/sh /etc/rc.shutdown';
  host.hostname = pkg;
  mount.devfs;
  mount.fstab = /usr/local/bastille/jails/pkg/fstab;
  path = /usr/local/bastille/jails/pkg/root;

  interface = bastille0;
  ip4.addr = 172.16.1.4;
  ip6 = disable;
}

[bdrewery]

Should also have this but it doesn't seem to matter much.

exec.poststop += 'zfs unjail pkg zroot/jdata/pkg';

[cedwards]

I've added this to the roadmap for an upcoming release. Thank you for your patience. 😄