ZFS zpool name question

[jemail99]

I have a freebsd 12.1 zfs on root server with the main pool named zroot.

When setting bastille_zfs_zpool="" would I use bastille_zfs_zpool="zroot" or would I create another dataset eg. /jails and then use bastille_zfs_zpool="zroot/jails"?

[MANDATORY] Bastille and FreeBSD version (paste bastille -v && freebsd-version -kru output) N/A 12.1-RELEASE-p10 12.1-RELEASE-p10 12.1-RELEASE-p10

[MANDATORY] How did you install bastille? (port/pkg/git) N/A

[optional] Steps to reproduce? N/A

[optional] Expected behavior not sure as there is no wiki example

[cedwards]

From the configuration:

## ZFS options
bastille_zfs_enable=""                                  ## default: ""
bastille_zfs_zpool=""                                   ## default: ""
bastille_zfs_prefix="bastille"                          ## default: "${bastille_zfs_zpool}/bastille"
bastille_zfs_mountpoint=${bastille_prefix}              ## default: "${bastille_prefix}"
bastille_zfs_options="-o compress=lz4 -o atime=off"     ## default: "-o compress=lz4 -o atime=off"

If you set bastille_zfs_enable=YES and bastille_zfs_zpool=zroot it will create zroot/bastille for your containers. It will create the datasets automatically when you first bootstrap a release.

Those two values (enable and zpool) are the only two required to be set for use. The others are optional customization.

[jemail99]

Thank you for your prompt reply. Could the documentation/wiki be updated to reflect your answer and give an example set of ZFS options for clarity? Is that something I could do?

[cedwards]

I would be happy to review a document that could be put into the /docs directory. This content is rendered at docs.bastillebsd.org and should (ideally) be the official documentation.

There are a couple references to ZFS and its configuration in the README.md. If you decide to type something up please review and include relevant content.

[jemail99]

There is nothing in docs.bastillebsd.org about zfs as far as I can see. https://bastillebsd.org/getting-started/ contains a section but should probably be updated to reflect README.md. I am unsure how to send you the doc with updates but I have left my thoughts on the necessary edit below.

I suggest that the README.md ZFS section be updated as follows:

To enable ZFS functionality two values are required by Bastille and must be configured in /usr/local/etc/bastille/bastille.conf, other values are optional. The first two default values in the /usr/local/etc/bastille/bastille.conf are empty by default. Populate these two values to enable ZFS functionality.

Default values in /usr/local/etc/bastille/bastille.conf

## ZFS options
bastille_zfs_enable=""                               ## default: ""
bastille_zfs_zpool=""                                ## default: ""
bastille_zfs_prefix="bastille"                       ## default: "${bastille_zfs_zpool}/bastille"
bastille_zfs_mountpoint=${bastille_prefix}           ## default: "${bastille_prefix}"
bastille_zfs_options="-o compress=lz4 -o atime=off"  ## default: "-o compress=lz4 -o atime=off"

You can change these two values by editing /usr/local/etc/bastille/bastille.conf using the following command # ee /usr/local/etc/bastille/bastille.conf

or add them directly to /usr/local/etc/bastille/bastille.conf from the command line using

# sysrc -f /usr/local/etc/bastille/bastille.conf bastille_zfs_enable=YES
# sysrc -f /usr/local/etc/bastille/bastille.conf bastille_zfs_zpool=ZPOOL_NAME

Replace ZPOOL_NAME with the zpool you want Bastille to use. Tip: zpool list and zpool status will help. If you get 'no pools available' you are likely not using ZFS and can safely ignore these settings.

An example /usr/local/etc/bastille/bastille.conf for a FreeBSD 12.1 release could be as follows:

## ZFS options
bastille_zfs_enable="YES"                            ## default: ""
bastille_zfs_zpool="zroot"                           ## default: ""
bastille_zfs_prefix="jails"                          ## default: "${bastille_zfs_zpool}/bastille"
bastille_zfs_mountpoint=${bastille_prefix}           ## default: "${bastille_prefix}"
bastille_zfs_options="-o compress=lz4 -o atime=off"  ## default: "-o compress=lz4 -o atime=off"

If you set bastille_zfs_enable=YES and bastille_zfs_zpool=zroot bastille will create zroot/bastille for your containers. It will create the datasets automatically when you first bootstrap a release.

[jemail99]

As an aside, where is bastille_prefix defined? Can bastille_prefix be user defined?

[chriswells0]

bastille_prefix is also in /usr/local/etc/bastille/bastille.conf.

[kjelderg]

I actually found the documentation for this quite confusing. I had an existing pool, m. I have settings like this:

ZFS options

bastille_zfs_enable="YES" ## default: "" bastille_zfs_zpool="m" ## default: "" bastille_zfs_prefix="bastille" ## default: "${bastille_zfs_zpool}/bastille"

So now try to bootstrap 13.0-RELEASE

freebsd@generic:/usr/local/bastille % sudo bastille bootstrap 13.0-RELEASE
Bootstrapping FreeBSD distfiles...
cannot create 'm/bastille/cache/13.0-RELEASE': parent does not exist
cannot create 'm/bastille/releases/13.0-RELEASE': parent does not exist
fetch: /usr/local/bastille/cache/13.0-RELEASE/MANIFEST: open(): No such file or directory
ls: /usr/local/bastille/cache/13.0-RELEASE: No such file or directory
cannot open 'm/bastille/cache/13.0-RELEASE': dataset does not exist
ls: /usr/local/bastille/releases/13.0-RELEASE: No such file or directory
cannot open 'm/bastille/releases/13.0-RELEASE': dataset does not exist
Bootstrap failed.

Well that's not quite right. Maybe I need to create a zfs for it.

freebsd@generic:/usr/local/bastille % sudo zfs create m/bastille
freebsd@generic:/usr/local/bastille % sudo bastille bootstrap 13.0-RELEASE
Bootstrapping FreeBSD distfiles...
cannot create 'm/bastille/cache/13.0-RELEASE': parent does not exist
cannot create 'm/bastille/releases/13.0-RELEASE': parent does not exist
fetch: /usr/local/bastille/cache/13.0-RELEASE/MANIFEST: open(): No such file or directory
ls: /usr/local/bastille/cache/13.0-RELEASE: No such file or directory
cannot open 'm/bastille/cache/13.0-RELEASE': dataset does not exist
ls: /usr/local/bastille/releases/13.0-RELEASE: No such file or directory
cannot open 'm/bastille/releases/13.0-RELEASE': dataset does not exist
Bootstrap failed.

Nope. Directories?

freebsd@generic:/usr/local/bastille % sudo mkdir /m/bastille/cache
freebsd@generic:/usr/local/bastille % sudo mkdir /m/bastille/releases
freebsd@generic:/usr/local/bastille % sudo bastille bootstrap 13.0-RELEASE
Bootstrapping FreeBSD distfiles...
cannot create 'm/bastille/cache/13.0-RELEASE': parent does not exist
cannot create 'm/bastille/releases/13.0-RELEASE': parent does not exist
fetch: /usr/local/bastille/cache/13.0-RELEASE/MANIFEST: open(): No such file or directory
ls: /usr/local/bastille/cache/13.0-RELEASE: No such file or directory
cannot open 'm/bastille/cache/13.0-RELEASE': dataset does not exist
ls: /usr/local/bastille/releases/13.0-RELEASE: No such file or directory
cannot open 'm/bastille/releases/13.0-RELEASE': dataset does not exist
Bootstrap failed.

Not that either. In fact...

freebsd@generic:/usr/local/bastille % sudo zfs create m/bastille/cache freebsd@generic:/usr/local/bastille % sudo zfs create m/bastille/releases
freebsd@generic:/usr/local/bastille % sudo zfs create m/bastille/jails freebsd@generic:/usr/local/bastille % sudo bastille bootstrap 13.0-RELEASE
Bootstrapping FreeBSD distfiles...
/usr/local/bastille/cache/13.0-RELEASE/base.tx 157 MB 2748 kBps 58s Validated checksum for 13.0-RELEASE: base.txz MANIFEST: b0f1a4b2530924d91eb8c46a88a6c5d12ff6a79d990d2fe98e0b7370c1b0f13f DOWNLOAD: b0f1a4b2530924d91eb8c46a88a6c5d12ff6a79d990d2fe98e0b7370c1b0f13f Extracting FreeBSD 13.0-RELEASE base.txz.

Bootstrap successful.
See 'bastille --help' for available commands.

[cedwards]

I notice that even in the first example /usr/local/bastille already exists. This should not exist before any bootstrap has been done. I may have run into this trying to migrate to ZFS in the past. That path is used in bootstrap testing to determine how much needs to be created.

I recommend deleting /usr/local/bastille completely and then the base config values you have look fine to me.

bastille_zfs_enable="YES" ## default: ""
bastille_zfs_zpool="m" ## default: ""
bastille_zfs_prefix="bastille" ## default: "${bastille_zfs_zpool}/bastille"

In a clean installation I would expect this to create the proper paths at /m/bastille/... at bootstrap.

[kjelderg]

You caught me. I was using local filesystems and migrating to using zfs. Thanks for the pro tip.

[kjelderg]

A quick note on remediation in case anyone stumbles upon this from a search:

chflags -R noschg /usr/local/bastille # a seeded bootstrap has noschg flags set on some files

rm -rf /usr/local/bastille

Now we should be on our merry way.

freebsd@generic:~ % sudo zfs destroy -r m/bastille freebsd@generic:~ % sudo bastille bootstrap 13.0-RELEASE Insecure permissions on /usr/local/bastille Try: chmod 0750 /usr/local/bastille freebsd@generic:~ % ls -alh /usr/local/ total 48 drwxr-xr-x 12 root wheel 512B Apr 11 06:56 . drwxr-xr-x 14 root wheel 512B Apr 11 04:12 .. drwxr-xr-x 4 root wheel 512B Apr 11 06:56 bastille

One more fix-up. Bad permissions on autocreates.

freebsd@generic:~ % sudo chmod 750 /usr/local/bastille/ freebsd@generic:~ % sudo bastille bootstrap 13.0-RELEASE Bootstrapping FreeBSD distfiles... cannot create 'm/bastille/backups': parent does not exist chmod: /usr/local/bastille/backups: No such file or directory cannot create 'm/bastille/jails': parent does not exist cannot create 'm/bastille/templates': parent does not exist ln: /usr/local/bastille/templates/default: No such file or directory /usr/local/bastille/cache/13.0-RELEASE/MANIFES 782 B 2829 kBps 00s /usr/local/bastille/cache/13.0-RELEASE/base.tx 157 MB 2570 kBps 01m03s Validated checksum for 13.0-RELEASE: base.txz MANIFEST: b0f1a4b2530924d91eb8c46a88a6c5d12ff6a79d990d2fe98e0b7370c1b0f13f DOWNLOAD: b0f1a4b2530924d91eb8c46a88a6c5d12ff6a79d990d2fe98e0b7370c1b0f13f Extracting FreeBSD 13.0-RELEASE base.txz.

Bootstrap successful. See 'bastille --help' for available commands.

Oh boy, that's not great. Now everything is in /usr/local/bastille and nothing is in the zfs indicated in the settings. Let's clear it out again (as above, chflags and rm -rf).

freebsd@generic:~ % sudo zfs create m/bastille freebsd@generic:~ % sudo bastille bootstrap 13.0-RELEASE
Bootstrapping FreeBSD distfiles... cannot create 'm/bastille': dataset already exists chmod: /usr/local/bastille: No such file or directory /usr/local/bastille/cache/13.0-RELEASE/MANIFES 782 B 2350 kBps 00s /usr/local/bastille/cache/13.0-RELEASE/base.tx 48% of 157 MB 2732 kBps 30s^C

Okay, now we are really close, but that zpool/zfs_prefix needs to be mounted at the bastille_prefix to get rid of some warnings on a bootstrap. Don't forget to do the mermissions changes.

freebsd@generic:~ % sudo zfs set mountpoint=/usr/local/bastille m/bastille freebsd@generic:~ % sudo chmod 0750 /usr/local/bastille/ freebsd@generic:~ % sudo bastille bootstrap 13.0-RELEASE
Bootstrapping FreeBSD distfiles... /usr/local/bastille/cache/13.0-RELEASE/MANIFES 782 B 2286 kBps 00s /usr/local/bastille/cache/13.0-RELEASE/base.tx 157 MB 3848 kBps 42s Validated checksum for 13.0-RELEASE: base.txz MANIFEST: b0f1a4b2530924d91eb8c46a88a6c5d12ff6a79d990d2fe98e0b7370c1b0f13f DOWNLOAD: b0f1a4b2530924d91eb8c46a88a6c5d12ff6a79d990d2fe98e0b7370c1b0f13f Extracting FreeBSD 13.0-RELEASE base.txz.

Bootstrap successful. See 'bastille --help' for available commands.

...easy as pi.

[cedwards]

Sorry you had trouble migrating from UFS to ZFS.

I'll add a task to create some documentation around this process.

[JRGTH]

Migrating from UFS to ZFS, ok listen:

This is a general recommend:

1: Export jail(s), they will be .TXZ standard format. 2: SAVE the EXPORTED jails and they CHECKSUM files to a SAFE location. 3: REDO the same or at least similar environment of the bastille setup in ZFS. 4: IMPORT the jails to the ZFS setup, the code will try to adapt the paths change etc. BUT always check the config files for manual adjust. 5: TADA.

Regards

[bmac2]

question answered. closing issue.