openzfs-docs
openzfs-docs copied to clipboard
openzfs
Unable to export rpool before first reboot
@rlaager, I have the following issue with the Ubuntu 20.04 Root on ZFS HOWTO:
I've been using the ZFS on Root for Ubuntu for several years since 18.04 LTS. I've even replicated much of it as ansible playbook. I completed step updates based on guide to work with 20.04, then 20.04.1, now I'm testing with 20.04.3 and having a problem.
Following: https://openzfs.github.io/openzfs-docs/Getting%20Started/Ubuntu/Ubuntu%2020.04%20Root%20on%20ZFS.html#overview
I use a VirtualBox environment to practice / refine the steps before attempting on bare metal. I'm testing with three 20GB VDI files which provide for a 3 disk Legacy Boot, ZFS Native Encryption, Raidz1 setup, as per guide mdm is being used for encrypted swap.
VirtualBox boots the Ubuntu Desktop 20.04.3 ISO image and I follow the steps provided in the guide. I have started from scratch several times and keep having the same issue:
I get to Section 5, GRUB Installation, Step 10 ... I'm unable to export the pools. Everything previous to this goes pretty smooth.
My troubleshooting trying to determine why the pool busy:
I'm not in a path used by rpool:
# pwd
/home/ansible
Everything appears to unmounted cleanly:
# lsof /mnt
lsof: WARNING: can't stat() fuse.gvfsd-fuse file system /run/user/999/gvfs
Output information may be incomplete.
# lsof | grep "/mnt"
lsof: WARNING: can't stat() fuse.gvfsd-fuse file system /run/user/999/gvfs
Output information may be incomplete.
# ls -l /mnt
total 0
# mount | grep -v zfs | tac | awk '/\/mnt/ {print $3}'
No swap is used at this point:
# cat /proc/swaps
Filename Type Size Used Priority
# free -mh
total used free shared buff/cache available
Mem: 7.8Gi 2.6Gi 3.2Gi 425Mi 2.0Gi 4.5Gi
Swap: 0B 0B 0B
The pool itself is in good shape:
# zpool list
NAME SIZE ALLOC FREE CKPOINT EXPANDSZ FRAG CAP DEDUP HEALTH ALTROOT
rpool 53G 1.56G 51.4G - - 0% 2% 1.00x ONLINE /mnt
# zpool status -v
pool: rpool
state: ONLINE
scan: none requested
config:
NAME STATE READ WRITE CKSUM
rpool ONLINE 0 0 0
raidz1-0 ONLINE 0 0 0
ata-VBOX_HARDDISK_VB195fbc7a-08e09291-part4 ONLINE 0 0 0
ata-VBOX_HARDDISK_VB8b7fa809-6ae551c5-part4 ONLINE 0 0 0
ata-VBOX_HARDDISK_VB52366d20-43e72e2a-part4 ONLINE 0 0 0
errors: No known data errors
# zfs list
NAME USED AVAIL REFER MOUNTPOINT
rpool 1.04G 33.2G 234K /mnt
rpool/ROOT 1.03G 33.2G 234K none
rpool/ROOT/ubuntu_p029lm 1.03G 33.2G 984M /mnt
rpool/ROOT/ubuntu_p029lm/srv 234K 33.2G 234K /mnt/srv
rpool/ROOT/ubuntu_p029lm/usr 629K 33.2G 234K /mnt/usr
rpool/ROOT/ubuntu_p029lm/usr/local 394K 33.2G 394K /mnt/usr/local
rpool/ROOT/ubuntu_p029lm/var 74.2M 33.2G 234K /mnt/var
rpool/ROOT/ubuntu_p029lm/var/games 234K 33.2G 234K /mnt/var/games
rpool/ROOT/ubuntu_p029lm/var/lib 72.2M 33.2G 1.50M /mnt/var/lib
rpool/ROOT/ubuntu_p029lm/var/lib/AccountsService 234K 33.2G 234K /mnt/var/lib/AccountsService
rpool/ROOT/ubuntu_p029lm/var/lib/NetworkManager 234K 33.2G 234K /mnt/var/lib/NetworkManager
rpool/ROOT/ubuntu_p029lm/var/lib/apt 51.1M 33.2G 51.1M /mnt/var/lib/apt
rpool/ROOT/ubuntu_p029lm/var/lib/dpkg 19.1M 33.2G 19.1M /mnt/var/lib/dpkg
rpool/ROOT/ubuntu_p029lm/var/log 480K 33.2G 480K /mnt/var/log
rpool/ROOT/ubuntu_p029lm/var/mail 234K 33.2G 234K /mnt/var/mail
rpool/ROOT/ubuntu_p029lm/var/snap 234K 33.2G 234K /mnt/var/snap
rpool/ROOT/ubuntu_p029lm/var/spool 330K 33.2G 330K /mnt/var/spool
rpool/ROOT/ubuntu_p029lm/var/www 234K 33.2G 234K /mnt/var/www
rpool/USERDATA 480K 33.2G 234K /mnt
rpool/USERDATA/root_p029lm 245K 33.2G 245K /mnt/root
Unable to export the pool:
# zpool export -a
cannot export 'rpool': pool is busy
# zpool export -f rpool
cannot export 'rpool': pool is busy
If I ignore this issue and attempt to reboot, the rpool will not be imported as its says it was used by another system. My understanding is that is normal if you are unable to export cleanly. Thus I'm trying to solve this.
I have the same problem. Can't export the pool because rpool is busy. Then can't import rpool because pool was previously in use from another system.
I think that "pool is busy" is a bug in openzfs itself because nothing was mounted.
Editing /etc/default/grub and adding zfsforce=yes to GRUB_CMDLINE_LINUX_DEFAULT makes it possible to import the rpool though.
Launch those commands before rebooting it will work 👍
mount | grep -v zfs | tac | awk '/\/mnt/ {print $3}' |xargs -i{} umount -lf {}
umount /mnt/*
umount -l /mnt
grep /mnt /proc/*/mounts | tac| awk '{print $3}' FS="/" | uniq | xargs -i{} kill -9 {}
zpool export -a
reboot -f
@DrGeek
Just took an environment and ran all the steps to try your suggestion. I still got the rpool is busy message, unable to export rpool and upon reboot failed to import rpool and get dropped at the (initramfs) prompt. It's not hard to recover from here, but just can not find what has that rpool busy.
@DrGeek - see if this sparks another idea.
I've implemented the ZFS 20.04 how-to as an Ansible role. The entire process works great minus this rpool busy issue. I've applied your suggestion in Ansible and got some additional output from it.
Note the umount: /mnt/boot: target is busy.
Relevant Ansible Task:
- name: Unmount chroot file systems
shell: |
mount | grep -v zfs | tac | awk '/\/mnt/ {print $3}' | xargs -i{} umount -lf {}
umount /mnt/*
umount -l /mnt
grep /mnt /proc*/mounts | tac | awk '{print $3}' FS="/" | uniq | xargs -i{} kill -9 {}
- name: Export all ZFS Pools
command:
"zpool export -a"
tags:
- unmount_chroot
The process fails on the zpool export 100% of the time as follows:
changed: [testlinux.rich-durso.us] => {
"changed": true,
"cmd": "mount | grep -v zfs | tac | awk '/\\/mnt/ {print $3}' | xargs -i{} umount -lf {}\numount /mnt/*\numount -l /mnt\ngrep /mnt /proc*/mounts | tac | awk '{print $3}' FS=\"/\" | uniq | xargs -i{} kill -9 {}\n",
"delta": "0:00:00.256326",
"end": "2022-03-23 17:54:37.031167",
"rc": 0,
"start": "2022-03-23 17:54:36.774841"
}
STDERR:
umount: /mnt/bin: not mounted.
umount: /mnt/boot: target is busy.
umount: /mnt/dev: not mounted.
umount: /mnt/etc: not mounted.
umount: /mnt/home: not mounted.
umount: /mnt/lib: not mounted.
umount: /mnt/lib32: not mounted.
umount: /mnt/lib64: not mounted.
umount: /mnt/libx32: not mounted.
umount: /mnt/media: not mounted.
umount: /mnt/mnt: not mounted.
umount: /mnt/opt: not mounted.
umount: /mnt/proc: not mounted.
umount: /mnt/run: not mounted.
umount: /mnt/sbin: not mounted.
umount: /mnt/sys: not mounted.
umount: /mnt/tmp: not mounted.
umount: /mnt/usr: not mounted.
umount: /mnt/var: not mounted.
TASK [zfs_on_root : Export all ZFS Pools] *****************************************************************************************************************************************************************************************************************************
task path: /home/rich/Sync/rich/ansible/roles/zfs_on_root/tasks/15-5_unmount_chroot.yml:16
fatal: [testlinux]: FAILED! => {
"changed": true,
"cmd": [
"zpool",
"export",
"-a"
],
"delta": "0:00:05.176945",
"end": "2022-03-23 17:54:42.413881",
"rc": 1,
"start": "2022-03-23 17:54:37.236936"
}
STDERR:
cannot export 'rpool': pool is busy
MSG:
non-zero return code
If I re-run just this task, buy its tag still fails, much less output:
changed: [testlinux.rich-durso.us] => {
"changed": true,
"cmd": "mount | grep -v zfs | tac | awk '/\\/mnt/ {print $3}' | xargs -i{} umount -lf {}\numount /mnt/*\numount -l /mnt\ngrep /mnt /proc*/mounts | tac | awk '{print $3}' FS=\"/\" | uniq | xargs -i{} kill -9 {}\n",
"delta": "0:00:00.005469",
"end": "2022-03-23 18:06:59.411353",
"rc": 0,
"start": "2022-03-23 18:06:59.405884"
}
STDERR:
umount: /mnt/*: no mount point specified.
umount: /mnt: not mounted.
TASK [zfs_on_root : Export all ZFS Pools] *****************************************************************************************************************************************************************************************************************************
task path: /home/rich/Sync/rich/ansible/roles/zfs_on_root/tasks/15-5_unmount_chroot.yml:16
fatal: [testlinux: FAILED! => {
"changed": true,
"cmd": [
"zpool",
"export",
"-a"
],
"delta": "0:00:00.045487",
"end": "2022-03-23 18:06:59.637535",
"rc": 1,
"start": "2022-03-23 18:06:59.592048"
}
STDERR:
cannot export 'rpool': pool is busy
MSG:
non-zero return code
@reefland I just ran into a similar issue with the NixOS Root on ZFS installation guide. In the last steps before exporting the zpools it says to run:
umount /mnt/boot/efis/*
umount /mnt/boot/efi
The first line runs fine but the second fails, but not because of the pool being busy but because nothing was mounted there. The problem is that all the pools were still mounted. I am running all this from a script and was not even getting the cannot export 'rpool': pool is busy messages and wasn't aware anything was wrong until I tried booting and the boot pool failed to be imported similar to what you experienced. I tried many things but the changing the previous umount commands to the following fixed it for me.
umount /mnt/boot/efis/*
umount -R /mnt
I was able to export my zpools and reboot into the installed OS.
