Mail.app halted on 'Unable to quarantine' when UserDirectory stored on ZFS (MacOS 14+)

[captain-haddock17]

Hi all ! Since my recent (december) upgrade from MacOS 10.13.x to 10.14.6, all my MacOS user accounts stored on ZFS can NOT have their e-mails (through Mail.app).

Short story : at first initialization of a fresh new User Account (with same @icloud.com account though):

1. Mail.app is OK, and e-mails are all downloaded, visible , etc ...
2. if you close , and reopen Mail (juste after, or after a logout/login, or a Reboot) : Mail launches (visible in the menu bar on top), but NO (any) window is visible, or can't be brought visible.
3. Console message says : kernelMail[3138] Unable to quarantine: 1

Side Note :

• All the iCloud apps are OK (Notes, Agenda, Contacts)
• depending on some ZFS params (xattr on/sa ?) , KeyChain can be confused and has to be reinitialized ...

Long Story

0. MacPro 5,1 (2010) with SSD (APFS - macOS 10.14.6 (18G4032)/ Darwin 18.7.0) and 3 internal SATA drives. sysctl {spl,zfs}.kext_version

spl.kext_version: 1.9.4-0 zfs.kext_version: 1.9.4-0

1. Most User accounts are stored on the internal 3xSATA Pool with ZFS FileSystem.

• Since MacOS 10.14 ;
  • Softlink can't be used anymore, typical. ls -s /Volumes/zDisk/Home/myser /Users/myuser
  • param HomeDirectory has to be used declared (one of:
    • through SysTPref->User&Groups
    • through dscl localhost -read /Local/Default/Users/myuser NFSHomeDirectory (then use -change)

2. OSX xattr on Mail folders seems to be OK.

• Classical User myadmin account on Apple HPFS

$ ls -lde@ Library/Mail

drwx------@ 5 myadmin staff 160 17 mar 09:12 Library/Mail com.apple.quarantine 19

$ xattr -lv Library/Mail

Library/Mail: com.apple.quarantine: 0082;00000000;Mail;

• User account on ZFS filesystem on zPool mounted on /Volumes/zDisk/Home

$ sudo ls -lde@ myuser/Library/Mail

drwx------@ 4 myuser admin 4 6 avr 14:11 myuser/Library/Mail com.apple.quarantine 19

$ sudo xattr -lv myuser/Library/Mail

myuser/Library/Mail: com.apple.quarantine: 0082;00000000;Mail;

Tests

Fidling with some of the ZFS parameters didn't solve the problem

• com.apple.devdisk on : because with poolonly , ZFS can NOT mount the given useraccount FileSystem, MacOS is testing/preventing the mount of an external HomeDirectory ?!?! see issue #754
• xattr = on (in first place) / sa (no better ?)

Common params

• aclmode passthrough
• aclinherit restricted
• com.apple.mimic_hfs on
• com.apple.browse on
• com.apple.ignoreowner off
• utf8only on
• normalization formD
• casesensitivity insensitive

_Other Fidlings _

• tried to put ../Library on a jHFS formatted ZVOL with a softlink ln -s /Volumes/myuser.Library /Volumes/zDisk/Home/myuser/Library
• declared Mail.app having Full Disk Access in "Security & Privacy" Prefs (no better 👎 )

Any clue ? Thanks !

Kind regards, William

[JMoVS]

One short question: Is it HFS+ or APFS? To the best of my knowledge Apple doesn't have HPFS and the technical background and intricacies are quite a bit different between APFS and HFS+. Since this is 10.14 I would guess APFS but could you clarify?

[JMoVS]

When you say it wasn't able to mount, did you make sure that:

• the mountpoint was defined and empty? (No hidden files either? What is the error message of ZFS attempting to mount?)
• put the zfs and the zpool binary in the Full disk access list?

[captain-haddock17]

Hello JMoVS,

Thanks for the follow-up :-)

• Typo APFS corrected in description
• MountPoint : (with com.apple.devdisk=poolonly
  • yes, directory was empty. I did several tests, with 2 different Users and ZFS dataset.
  • I could do a zfs mountafterwards.
  • On bootime, these 2 ZFS dataset didn't mount. I figured it out with com.apple.devdisk=on and this solved the issue (maybe a side effect ?).
  • I did not tried to declare an entry in fstab, hoping to force the sequence of disk mounts.

see issue #754.

Sorry, impossible to put spl.kextand zfs.kext in Full disk access. (greyed at the selection through gateKeeper)

[captain-haddock17]

(sorry bad manoeuvre has closed this issue !!)

[JMoVS]

@captain-haddock17 Try to put the zfs and zpool binaries to full access as well as terminal.app. You'll find the binaries if you run which zfs and use that path to figure out where they are

[JMoVS]

also you might run into a timing issue with your setup where mac OS runs the zpool import all launchd job after it already tried to load the user - resulting in you running into problems. We'd have to ask I think @cbreak-black or maybe @rottegift on how they're handling user home dirs on ZFS.

[JMoVS]

also what is the current mountpoint? Mojave has very strict rules on where you can mount stuff and where you're not allowed to mount stuff...

[captain-haddock17]

• Terminal.app was already in Full disk access

• some more testing with cmpxat v2 from The Eclectic Light Company on some real Mail.app file 7.emlx

  • xattr -lv 7.emlx

7.emlx: com.apple.quarantine: 0082;5e21acf6;Mail;

(file renamed to from-APFS for the sake)

• cp -a from-APFS --> ZFS-xattr=on
• cp -a from-APFS --> ZFS-xattr=sa
• cmpxat from-APFS ZFS-xattr=on :: result OK
• cmpxat from-APFS ZFS-xattr=sa :: result OK

[JMoVS]

I have mail.app's data (~/Library/Mail) in a dataset which works fine. My settings are devdisk=on, mimic_hfs=on and xattr=on

I don't have experience with having the container stuff on ZFS though unfortunetly

[JMoVS]

you could try if everything else fails to create a dataset for mail, mount that to that location with the settinhs I mentioned (and all other settings being just inherited as is) and see if that works

[captain-haddock17]

Some clue :

While I was trying to do some more test cases, with copy of /Users/myUser to a ZFS zvol with APFS format, say sudo cp -a /Users/rover-yachts.5 /Volumes/zmacOS/Users

I got some strange error messages, namely:

cp: /Users/rover-yachts.5/Library/Containers/com.apple.mail/Data/DataVaults: Operation not permitted

no ls -lAnor matter -lvcould show this file /Library/Containers/com.apple.mail/Data/DataVaults

PS : needless to mention : Terminal.app has Full Disk Access on. :-)

Quote from The Eclectic Light Company page No Entry: access controls in Mojave:

Introduced in later releases of High Sierra, and used more in Mojave, are folders to which only Apple’s software has even read access, DataVaults. My account here is largely based on comments generously provided here by an anonymous source, as these don’t appear to have been mentioned anywhere by Apple (not even at WWDC 2018), nor can I find other descriptions.

DataVaults are folders to which neither the user nor third-party software has any access at all. The only software which can see and work with their contents are certain Apple-signed products which have a specific entitlement to do so. At present, all Macs running Mojave have at least one, which contains the QuickLook cache, at /var/folders/t9/[long ID]/C/com.apple.QuickLook.thumbnailcache Depending on what other features and apps you use, you may also find them at three additional locations:

• ~/Library/VoiceTrigger/SAT
• ~/Library/Containers/com.apple.mail/Data/DataVaults
• /var/folders/0z/[long ID]/0/com.apple.nsurlsessiond

I found some interesting reading about Mojave's new behavior:

• Access to ~/Library/Containers/com.apple.mail/Data/ denied
• Apple DocRef about Entitlements

[captain-haddock17]

Should ZFS need a special zfs function in order to handle the very special file flag UF_DATAVAULT put on those directories ?

Ref : The Eclectic Light Company , last § in No Entry: access controls in Mojave

[captain-haddock17]

More info about Mail.app

$ codesign -d --entitlements - /Applications/Mail.app/

(...)
	<key>com.apple.rootless.storage.Mail</key>	<true/>
	<key>com.apple.security.app-sandbox</key>	<true/>
	<key>com.apple.security.cs.disable-library-validation</key>	<true/>
	<key>com.apple.private.security.storage.Mail</key>	<true/>

[captain-haddock17]

I found a workaround, as suggested by JMoVS, and my personal test cases;

Put ~/Library/Mail and ~/Library/Containers on a ZFS zvol and making a softlink from the regular ZFS Filesystem to this jHFS (or APFS) formated zvol.

It's really a pain, because

• it's not very practical to create as many zvoldisks that will be seen on your desktop as you have as accounts (5 for me).
• and putting these sub-librairies on one zvoldisks will de-normalize somewhat my backups and archives per User.

Positive : At least, I can go ahead and keep ZFS as main storage solution 👍

[beren12]

if you disable SIP do you still get the error? Just wondering

[JMoVS]

@captain-haddock17 You don't need to put ~/Library/Mail on APFS I think. I have SIP only partially enabled but the core problem is AFAIK in the Container. But the Emails are stored in ~/Library/Mail, so maybe try to leave ~/Library/Mail in ZFS and just put the Container on an APFS zvol ;-)

[JMoVS]

also you could try to not solve this via a softlink but instead do it via specifiying the mountpoint, making the system a bit cleaner

[captain-haddock17]

Some more investigations:

Launching Mail.app on some empty directory (he will recreate subdirs) Got error pannel at launch

Mail cannot open ... Mail is saving data on /Macintosh HD/Users/myuser/Librray/Mail Note: always same directory name is labeled...

In System log

Error kernel Sandbox: Mail(41788) deny(1) file-write* /Volumes/zDisk/Users/myuser/Library/Mail

Disk info/params Owners =Enabled or Disabled is a property shown on disk either:

• diskutil info disk#.#
• through "CMD+I" (with cadnass off) on the Drive shown in Finder or Desktop

Test Conditions with SIP disabled: (huh !?! maximum chances to get some positive results...)

ZFS partition /Volumes/zDisk/Users/myuser

• xattr = on / sa (sees that it doesn(t care)

• Mac User HomeDirectory set to /Volumes/zDisk/Users/myuser

• ~/Library/Keychains softlink on main startup disk /Users/myuser

• ~/Library/Caches softlink on main startup disk /Users/myuser

• ~/Library/Containers softlink on main startup disk /Users/myuser

• ~/Library/Group\ Containers softlink on main startup disk /Users/myuser

• ~/Library/Application\ Support softlink on main startup disk /Users/myuser

• varyinging Soft links on Library/Mail (with proper ownership chown -h) as i.e ln -s /Volumes/Macintosh\ HD/Test/Mail /Volumes/zDisk/Users/myuser/Libraray/Mail

Test cases

• on internal (synthesized) APFS (Owners=Enabled): OK anywhere on disk
• on internal, virtual ZFS : ERROR
• on external, virtual ZVOL APFS formated (Owners=__/Disabled): ERROR
• on external, virtual ZVOL JHFS+ formated (Owners=__/Disabled): ERROR
• on external, physical USB disk JHFS+ formated (Owners=__/Disabled): ERROR
• on external, physical USB disk APFS formated (Owners=Enabled or Disabled): ERROR
• on created/owned Image Disk JHFS+ formated , *.dmg stored in /Users

Did some more test on weird /Macintosh HD/ as first level of root filesystem ... actually it should be / I tryied a softlink ln -s /Volumes/zDisk /Macintosh\ HD Still same issue "mail cannot open ..."

First conclusion (?)

Maybe the first clue is that we must have (at least)

• Internal + Owners=Enabled

because on regular external disks, formated by regular plain MacOS APFS ou JHFS+ doesn't work.

[captain-haddock17]

Could be related to Sandbox (Jail) forcing special constraints on disk ?

• startup disk ?
• Internal
• Physical ?
• Owners=Enabled

Could ZFS on OSX adapt ZFS Filesystems so as to show mounted ZFS as these kind of internal disks ?

[captain-haddock17]

Other clue: With Macintosh\ HD as startup-disk Mail.app starting on empty folder ~/Library/Mail

Pointing to the regular /Users folder

• ln -s /Users/mysuser/Library/Mail /Volumes/zDisk/Users/myuser/Library/Mail is OK

Pointing to equivalent same /Users folder

• ln -s /Volumes/Macintosh\ HD/Users/mysuser/Library/Mail /Volumes/zDisk/Users/myuser/Library/Mail has same ERROR as above.

??? WTF ?

updated:

Pointing from the regular /Users/myuser/Library/Mail folder to else-where is OK as long as it is on a JHFS+ or APFS formatted filesystem (ZVOL is OK)

• mv -v /Users/mysuser/Library/Mail /Users/mysuser/Library/Mail-local
• ln -sv /Volumes/zLibraries/myuser/Library/Mail /Users/mysuser/Library/Mail

If .../Mail directory is located on a regular ZFS formated filesystem, Mail.app is emitting Unable to quarantine errors in the MacOS Console log.

That's where I'm at.

[JMoVS]

Have you tried to run chown -R $User on the Library/Mail Dataset?

why are you using softlinks and not simply mountpoints? I have no problems with a Dataset that is mounted to ~/Library/Mail that contains my mail data. Mountpoints are more solid than links. I don't expect softlinks to work well.

chown -R $User is what I needed for my permissions of the datasets and then I just mount them to the respective place, eg ~/Library/Mail

[captain-haddock17]

Hi, i have checked if my ownerships where ok, but anyway I will run chown to be sure.

Thanks for mentionning that your config is ok, this gives me some more energy to go futher in my investigations :-)

[JMoVS]

Just note that I don't have my home directory on ZFS. You can jump in on IRC (freenode #openzfs-osx ) or try to ask @rottegift - I think he runs his home dir on ZFS but not sure if he's on 10.14.

Good luck! ;-)

[lundman]

Sorry I'm a bit absent, just started the big port of OpenZFS v2 to osx. More guys are generally in the irc channel as JMoVS suggested.

[captain-haddock17]

Something has to do with MacOS Sandboxing (Jail) of apps.

If found out this list sudo grep -i mail /usr/share/sandbox/*.sb

/usr/share/sandbox/mds.sb: (global-name "com.apple.mdworker.mail") /usr/share/sandbox/mdworker-mail.sb:;; Make changes just for Mail importer /usr/share/sandbox/mdworker-mail.sb:;; Allow Mail to read mail messages anywhere /usr/share/sandbox/mdworker-mail.sb:(allow file-read* (regex #"/Library/Mail(/|$)")) /usr/share/sandbox/mdworker-mail.sb: (allow file-read* (home-subpath "/Library/Mail")))) /usr/share/sandbox/mdworker-mail.sb: (allow file-read* (param-subpath "_LIBRARY" "/Mail")))) /usr/share/sandbox/mdworker-mail.sb:(if (positive? (string-length (param "_MAIL"))) /usr/share/sandbox/mdworker-mail.sb: (allow file-read* (subpath (param "_MAIL"))))) /usr/share/sandbox/mdworker-mail.sb:(if (positive? (string-length (param "_MAIL_V2"))) /usr/share/sandbox/mdworker-mail.sb: (allow file-read* (subpath (param "_MAIL_V2"))))) /usr/share/sandbox/quicklook-satellite-general.sb:;; Mail /usr/share/sandbox/quicklook-satellite-general.sb:(allow file-read* (home-subpath "/Library/Mail")) /usr/share/sandbox/quicklook-satellite-general.sb:(preference-read "com.apple.mail-shared")

Could be Mail.app do comply with the new security development rules ?

I think that this issue about Mail.app has a relation-ship with how ZFS can handle user's HomeDirectory #754 and special file flags #760 .