zfs Spotlight partially broken when com.apple.mimic

I found that Spotlight only partially works when com.apple.mimic_hfs is set to off. I use Alfredapp for quick searching of files and found that it suddenly didn't find things anymore. Also search in Mail.app was broken.

I debugged manually and found that even though Spotlight (the app) itself was able to find files, 3rd party and mdls weren't able to find/get details anymore.

To reproduce:

Use dataset with com.apple.mimic_hfs=off and try to query mdls SOMEFILE_ON_DATASET
mdls should return nothing
If you try and run mdls on something on apfs/hfs, it will return basic info
Setting com.apple.mimic_hfs=on and remounting the dataset should yield that mdls prints out information again and that 3rd party app's search works again

Nov 25 '18 18:11 JMoVS

The bug can be reproduced/observed when using the Alfred Metadata Tool https://cachefly.alfredapp.com/tools/AlfredMetadataTool_v1.1.zip and checking on files/folders.

Spotlight itself finds the files fine
The indexing status seems fine
When com.apple.mimic_hfs is set to off, no metadata is returned in the utility.

Let me know if you can reproduce now ;-)

Mar 14 '19 21:03 JMoVS

dtrussing mds when touching a new file, causes it to fire off 3 getattrlist() calls and one listxattr(). That is all the interaction it does with ZFS.

mimic_hfs=on

## mimic_hfs=on - working

116/0x30ae:  fsgetpath("/Volumes/tank/Hello.World2.txt\0", 0x400, 0x7FA6264CFE64) = 31 0

116/0x30ae:  getattrlist("/Volumes/tank/Hello.World2.txt\0", 0x7000011D1568, 0x7000011D1580, 0x5c 0x9) <commonattr 0x9001C00A, volattr 0x0, dirattr 0x2, fileattr 0x0, forkattr 0x0> = 0 0
ATTR_CMN_DEVID,ATTR_CMN_OBJTYPE,ATTR_CMN_FNDRINFO,ATTR_CMN_OWNERID,ATTR_CMN_GRPID,
ATTR_DIR_ENTRYCOUNT,
getattrlist -ATTR_CMN_DEVID,ATTR_CMN_OBJTYPE,ATTR_CMN_FNDRINFO,ATTR_CMN_OWNERID,ATTR_CMN_GRPID,ATTR_DIR_ENTRYCOUNT /Volumes/tank/Hello.World2.txt
    ATTR_CMN_DEVID      = 0x37000008 (major=55, minor=8)
    ATTR_CMN_OBJTYPE    = 1 (VREG)
    ATTR_CMN_FNDRINFO   = 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
                          00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
    ATTR_CMN_OWNERID    = 0 (root)
    ATTR_CMN_GRPID      = 0 (wheel)

116/0x30ae:  listxattr(0x7000011D16B0, 0x7000011D1AB0, 0x1000) = 0 0

116/0x30ae:  getattrlist("/Volumes/tank/Hello.World2.txt\0", 0x7000011CCEA8, 0x7000011CCAC0, 0x3da 0x25) <commonattr 0x903FDE0D, volattr 0x0, dirattr 0x5, fileattr 0x3601, forkattr 0x0> = 0 0
ATTR_CMN_NAME,ATTR_CMN_FSID,ATTR_CMN_OBJTYPE,ATTR_CMN_CRTIME,ATTR_CMN_MODTIME,ATTR_CMN_CHGTIME,ATTR_CMN_ACCTIME,ATTR_CMN_FNDRINFO,ATTR_CMN_OWNERID,ATTR_CMN_GRPID,ATTR_CMN_ACCESSMASK,ATTR_CMN_FLAGS,ATTR_CMN_GEN_COUNT,ATTR_CMN_DOCUMENT_ID,ATTR_CMN_USERACCESS,
ATTR_DIR_LINKCOUNT,ATTR_DIR_MOUNTSTATUS,
ATTR_FILE_LINKCOUNT,ATTR_FILE_DATALENGTH,ATTR_FILE_DATAALLOCSIZE,ATTR_FILE_RSRCLENGTH,ATTR_FILE_RSRCALLOCSIZE,

bash-3.2# ~/FSMegaInfo getattrlist -FSOPT_ATTR_CMN_EXTENDED -ATTR_CMN_NAME,ATTR_CMN_FSID,ATTR_CMN_OBJTYPE,ATTR_CMN_CRTIME,ATTR_CMN_MODTIME,ATTR_CMN_CHGTIME,ATTR_CMN_ACCTIME,ATTR_CMN_FNDRINFO,ATTR_CMN_OWNERID,ATTR_CMN_GRPID,ATTR_CMN_ACCESSMASK,ATTR_CMN_FLAGS,ATTR_CMN_GEN_COUNT,ATTR_CMN_DOCUMENT_ID,ATTR_CMN_USERACCESS,ATTR_DIR_LINKCOUNT,ATTR_DIR_MOUNTSTATUS,ATTR_FILE_LINKCOUNT,ATTR_FILE_DATALENGTH,ATTR_FILE_DATAALLOCSIZE,ATTR_FILE_RSRCLENGTH,ATTR_FILE_RSRCALLOCSIZE /Volumes/tank/Hello.World2.txt

    ATTR_CMN_NAME           = 'Hello.World2.txt'
    ATTR_CMN_FSID           = [0x37000008, 0x0000001f]
    ATTR_CMN_OBJTYPE        = 1 (VREG)
    ATTR_CMN_CRTIME         = 1555893202.231657454 (Mon Apr 22 00:33:22 2019)
    ATTR_CMN_MODTIME        = 1555893202.231657454 (Mon Apr 22 00:33:22 2019)
    ATTR_CMN_CHGTIME        = 1555893202.231657454 (Mon Apr 22 00:33:22 2019)
    ATTR_CMN_ACCTIME        = 1555893203.331617708 (Mon Apr 22 00:33:23 2019)
    ATTR_CMN_FNDRINFO       = 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
                              00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
    ATTR_CMN_OWNERID        = 0 (root)
    ATTR_CMN_GRPID          = 0 (wheel)
    ATTR_CMN_ACCESSMASK     = 0x000081a4 (-rw-r--r--)
    ATTR_CMN_NAMEDATTRCOUNT = 0
    ATTR_CMN_FLAGS          = 0x00000006
    ATTR_CMN_GEN_COUNT      = 1
    ATTR_CMN_DOCUMENT_ID    = 0
    ATTR_CMN_USERACCESS     = 0x00000000
    ATTR_FILE_LINKCOUNT     = 512
    ATTR_FILE_DATALENGTH    = 0 (0 bytes)
    ATTR_FILE_DATAALLOCSIZE = 0 (0 bytes)
    ATTR_FILE_RSRCLENGTH    = 7812730813493018624 (7276172576 GB)
    ATTR_FILE_RSRCALLOCSIZE = 3631146438317518447 (3381768649 GB)


116/0x30ae:  getattrlist("/.vol/922746888/177\0", 0x7000011D3F48, 0x7000011D47B0, 0x20 0x0) <commonattr 0x4000000, volattr 0x0, dirattr 0x0, fileattr 0x0, forkattr 0x0> = 0 0
ATTR_CMN_PARENTID,
~/FSMegaInfo getattrlist -ATTR_CMN_PARENTID /.vol/922746888/177
getattrlist -ATTR_CMN_PARENTID /.vol/922746888/177
    ATTR_CMN_PARENTID = 2

mimic_hfs=off

stat '/Volumes/tank/doesnt.txt'
    st_dev     = 0x37000007 (major=55, minor=7)
    st_ino     = 135
    st_mode    = 0x000081a4 (-rw-r--r--)
    st_nlink   = 1
    st_uid     = 0 (root)
    st_gid     = 0 (wheel)
    st_rdev    = 0x00000000 (major=0, minor=0, console)
    st_atime   = 1555892744.187127814 (Mon Apr 22 00:25:44 2019)
    st_mtime   = 1555892743.90322708 (Mon Apr 22 00:25:43 2019)
    st_ctime   = 1555892743.90322708 (Mon Apr 22 00:25:43 2019)
    st_size    = 0 (0 bytes)
    st_blocks  = 1
    st_blksize = 131072
    st_flags   = 0x00000000
    st_gen     = 0

116/0x2f17:  fsgetpath("/Volumes/tank/doesnt2.txt\0", 0x400, 0x7FA62885E944) = 26 0

116/0x2f17:  getattrlist("/Volumes/tank/doesnt2.txt\0", 0x700000DB9568, 0x700000DB9580, 0x5c 0x9) <commonattr 0x9001C00A, volattr 0x0, dirattr 0x2, fileattr 0x0, forkattr 0x0> = 0 0
ATTR_CMN_DEVID,ATTR_CMN_OBJTYPE,ATTR_CMN_FNDRINFO,ATTR_CMN_OWNERID,ATTR_CMN_GRPID,
ATTR_DIR_ENTRYCOUNT,

~/FSMegaInfo getattrlist -ATTR_CMN_DEVID,ATTR_CMN_OBJTYPE,ATTR_CMN_FNDRINFO,ATTR_CMN_OWNERID,ATTR_CMN_GRPID,ATTR_DIR_ENTRYCOUNT /Volumes/tank/doesnt.txt
getattrlist -ATTR_CMN_DEVID,ATTR_CMN_OBJTYPE,ATTR_CMN_FNDRINFO,ATTR_CMN_OWNERID,ATTR_CMN_GRPID,ATTR_DIR_ENTRYCOUNT /Volumes/tank/doesnt.txt

    ATTR_CMN_DEVID      = 0x37000009 (major=55, minor=9)
    ATTR_CMN_OBJTYPE    = 1 (VREG)
    ATTR_CMN_FNDRINFO   = 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
                          00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
    ATTR_CMN_OWNERID    = 0 (root)
    ATTR_CMN_GRPID      = 0 (wheel)


116/0x2f17:  listxattr(0x700000DB96B0, 0x700000DB9AB0, 0x1000) = 0 0

116/0x2f17:  getattrlist("/Volumes/tank/doesnt2.txt\0", 0x700000DB4EA8, 0x700000DB4AC0, 0x3da 0x25) <commonattr 0x903FDE0D, volattr 0x0, dirattr 0x5, fileattr 0x3601, forkattr 0x0> = 0 0
ATTR_CMN_NAME,ATTR_CMN_FSID,ATTR_CMN_OBJTYPE,ATTR_CMN_CRTIME,ATTR_CMN_MODTIME,ATTR_CMN_CHGTIME,ATTR_CMN_ACCTIME,ATTR_CMN_FNDRINFO,ATTR_CMN_OWNERID,ATTR_CMN_GRPID,ATTR_CMN_ACCESSMASK,ATTR_CMN_FLAGS,ATTR_CMN_GEN_COUNT,ATTR_CMN_DOCUMENT_ID,ATTR_CMN_USERACCESS,
ATTR_DIR_LINKCOUNT,ATTR_DIR_MOUNTSTATUS,
ATTR_FILE_LINKCOUNT,ATTR_FILE_DATALENGTH,ATTR_FILE_DATAALLOCSIZE,ATTR_FILE_RSRCLENGTH,ATTR_FILE_RSRCALLOCSIZE,

~/FSMegaInfo getattrlist -FSOPT_ATTR_CMN_EXTENDED -ATTR_CMN_NAME,ATTR_CMN_FSID,ATTR_CMN_OBJTYPE,ATTR_CMN_CRTIME,ATTR_CMN_MODTIME,ATTR_CMN_CHGTIME,ATTR_CMN_ACCTIME,ATTR_CMN_FNDRINFO,ATTR_CMN_OWNERID,ATTR_CMN_GRPID,ATTR_CMN_ACCESSMASK,ATTR_CMN_FLAGS,ATTR_CMN_GEN_COUNT,ATTR_CMN_DOCUMENT_ID,ATTR_CMN_USERACCESS,ATTR_DIR_LINKCOUNT,ATTR_DIR_MOUNTSTATUS,ATTR_FILE_LINKCOUNT,ATTR_FILE_DATALENGTH,ATTR_FILE_DATAALLOCSIZE,ATTR_FILE_RSRCLENGTH,ATTR_FILE_RSRCALLOCSIZE /Volumes/tank/doesnt.txt

    ATTR_CMN_NAME           = 'doesnt.txt'
    ATTR_CMN_FSID           = [0x37000009, 0x0000001f]
    ATTR_CMN_OBJTYPE        = 1 (VREG)
    ATTR_CMN_CRTIME         = 1555892743.90322708 (Mon Apr 22 00:25:43 2019)
    ATTR_CMN_MODTIME        = 1555892743.90322708 (Mon Apr 22 00:25:43 2019)
    ATTR_CMN_CHGTIME        = 1555892743.90322708 (Mon Apr 22 00:25:43 2019)
    ATTR_CMN_ACCTIME        = 1555892743.90322708 (Mon Apr 22 00:25:43 2019)
    ATTR_CMN_FNDRINFO       = 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
                              00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
    ATTR_CMN_OWNERID        = 0 (root)
    ATTR_CMN_GRPID          = 0 (wheel)
    ATTR_CMN_ACCESSMASK     = 0x000081a4 (-rw-r--r--)
    ATTR_CMN_NAMEDATTRCOUNT = 0
    ATTR_CMN_FLAGS          = 0x00000006
    ATTR_CMN_GEN_COUNT      = 1
    ATTR_CMN_DOCUMENT_ID    = 0
    ATTR_CMN_USERACCESS     = 0x00000000
    ATTR_FILE_LINKCOUNT     = 512
    ATTR_FILE_DATALENGTH    = 0 (0 bytes)
    ATTR_FILE_DATAALLOCSIZE = 0 (0 bytes)
    ATTR_FILE_RSRCLENGTH    = 8315174762296901632 (7744109968 GB)
    ATTR_FILE_RSRCALLOCSIZE = 128060694099054 (119265 GB)

116/0x2f17:  getattrlist("/.vol/922746887/138\0", 0x700000DBBF48, 0x700000DBC7B0, 0x20 0x0) <commonattr 0x4000000, volattr 0x0, dirattr 0x0, fileattr 0x0, forkattr 0x0> = 0 0
ATTR_CMN_PARENTID,

getattrlist -ATTR_CMN_PARENTID /.vol/922746889/135
    ATTR_CMN_PARENTID = 2

Apr 22 '19 00:04 lundman

Yeah, my addition to FSMegaTool to add ATTR_CMN_GEN_COUNT and ATTR_CMN_DOCUMENT_ID is wrong, so it looks at wrong offset. Without those two it is

# Not working
    ATTR_CMN_USERACCESS     = 0x00000006
    ATTR_FILE_LINKCOUNT     = 1
    ATTR_FILE_DATALENGTH    = 0 (0 bytes)
    ATTR_FILE_DATAALLOCSIZE = 512 (512 bytes)
    ATTR_FILE_RSRCLENGTH    = 0 (0 bytes)
    ATTR_FILE_RSRCALLOCSIZE = 0 (0 bytes)

# Working
    ATTR_CMN_USERACCESS     = 0x00000006
    ATTR_FILE_LINKCOUNT     = 1
    ATTR_FILE_DATALENGTH    = 0 (0 bytes)
    ATTR_FILE_DATAALLOCSIZE = 512 (512 bytes)
    ATTR_FILE_RSRCLENGTH    = 0 (0 bytes)
    ATTR_FILE_RSRCALLOCSIZE = 0 (0 bytes)

I do not believe we do anything wrong in our getattr() reply.

Apr 22 '19 02:04 lundman

With mimic, it is worth remembering that the only thing we do differently, is vfsstatfs->f_fstypename, "hfs".

Interestingly, if I reply with apfs it also fails. I wonder if there is a compatibility mode for hfs, that we have latched onto.

Looking at replying apfs though, the only other thing they might detect is the reply in CAPABILITIES.

Apr 22 '19 02:04 lundman

Capabilities, we differ:

# Difference to ZFS=YES, then APFS is opposite.
 VOL_CAP_FMT_CASE_SENSITIVE      = YES 
 VOL_CAP_INT_SEARCHFS            = NO
 VOL_CAP_INT_COPYFILE            = NO
 VOL_CAP_INT_ALLOCATE            = NO
... and others (0xf0000)apfs
# apfs
  VOL_CAP_FMT_WRITE_GENERATION_COUNT | VOL_CAP_FMT_NO_IMMUTABLE_FILES | 
  VOL_CAP_FMT_NO_PERMISSIONS

 ... and others (0x7c0000)apfs    ... and others (0x600000)zfs
# apfs
VOL_CAP_FMT_DIR_HARDLINKS VOL_CAP_FMT_DOCUMENT_ID 
VOL_CAP_FMT_WRITE_GENERATION_COUNT VOL_CAP_FMT_NO_IMMUTABLE_FILES VOL_CAP_FMT_NO_PERMISSIONS
# zfs
VOL_CAP_FMT_NO_IMMUTABLE_FILES VOL_CAP_FMT_NO_PERMISSIONS 

... and others (0xf8000)apfs
# apfs 
VOL_CAP_INT_REMOTE_EVENT VOL_CAP_INT_CLONE VOL_CAP_INT_SNAPSHOT VOL_CAP_INT_RENAME_SWAP 
VOL_CAP_INT_RENAME_EXCL

Apr 22 '19 02:04 lundman

Alas, lying and having identical Capabilities makes no difference.

Apr 22 '19 02:04 lundman

hmm, but if we lie about our capabilities and don't have them, would we see that it's trying to call an ioctl or use a capability we don't have and fail silently?

Apr 23 '19 07:04 JMoVS

dtruss shows all calls that mds does, and none of them are special. We also get noisy when something calls something we do not have, or support, - so we don't miss these cases :)

Apr 23 '19 07:04 lundman

The smallest set where we have to report as "hfs" is if process_name() == "mds" or "coreservicesd". Then it works.

coreservicesd stack looks like:

              kernel.development`vfs_statfs+0xb
              zfs`zfs_vfs_getattr+0x2cc
              kernel.development`getattrlist_internal+0x6a6
              kernel.development`getattrlistat_internal+0x11f
              kernel.development`getattrlist+0x70
              kernel.development`unix_syscall64+0x2f8
              kernel.development`hndl_unix_scall64+0x16

              libsystem_kernel.dylib`__getattrlist+0xa
              CarbonCore`FSNode_SetFSSharedDeviceInfo+0x58
              CarbonCore`FSNode_VolumeMounted+0xa5
              CarbonCore`FSNodeServer_SyncSystemUniverseInternal+0x4d7
              CarbonCore`FSNodeServer_SyncAllUniverses+0x7c
              CoreFoundation`__CFMachPortPerform+0x11a

Which is what you'd expect to see if it wanted to look up the fstype.

Apr 26 '19 06:04 lundman

ok. Any news from Apple yet?

Apr 27 '19 11:04 JMoVS

zfs
zfs copied to clipboard

Spotlight partially broken when com.apple.mimic_hfs=off on Mojave

zfs zfs copied to clipboard

Spotlight partially broken when com.apple.mimic_hfs=off on Mojave

zfs
zfs copied to clipboard