Spotlight problems as of High Sierra and continuously in Mojave

[JMoVS]

There are a couple of issues with spotlight it seems as of High Sierra which may or may not be related but have gotten worse with Mojave:

• https://github.com/openzfsonosx/zfs/issues/613
• https://github.com/openzfsonosx/zfs/issues/649
• https://github.com/openzfsonosx/zfs/issues/671

@cbreak-black may also have spotlight issues related to newer builds on High Sierra

@lundman What can we do so that we can help you fix those?

[JMoVS]

I myself have the problem that with some datasets, mds gets into 100% usage of all CPU cores after a while, the dataset can be exported with force but the pool cannot be exported and is reported "busy".

[rottegift]

I'd consider (and have done, myself) making the indexing nicer. That will help you with the CPU, or at least responsiveness.

Make changes in:

/System/Library/LaunchDaemons/com.apple.metadata.mds.scan.plist
and in
/System/Library/LaunchAgents/
com.apple.mdworker.sizing.plist
com.apple.mdworker.32bit.plist
com.apple.mdworker.single.plist
com.apple.mdworker.bundles.plist
com.apple.mdworker.isolation.plist
com.apple.mdworker.lsb.plist
com.apple.mdworker.mail.plist
com.apple.mdworker.shared.plist

Emacs will edit these for you just fine; other editors are possibly smart enough to decode the binary plist format for you, or you can do:

plutil  -convert xml1 /System/Library/LaunchAgents/com.apple.mdworker.shared.plist

(for example)

and then edit the resulting text file. Launchd will eventually turn the plist files back to binary format.

Add the following key:

        <key>Nice</key>
        <integer>10</integer>

in the dict that contains the POSIXSpawnType key.

Change the </false> to </true> below the <key>LowPriorityBackgroundIO</key>.

If the key is missing in one of the plists just add it like the Nice key above.

You will need to log out and log in again, or reboot, for the new keys to take effect, although if you are feeling intrepid you can use launchctl to unload and reload each plist file; you will have to manually kill the running mds/mdworker processes.

Other things you can do:

• Use System Preferences > Spotlight > Privacy pane aggressively to prune off directories that have many files in them.

• Use mdutil -vas to see what's causing the re-indexing, and use mdutil -i off or mdutil -E to get rid of anything superfluous or stuck

• If you don't need it indexed, whenever you do a zfs create, do a "sudo touch /Volumes/new/zfs/dataset/.metadata_never_index"

(And if you feel really intrepid, make zed do that by default, unless some property is set !)

[JMoVS]

@rottegift I never have problems with spotlight but at random times when something is clearly wrong and it just explodes usage wise. The usage explosion (where I can't even get CPU usage down and responsiveness up by killing mds, which doesn't work in these situations) is a different issue than a normal performance issue.

But thank you for documenting it, haven't seen that detailed documenting anywhere yet.

[cbreak-black]

I had similar problems lately. Maybe since I upgraded to 10.13.6, but I'm not sure. The symptoms are unusually high CPU load (not just normal indexing), and spotlight / quicklook / other programs becomming unresponsive.

Attached a spindump from when it happened, truncated for readability (the full thing is over 6MB). Possibly related symptoms: I lately sometimes can no longer turn off my computer, it hangs with the spinning thingie on the black screen.

MDS-Spindump.txt

[lundman]

That spindump is showing zget getting stuck, waiting a tiny bit, and trying again.

[JMoVS]

The zget branch feels like it got a memory leak. At least my system wasn‘t happy after 30 hours of uptime... Will check whether it‘s the branch or sth else

[JMoVS]

spindump with current master when the problem appears:

https://d.pr/free/f/bihqKA

[cbreak-black]

An other spindump when mds becomes CPU hungry. This is using 5ffe3d6cefe6da. mds-spindump-shorter.txt