Cannot load 3rd party kexts

[fpgaminer]

Describe the issue 3rd party kexts, like macFUSE or the kext I am working on personally, won't load. After reboot I get a pop-up message that says "System Extension Error", "An error occurred with your system extensions during startup and they need to be rebuilt before they can be used. Please go to Security & Privacy System Preferences to re-enable them."

If I go to Preferences and do as it asks, after reboot the same message appears.

SIP is disabled, developer mode is on, etc.

The same kexts work fine on bare metal.

Configuration

• UTM Version: 3.2.3 (57)
• Host OS Version: macOS 12.3.1
• Intel or Apple Silicon? Apple M1 Max

Logs Output of log show --debug | grep -i macfuse:

2022-05-16 10:36:20.878345-0700 0x263      Default     0x0                  100    0    kernelmanagerd: validating extension at /Library/Filesystems/macfuse.fs/Contents/Extensions/12/macfuse.kext
2022-05-16 10:36:20.889780-0700 0x263      Default     0x0                  100    0    kernelmanagerd: [com.apple.kernelmanagerd.logging:LoadRequestResolution] checking bundle path /Library/Filesystems/macfuse.fs/Contents/Extensions/12/macfuse.kext for Kext io.macfuse.filesystems.macfuse v2085.20 in executable kext bundle io.macfuse.filesystems.macfuse at /Library/Filesystems/macfuse.fs/Contents/Extensions/12/macfuse.kext
2022-05-16 10:36:20.889807-0700 0x263      Default     0x0                  100    0    kernelmanagerd: [com.apple.kernelmanagerd.logging:LoadRequestResolution] checking unstaged bundle path /Library/Filesystems/macfuse.fs/Contents/Extensions/12/macfuse.kext for Kext io.macfuse.filesystems.macfuse v2085.20 in executable kext bundle io.macfuse.filesystems.macfuse at /Library/Filesystems/macfuse.fs/Contents/Extensions/12/macfuse.kext
	Kext io.macfuse.filesystems.macfuse v2085.20 in executable kext bundle io.macfuse.filesystems.macfuse at /Library/Filesystems/macfuse.fs/Contents/Extensions/12/macfuse.kext 
2022-05-16 10:36:20.961129-0700 0x54c      Default     0x0                  180    0    syspolicyd: [com.apple.syspolicy.kext:default] canLoadKernelExtension - direct, evaluate, Kext (3T5GSNBU6W, io.macfuse.filesystems.macfuse) version: 25d8ea0d2b621051e8900f62f6613c904076c8f9
2022-05-16 10:36:20.962087-0700 0x54c      Default     0x0                  180    0    syspolicyd: [com.apple.syspolicy.kext:default] Kernel Extension ALLOWED: Kext (3T5GSNBU6W, io.macfuse.filesystems.macfuse) version: 25d8ea0d2b621051e8900f62f6613c904076c8f9
	Kext io.macfuse.filesystems.macfuse v2085.20 in executable kext bundle io.macfuse.filesystems.macfuse at /Library/Filesystems/macfuse.fs/Contents/Extensions/12/macfuse.kext 
	/Library/Filesystems/macfuse.fs/Contents/Extensions/12/macfuse.kext
	Kext io.macfuse.filesystems.macfuse v2085.20 in executable kext bundle io.macfuse.filesystems.macfuse at /Library/Filesystems/macfuse.fs/Contents/Extensions/12/macfuse.kext (new)
		/Library/Filesystems/macfuse.fs/Contents/Extensions/12/macfuse.kext
2022-05-16 10:36:21.282160-0700 0x54c      Default     0x0                  180    0    syspolicyd: [com.apple.syspolicy.kext:default] canLoadKernelExtension - direct, evaluate, Kext (3T5GSNBU6W, io.macfuse.filesystems.macfuse) version: 25d8ea0d2b621051e8900f62f6613c904076c8f9
2022-05-16 10:36:21.282557-0700 0x54c      Default     0x0                  180    0    syspolicyd: [com.apple.syspolicy.kext:default] Kernel Extension ALLOWED: Kext (3T5GSNBU6W, io.macfuse.filesystems.macfuse) version: 25d8ea0d2b621051e8900f62f6613c904076c8f9
2022-05-16 10:36:21.282857-0700 0x54c      Default     0x0                  180    0    syspolicyd: [com.apple.syspolicy.kext:default] Triggered kext upgrade: Kext (3T5GSNBU6W, io.macfuse.filesystems.macfuse) version: 25d8ea0d2b621051e8900f62f6613c904076c8f9
2022-05-16 10:36:21.282978-0700 0x54c      Default     0x0                  180    0    syspolicyd: [com.apple.syspolicy.kext:default] Blocking upgrade dialog for kext that is already pending upgrade: Kext (3T5GSNBU6W, io.macfuse.filesystems.macfuse) version: 25d8ea0d2b621051e8900f62f6613c904076c8f9
2022-05-16 10:36:22.559237-0700 0x2e5      Default     0x0                  100    0    kernelmanagerd: validating extension at /Library/Filesystems/macfuse.fs/Contents/Extensions/12/macfuse.kext
2022-05-16 10:36:22.560082-0700 0x2e5      Default     0x0                  100    0    kernelmanagerd: [com.apple.kernelmanagerd.logging:LoadRequestResolution] checking bundle path /Library/Filesystems/macfuse.fs/Contents/Extensions/12/macfuse.kext for Kext io.macfuse.filesystems.macfuse v2085.20 in executable kext bundle io.macfuse.filesystems.macfuse at /Library/Filesystems/macfuse.fs/Contents/Extensions/12/macfuse.kext
2022-05-16 10:36:22.560102-0700 0x2e5      Default     0x0                  100    0    kernelmanagerd: [com.apple.kernelmanagerd.logging:LoadRequestResolution] checking unstaged bundle path /Library/Filesystems/macfuse.fs/Contents/Extensions/12/macfuse.kext for Kext io.macfuse.filesystems.macfuse v2085.20 in executable kext bundle io.macfuse.filesystems.macfuse at /Library/Filesystems/macfuse.fs/Contents/Extensions/12/macfuse.kext
2022-05-16 10:42:44.432792-0700 0x184f     Default     0x0                  100    0    kernelmanagerd: [com.apple.kernelmanagerd.logging:LoadRequestResolution] loadExtensions(withPaths:withIdentifiers:withPersonalityNames:withDependencyAndFolderPaths:options:withReply:): Loading paths: ["/Library/Filesystems/macfuse.fs/Contents/Extensions/12/macfuse.kext"], identifiers: [], personalityNames: []
2022-05-16 10:42:44.493065-0700 0x184f     Default     0x0                  100    0    kernelmanagerd: validating extension at /Library/Filesystems/macfuse.fs/Contents/Extensions/12/macfuse.kext
2022-05-16 10:42:44.493840-0700 0x184f     Default     0x0                  100    0    kernelmanagerd: [com.apple.kernelmanagerd.logging:LoadRequestResolution] checking bundle path /Library/Filesystems/macfuse.fs/Contents/Extensions/12/macfuse.kext for Kext io.macfuse.filesystems.macfuse v2085.20 in executable kext bundle io.macfuse.filesystems.macfuse at /Library/Filesystems/macfuse.fs/Contents/Extensions/12/macfuse.kext
2022-05-16 10:42:44.493858-0700 0x184f     Default     0x0                  100    0    kernelmanagerd: [com.apple.kernelmanagerd.logging:LoadRequestResolution] checking unstaged bundle path /Library/Filesystems/macfuse.fs/Contents/Extensions/12/macfuse.kext for Kext io.macfuse.filesystems.macfuse v2085.20 in executable kext bundle io.macfuse.filesystems.macfuse at /Library/Filesystems/macfuse.fs/Contents/Extensions/12/macfuse.kext
2022-05-16 10:42:44.509628-0700 0x184f     Default     0x0                  100    0    kernelmanagerd: validating extension at /Library/Filesystems/macfuse.fs/Contents/Extensions/12/macfuse.kext
2022-05-16 10:42:44.510237-0700 0x184f     Default     0x0                  100    0    kernelmanagerd: [com.apple.kernelmanagerd.logging:LoadRequestResolution] At least one of the request extensions / bundled extensions io.macfuse.filesystems.macfuse are NOT in the exclude list. Will do full resolution of load request
	Kext io.macfuse.filesystems.macfuse v2085.20 in executable kext bundle io.macfuse.filesystems.macfuse at /Library/Filesystems/macfuse.fs/Contents/Extensions/12/macfuse.kext
2022-05-16 10:42:44.520349-0700 0x184f     Default     0x0                  100    0    kernelmanagerd: validating extension at /Library/Filesystems/macfuse.fs/Contents/Extensions/12/macfuse.kext
	Kext io.macfuse.filesystems.macfuse v2085.20 in executable kext bundle io.macfuse.filesystems.macfuse at /Library/Filesystems/macfuse.fs/Contents/Extensions/12/macfuse.kext 
2022-05-16 10:42:44.531168-0700 0x1953     Default     0x0                  180    0    syspolicyd: [com.apple.syspolicy.kext:default] canLoadKernelExtension - direct, evaluate, Kext (3T5GSNBU6W, io.macfuse.filesystems.macfuse) version: 25d8ea0d2b621051e8900f62f6613c904076c8f9
2022-05-16 10:42:44.531854-0700 0x1953     Default     0x0                  180    0    syspolicyd: [com.apple.syspolicy.kext:default] Kernel Extension ALLOWED: Kext (3T5GSNBU6W, io.macfuse.filesystems.macfuse) version: 25d8ea0d2b621051e8900f62f6613c904076c8f9
	Kext io.macfuse.filesystems.macfuse v2085.20 in executable kext bundle io.macfuse.filesystems.macfuse at /Library/Filesystems/macfuse.fs/Contents/Extensions/12/macfuse.kext 
	/Library/Filesystems/macfuse.fs/Contents/Extensions/12/macfuse.kext
	Kext io.macfuse.filesystems.macfuse v2085.20 in executable kext bundle io.macfuse.filesystems.macfuse at /Library/Filesystems/macfuse.fs/Contents/Extensions/12/macfuse.kext (new)
		/Library/Filesystems/macfuse.fs/Contents/Extensions/12/macfuse.kext
	/Library/Filesystems/macfuse.fs/Contents/Extensions/12/macfuse.kext
2022-05-16 10:42:44.994916-0700 0x184d     Default     0x0                  180    0    syspolicyd: [com.apple.syspolicy.kext:default] canLoadKernelExtension - direct, evaluate, Kext (3T5GSNBU6W, io.macfuse.filesystems.macfuse) version: 25d8ea0d2b621051e8900f62f6613c904076c8f9
2022-05-16 10:42:44.995262-0700 0x184d     Default     0x0                  180    0    syspolicyd: [com.apple.syspolicy.kext:default] Kernel Extension ALLOWED: Kext (3T5GSNBU6W, io.macfuse.filesystems.macfuse) version: 25d8ea0d2b621051e8900f62f6613c904076c8f9
2022-05-16 10:42:44.995521-0700 0x184d     Default     0x0                  180    0    syspolicyd: [com.apple.syspolicy.kext:default] Triggered kext upgrade: Kext (3T5GSNBU6W, io.macfuse.filesystems.macfuse) version: 25d8ea0d2b621051e8900f62f6613c904076c8f9
2022-05-16 10:42:44.995636-0700 0x184d     Default     0x0                  180    0    syspolicyd: [com.apple.syspolicy.kext:default] Blocking upgrade dialog for kext that is already pending upgrade: Kext (3T5GSNBU6W, io.macfuse.filesystems.macfuse) version: 25d8ea0d2b621051e8900f62f6613c904076c8f9
2022-05-16 10:42:44.995862-0700 0x184f     Error       0x0                  100    0    kernelmanagerd: failed to load/rebuild extensions: Extension with identifiers io.macfuse.filesystems.macfuse not approved to load. Please approve using System Preferences.
2022-05-16 10:42:45.006380-0700 0x1961     Error       0x0                  644    0    load_macfuse: (KernelManagement) Remote encountered error: Error Domain=KMErrorDomain Code=27 "Extension with identifiers io.macfuse.filesystems.macfuse not approved to load. Please approve using System Preferences." UserInfo={NSLocalizedDescription=Extension with identifiers io.macfuse.filesystems.macfuse not approved to load. Please approve using System Preferences.}
2022-05-16 10:42:45.006466-0700 0x1961     Error       0x0                  644    0    load_macfuse: (KernelManagement) Extension load failed: <private>

[osy]

Same root cause as #3526

[steven-michaud]

This bug still exists in macOS 13.0 Beta (build 22A5266r), running as both host and guest. I tested with UTM 3.2.4.

So Apple hasn't fixed it in Ventura -- at least not yet.

[steven-michaud]

This bug (Apple's bug) still exists in macOS 13.0 Beta 2 (build 22A5286j).

[steven-michaud]

Apple's bug still exists in macOS 13.0 beta 3 (build 22A5295h).

[jteloh]

Perhaps Applespecified it that way. They have moving away from kexts for awhile.

Apple's bug still exists in macOS 13.0 beta 3 (build 22A5295h).

[steven-michaud]

That's possible. But it may also just be an oversight -- Apple still supports them in macOS 13 "on bare metal". Only time will tell.

I'll keep testing new Ventura betas. But I'll only report here if I find one that works.

[lundman]

Ah just came across this problem myself, sadly after I posted about it on the dev forums. Oh well, maybe I'll get a reply.

[lundman]

Still doesn't work in Ventura 13.0 and UTM v4.0.8

[dariaphoebe]

I reported the inability of enabling kexts on machines running via virtualization.framework as to Apple as FB10145502 (third party kexts cannot be used in Virtualization framework apps) a bit ago and as yet have heard nothing, for what it's worth

[dariaphoebe]

To follow up on this, using bputil from a terminal in recovery mode gives what's probably the most useful hint: Error Domain 401, "Failed to create local policy", underlying error 0x600001f64150, domain=com.apple.bootpolicy, code=2, "SEP communication (2)"

So it's almost certainly going to be that Virtualization.framework neither provides access to the real SEP nor emulates one fully, and so, until Apple decides to properly support Virtualization, we're just sad.

[steven-michaud]

@dariaphoebe, I'm thinking of trying to debug this on an Intel Mac, since I can use HookCase there. Any suggestions how I might go about it, or is it a lost cause? Is the Intel environment just too different from the Apple Silicon environment?

If I'm really lucky, I might be able to find a hackish workaround for the problem. Something like https://github.com/utmapp/UTM/issues/3904#issuecomment-1100924393. But I might also end up wasting a lot of time.

[lundman]

Ah I only shared the log output in the Apple Dev Forums; the only lines I see are:

: Kcgen roundtrip failed with: Boot policy error: Error creating linked manifest: code BOOTPOLICY_ERROR_ACM   
: Kcgen roundtrip failed checkpoint saveAuxkc: status:error fatalError:Optional("Boot policy error: Error creating linked manifest: code BOOTPOLICY_ERROR_ACM")    
: Kcgen roundtrip failed: missing last checkpoint or errors found    
: Deleting Preboot content

but could be unrelated.

[steven-michaud]

I've made some progress debugging this problem. But I haven't, unfortunately, found a workaround. And what I have found indicates it happens at a very low level, and is almost certainly the result of a deliberate policy decision by Apple.

After much digging around in kmutil log show logs, I ended up in more or less the same place as @lundman. As best I can tell, the error he found (BOOTPOLICY_ERROR_ACM) is the key to this bug. This doesn't happen on bare metal, on either macOS 12 or 13.

One of the things I first noticed is that there isn't any "auxiliary kext collection" at all in macOS 12 and 13 VMs (created using the Virtualization framework). (Note the output from kmutil inspect.) This despite the fact that one of its would-be components is an Apple-signed kext in /Library/Apple/System/Library/Extensions -- RemoteVirtualInterface.kext. (This is also not initially loaded on bare metal. It gets loaded along with your first third-party kext.) A necessary step in creating one is to create a "linked manifest" -- a file ending in *.im4m that gets installed to a LocalPolicy subdirectory of /System/Volumes/iSCPreboot. This happens with no problems on bare metal. But in a VM you see the following errors:

macOS 13

kcgend: (libbootpolicy.dylib) [com.apple.BootPolicy:Library] BootPolicy: bootpolicy_create_linked_manifest: entry
kcgend: (libbootpolicy.dylib) [com.apple.BootPolicy:Library] BootPolicy: SEP command 33 (v7) returned 7: bpbc 5150, 0, 0, 0
kcgend: (libbootpolicy.dylib) [com.apple.BootPolicy:Library] BootPolicy: assert: bpe == 0  (/AppleInternal/Library/BuildRoots/0d3d51bd-555a-11ed-88cc-a23c4f261b56/Library/Caches/com.apple.xbs/Sources/BootPolicy/dylib/dylib.c:2437)
kcgend: (libbootpolicy.dylib) [com.apple.BootPolicy:Library] BootPolicy: bootpolicy_create_linked_manifest: exit: ACM (7)
kcgend: Could not createLinkedManifest and/or update auxKC receipt: Boot policy error: Error creating linked manifest: code ACM (7)

macOS 12

kcgend: (libbootpolicy.dylib) [com.apple.BootPolicy:Library] BootPolicy: bootpolicy_create_linked_manifest: entry
kcgend: (libbootpolicy.dylib) [com.apple.BootPolicy:Library] BootPolicy: SEP command 33 (v7) returned 7: bpbc 4177, 0, 0, 0
kcgend: (libbootpolicy.dylib) [com.apple.BootPolicy:Library] BootPolicy: assert: bpe == 0  (/AppleInternal/Library/BuildRoots/d23df31a-1fa6-11ed-91f3-e2c7c6032f02/Library/Caches/com.apple.xbs/Sources/BootPolicy/dylib/dylib.c:2558)
kcgend: (libbootpolicy.dylib) [com.apple.BootPolicy:Library] BootPolicy: bootpolicy_create_linked_manifest: exit: ACM (7)
...
kcgend: persisting error: Boot policy error: Error creating linked manifest: code BOOTPOLICY_ERROR_ACM
...
kernelmanagerd: Kcgen roundtrip failed with: Boot policy error: Error creating linked manifest: code BOOTPOLICY_ERROR_ACM

So a "linked manifest" is an expression of "local policy", and also of "boot policy". I'd guess "SEP" stands for something like "Secure Enclave Proxy". It's accessed through several software layers (including /usr/lib/libbootpolicy.dylib), the lowest of which seems to be /System/Library/Extensions/BootPolicy.kext. I don't know if the Virtualization frameworks' VMs just emulate Apple's "Secure Enclave" (I suspect they do). But, unlike @dariaphoebe, I had no problems communicating with the "SEP" in my VMs, on either macOS 12 or 13.

The "SEP command" that fails in VMs is "SEP command 33". It always succeeds on bare metal (at least in my experience). I see no references to it online. Over time I'll try to learn more about it. If Virtualization framework VMs emulate the "Secure Enclave", it should be possible, at least in principle, to mess with that emulation, to make "command 33" succeed. I'm sure this won't be easy, though, and I may never learn how to do it.

Someone at Apple must have made a deliberate decision to make "command 33" fail in Virtualization framework VMs. That sits oddly with other indications that they intended to support third-party kexts in these VMs -- for example they let you allow third-party kexts in the Recovery Partition. But Apple is a large company, and this won't be the first time the left hand doesn't know what the right hand is doing.

My logs have lots of entries headed "kcgend:". It's completely undocumented. But as best I can tell it's what actually writes "linked manifests" and kernel caches as your Mac boots. It's not meant to run in a VM, and normally doesn't (the result of another deliberate decision by someone at Apple). But I was able to find a way around this: Just run sudo /usr/libexec/kcgend once in a VM, and it will always run on subsequent boots. Be aware that this will reboot your VM. When I first tried this trick I waited with bated breath for the reboot to finish -- only to discover that this wasn't the workaround I'd hoped it would be :-(

[steven-michaud]

I've learned that "SEP" means "Secure Enclave Processor" ... probably. And "ACM" might mean "Authenticated Code Module".

[steven-michaud]

After more digging around, I discovered there are three kernel extensions that can receive "SEP commands". (These are the only three kexts that have readFromSEPBuffer() commands.)

BootPolicy.kext
AppleCredentialManager.kext
AppleSSE.kext

Then I wrote a dtrace script to trace when any of them receives a SEP command. SEPCommand.d.txt

Rename this file from SEPCommand.d.txt to SEPCommand.d, then run sudo dtrace -s SEPCommand.d on a machine that hosts UTM VMs that use the Virtualization framework. You may first need to turn off SIP (with csrutil disable while booted into the Recovery Partition).

The easiest way to trigger SEP commands to BootPolicy.kext is to check for macOS updates (in the System Preferences panel). The easiest way for AppleCredentialManager.kext is to run Safari.

But you don't see anything when you run or reboot a UTM VM (that uses the Virtualization framework). This tells me that Apple's implementation of the Secure Enclave in the Virtualization framework is an emulation -- possibly not a complete one. It doesn't call out to the Secure Enclave Processor in the host.

Another interesting bit: There are three "AppleVP" kexts, one of which (AppleVPBootPolicy.kext) also contains a readFromSEPBuffer() method. I missed it because dtrace (on the host) doesn't list of any of its methods, because it's never loaded on a "host". All three of these only run in a Virtualization framework VM. Unfortunately, dtrace won't work on any of them there. Presumably these contain at least part of Apple's emulation of the SEP.

AppleVPBootPolicy.kext
AppleVPKeyStore.kext
AppleVPCredentialManager.kext

More interesting stuff: All but one of the "SEP methods" in AppleVPBootPolicy.kext are no-ops. (The only one that isn't is BootPolicyUserClient::validateSEPCommand(), called from BootPolicyUserClient::extPerform().) I'd bet this means that all boot-related emulation of the SEP is implemented there. So it might be there that we should look for an explanation of the "SEP command 33" problem.

[steven-michaud]

I've found out what "SEP command 33" is -- _command_create_linked_manifest() in AppleVPBootPolicy.kext. There's an array named _command_functions[], and a pointer to _command_create_linked_manifest() exists in it at offset 33 X 8.

I've also found where _command_create_linked_manifest() fails, I think. My command of ARM64 assembler is a bit wobbly. But as best I can tell, _command_create_linked_manifest() only returns 7 (== BOOTPOLICY_ERROR_ACM) if its call to _validate_acm_context() fails.

So next I need to figure out why the call to _validate_acm_context() fails.

[steven-michaud]

I've discovered some information about why _validate_acm_context() fails. But it's very cryptic, and I can't really interpret it:

kcgend: (libbootpolicy.dylib) [com.apple.BootPolicy:Library] BootPolicy: bootpolicy_create_linked_manifest: entry
kernel: (AppleVPBootPolicy) BootPolicy: creating linked manifest
kernel: (AppleVPBootPolicy) BootPolicy: loading misc storage
kernel: (AppleVPCredentialManager) ACM: verifyPolicy: Verifying BootPolicy (V) on CS[100] (preflight=NO, secure=YES, checkKeybagUUID=NO).
kernel: (AppleVPCredentialManager) ACM: dumpVerificationStatus: BootPolicy (V) on CS[100] *NOT* SATISFIED: 01R -> R.
kcgend: (libbootpolicy.dylib) [com.apple.BootPolicy:Library] BootPolicy: SEP command 33 (v7) returned 7: bpbc 4177, 0, 0, 0
kcgend: (libbootpolicy.dylib) [com.apple.BootPolicy:Library] BootPolicy: assert: bpe == 0  (/AppleInternal/Library/BuildRoots/d23df31a-1fa6-11ed-91f3-e2c7c6032f02/Library/Caches/com.apple.xbs/Sources/BootPolicy/dylib/dylib.c:2558)
kcgend: (libbootpolicy.dylib) [com.apple.BootPolicy:Library] BootPolicy: bootpolicy_create_linked_manifest: exit: ACM (7)

There are many other cases where _validate_acm_context() succeeds, even in a VM, even while booting. Compare this, picked more or less at random:

syspolicyd: (libbootpolicy.dylib) [com.apple.BootPolicy:Library] BootPolicy: reading blessed macOS local policy /System/Volumes/iSCPreboot/BE2ACA7D-8B5E-4E89-88F2-EE05D5354D88/LocalPolicy/1A59E46C2D579AD45C6282DE32E7B3989AD35CB73D301FD7FFE22D9F11454C59683535F41C9822598EE74081553DC5BA.img4
kernel: (AppleVPBootPolicy) BootPolicy: updating local policy User Authorised Kext List
kernel: (AppleVPBootPolicy) BootPolicy: loading misc storage
kernel: (AppleVPCredentialManager) ACM: verifyPolicy: Verifying BootPolicy (V) on CS[118] (preflight=NO, secure=YES, checkKeybagUUID=NO).
kernel: (AppleVPCredentialManager) ACM: dumpVerificationStatus: BootPolicy (V) on CS[118] SATISFIED: 01S -> S.
kernel: (AppleVPBootPolicy) BootPolicy: Local policy signing
kernel: (AppleVPBootPolicy) BootPolicy: AP hacktivation signing
syspolicyd: (libbootpolicy.dylib) [com.apple.BootPolicy:Library] BootPolicy: SEP command 23 (v7) returned 0

For what it's worth, all the failures happen when the call to _validate_acm_context() is made (ultimately) from kcgend. Calls from syspolicyd, for example, work just fine. I noted above that I had to play a hackish trick to get kcgend to run at startup on a Virtualization framework VM. Maybe it's not working entirely correctly. I'll keep playing with it. But at this point I'm not very optimistic I'll be able to do anything about its problems.

"SEP command 23" is _command_update_local_policy_uakl() in AppleVPBootPolicy.kext.

Above I theorized that these failures are the result of a deliberate policy decision by someone at Apple. I still think that. But now I believe I focused too much on the "SEP command 33" failures themselves. They may be just accidental. Now I suspect that the crucial decision was to not allow kcgend to run in Virtualization framework VMs. I found a way to make it run. But (it seems) I haven't yet been able to make it run correctly.

My logs above came from the output (on a VM) of log show --predicate 'eventMessage contains "BootPolicy"'. A lot of the lines are missing from the output of kmutil log show. I'm discovering how much more useful log show is than the Console app's GUI. Previously I thought most of the output you see when you start streaming in the Console app is simply lost if you're not streaming. I couldn't have been more wrong.

Interestingly, I get no output from running log show --predicate 'eventMessage contains "hacktivation"' on my macOS host. I'm not entirely sure what to think about that.

[ktprograms]

I've found some potentially useful stuff. The AuxilaryStorage image appears to be an iBoot1 bootloader. According to https://github.com/AsahiLinux/docs/wiki/SW:Boot#stage-1-llbiboot1, it reads a local boot policy from /<volume-group-uuid>/LocalPolicy/<policy-hash>.img4, which contains (among other things) a smb2 key, described as bool: Third-party kernel extensions enabled.

According to that wiki page, it should be in "metadata keys", but although running strings on that file (in the VM) does have some of those keys (such as vuid and lobo), I can't find the other keys (like smb* or sip*).

I'm thinking of asking the how to get these metadata keys on the Asahi IRC, but I'm not sure which channel to use (as they don't have an offtopic channel). The closest I can think of is #asahi-alt - For discussion and support for unofficial third-party distro ports.

[steven-michaud]

Yes, @ktprograms, it'd be interesting to know the contents of the local boot policy file in a VM that's been set up to support loading third party kexts (while booted into the Recovery Partition). I'd bet the bool: Third-party kernel extensions enabled key is set to true, and that it makes no difference -- that "SEP command 33" still fails.

I've got some more ideas about how to make kcgend do what it should in a VM. I'll report back in the next few days whether or not they worked out.

They're based on two observations, one of which is a new discovery:

1. /System/Library/LaunchDaemons/com.apple.kcgend.plist has the following two keys. As best I can tell (from xnu kernel source code), osenvironment is stored in the "device tree" under /chosen. I want to figure out how to read the "device tree", and how to manipulate this property in it.

<key>RunAtLoad</key>
<true/>
<key>LimitLoadToHardware</key>
<dict>
    <key>osenvironment</key>
        <array>
            <string>kcgen</string>
        </array>
</dict>

2. There's a AppleARMWatchdogTimer.kext that loads on bare metal, but not in a VM (even sudo kmutil load -p /System/Library/Extensions/AppleARMWatchdogTimer.kext won't make it load). It's used to create a "watchdog monitor" on bare metal, which logs messages with the format kcgend: Watchdog ... (these appear in the output of kmutil log show). This doesn't happen in a VM. The messages from kernelmanagerd and kcgend are grouped differently (in kmutil log show output) on bare metal and in VMs. I wonder if the watchdog monitor is used to coordinate the actions of kernelmanagerd and kcgend. The problems with kcgendon VMs, where this monitor doesn't operate, may be due to these two daemons stomping on each other's work. With luck, this problem will be solved by figuring out how to manipulate the "device tree".

[steven-michaud]

The page at https://github.com/AsahiLinux/docs/wiki/SW:Boot is very interesting. I notice that it mentions (and at least partially defines) "AP" and "kcOS" -- both of which show up in my research (in logs and in kexts' machine code).

[steven-michaud]

I want to figure out how to read the "device tree", and how to manipulate this property in it.

Anyone know of a utility that can do this? It will probably need explicit Apple Silicon support.

A quick look on the web tells me I may need to write my own -- possibly as a "driver extension" (https://developer.apple.com/documentation/kernel/implementing_drivers_system_extensions_and_kexts).

Edit: I already tried sudo nvram boot-args="osenvironment=kcgen". That didn't work. It had no effect on kcgend, and didn't change the value of hw.osenvironment (as seen by the output of sysctl hw.osenvironment). And no, sudo sysctl can't be used to change the value of hw.osenvironment.

Edit: sudo nvram osenvironment="kcgen" doesn't work, either :-(

Edit: I also tried sudo nvram "chosen/osenvironment"="kcgen". That didn't work, either.

Edit: The ioreg utility can read the device tree -- though maybe not all of it. But I haven't yet found anything that can write to it. Actually, you need to use ioreg -p IODeviceTree to display the device tree, including its chosen branch.

Edit: The source code for Apple's ioreg utility is available -- https://github.com/apple-oss-distributions/IOKitTools/tree/IOKitTools-122. With luck, I can use that to figure out how to write the device tree, without having to mess with writing a driver extension. But it will still take me a while -- possibly a couple of weeks. In which case I probably won't be able to continue my debugging here until sometime next year.

[steven-michaud]

WRT reading the local boot policy: I think bputil can already do this. For example, here's the output of sudo bputil -d on one of my VMs. Notice the value of "3rd Party Kexts Status":

Current local policy:
OS environment:
OS Type                                       : macOS (overriden)
OS Pairing Status                             : Paired
Local Policy Nonce Hash                 (lpnh): 5FC3F6F1403AAEB8EB7694D3F091C82588F2AF5621CC431A4681623BE97BB7FF30AF7E8C5E98CDD1E445F35EB3775139
Remote Policy Nonce Hash                (rpnh): 9C1D68C5C7F10D6E57888625E86AD4DF2D33AE4EE91DBE2112B976D90AA34A474A22102D83C5C26F162D8D1E034B5F11
Recovery OS Policy Nonce Hash           (ronh): 54F5741CEBFC75CB7B01C02968C712A12328B907F470C85DC88D92305542E557264B12BCD72D9BBA148C3298FE10DB3D

Local policy:
Pairing Integrity                             : Valid
Signature Type                                : Other
Unique Chip ID                          (ECID): 0x4D2E2033F0060E0B
Board ID                                (BORD): 0x20
Chip ID                                 (CHIP): 0xFE00
Certificate Epoch                       (CEPO): 0x1
Security Domain                         (SDOM): 0x1
Production Status                       (CPRO): 1
Security Mode                           (CSEC): 1
OS Version                              (love): 21.7.217.0.0,0
Volume Group UUID                       (vuid): BE2ACA7D-8B5E-4E89-88F2-EE05D5354D88
KEK Group UUID                          (kuid): 00000000-0000-0000-0000-000000000000
Local Policy Nonce Hash                 (lpnh): 5FC3F6F1403AAEB8EB7694D3F091C82588F2AF5621CC431A4681623BE97BB7FF30AF7E8C5E98CDD1E445F35EB3775139
Remote Policy Nonce Hash                (rpnh): 9C1D68C5C7F10D6E57888625E86AD4DF2D33AE4EE91DBE2112B976D90AA34A474A22102D83C5C26F162D8D1E034B5F11
Next Stage Image4 Hash                  (nsih): 22954A85F4FDE9B179529AC61EA78C31F00B0E22DF9BE92235BBD80F548B7752FBA4C98D9B31DE9081EAC8EC5DEDCF2A
User Authorized Kext List Hash          (auxp): 0E3CE57A947E28CAADAAF29B9788D4740558B145F696AC594FB17F02CF284EF74B314AFA93A01DECBF7900373C2AC626
Auxiliary Kernel Cache Image4 Hash      (auxi): absent
Kext Receipt Hash                       (auxr): absent
CustomKC or fuOS Image4 Hash            (coih): absent
Security Mode:               Permissive (smb0 && smb1): 1
3rd Party Kexts Status:      Enabled    (smb2): 1
User-allowed MDM Control:    Enabled    (smb3): 1
DEP-allowed MDM Control:     Disabled   (smb4): absent
SIP Status:                  Customized (sip0): 7f
Signed System Volume Status: Disabled   (sip1): 1
Kernel CTRR Status:          Disabled   (sip2): 1
Boot Args Filtering Status:  Disabled   (sip3): 1

[steven-michaud]

Apple actually has pretty decent documentation on the local policy file:

https://support.apple.com/guide/security/contents-a-localpolicy-file-mac-apple-silicon-secc745a0845/web

[steven-michaud]

Here's a progress report ... or better yet a lack-of-progress report.

I pretty quickly discovered that the osenvironment stuff is a red herring: I found kernelmanagerd: kcgen activation settings: [kcgen enabled] messages on both the host and my VMs. This tells me osenvironment is already being set correctly (at the appropriate time during the boot process) in Virtualization framework VMs.

Then I tried to force AppleARMWatchdogTimer.kext to load in my VMs. Like I mentioned above, you can't just load it using kmutil load. In fact the boot kext cache seems to be immutable. You can only add kexts, even Apple-signed ones, to the auxiliary kext cache. But of course that's currently not possible in Virtualization framework VMs.

In principle you should be able to create a custom kext cache (using kmutil create) and load it (in recovery mode) using kmutil configure-boot. The following two links show you how to do this:

https://kernelshaman.blogspot.com/2021/02/building-xnu-for-macos-112-intel-apple.html https://github.com/AsahiLinux/macvdmtool/blob/main/README.md

But I couldn't get it to work. I tried adding AppleARMWatchdogTimer.kext to my custom kext cache, creating one with no additional kexts, and even loading the immutable kext cache (from Apple) as a custom kext cache. All failed in more or less the same way: I got a bunch of errors from AppleVPKeyStore.kext, then kernelmanagerd: Kernel requested shutdown. Goodbye!

I suspect Apple's kernel.release.vmapple (the kernel used in Virtualization framework VMs) just doesn't like what I'm trying to do. I could try building a custom kernel, following kernelshaman's instructions. But that would be very time consuming, and might just lead me down another rabbit hole.

I'm giving up on this, at least for the time being.

[dariaphoebe]

Since there was "Ask Apple" this week, I did. Quinn confirmed it's not going to work yet, and i'd consider that definitive.

[steven-michaud]

I could try building a custom kernel, following kernelshaman's instructions.

I did this, and managed to get it to work. But yes, it took me down another rabbit hole. I did discover one useful bit of information, though: The failures have nothing to do with the kernel itself (custom or otherwise). In fact the kernel never runs at all when you try to use kmutil configure-boot to load a custom kext cache. I found this out by putting an infinite loop in the custom kernel's code, very early in the boot process. It never gets hit.

This may be some kind of permissions issue (though I loosened them as much as possible). But I think it's more likely that code somewhere in kernelmanagerd simply stops it from working.

[ktprograms]

Could you test if the loop gets hit on physical hardware?

[steven-michaud]

It does get hit on physical hardware.

To play it safer, I changed the loop to a panic() call. Then my machine kept rebooting until I pressed the power button long enough to turn it off. I didn't see the normal panic screen -- probably the kernel hadn't yet loaded support for it. To restore my machine to its previous state, I booted into recovery mode and ran bputil -f. This deactivates the custom kernel and turns on full security.

So kernelmanagerd (or something called via kernelmanagerd) only prevents booting from a custom kernel in a Virtualization framework virtual machine. I'll dig into this further and try to find out exactly how.

I didn't do a full test of my custom kernel (and its custom kext cache) on physical hardware. I don't want to risk harming my machine. The logical place to do this kind of test is in a virtual machine. Clearly Apple has gone to some trouble to prevent tests of custom kernels and third party kernel extensions from working in a Virtualization framework VM. It's rather hard to understand why :-(

[ktprograms]

Thanks for testing that. I wonder if the VM's bootloader is somehow hardcoding "allowed" kernels. Do you think trying to boot a kernel with a different version that was used in another VM might work? (You could check with uname -a, I think).

Also, is kernelmanagerd used for creating bootable kernels, or does it do stuff at boot time? If it only creates the kernels, as long as it's "valid", I think it would be a bootloader problem.

[steven-michaud]

I doubt that kernelmanagerd hardcodes allowed kernels (though kmutil create (which works through kernelmanagerd) does insist (on macOS 13) that you have a KDK installed that matches the currently running kernel). Though if it does, it shouldn't be hard to find the code that implements this restriction.

kernelmanagerd doesn't create kernels. It works with whatever kernel is "installed" -- the one that the boot kext collection is linked to. And as best I can tell it also doesn't create boot kext collections. Apple makes those in some other way, then puts them in the "right" location (which you can see by running kmutil inspect). Once installed in this way, a boot kext collection is immutable, at least by the "normal" activity of macOS. What can be changed is the auxiliary kext collection. But, as we've discovered, this doesn't work in a Virtualization framework VM. And (on physical hardware) the boot kext collection can also be (temporarily) replaced by a custom kext collection.

From kernelmanagerd's man page and from what I've been able to discover, kernelmanagerd manages the kernel-specific part of rebooting -- from making requested (staged) changes to the auxiliary kext collection to triggering a kernel's reloading. As we've seen, it can abort this process and log the (misleading) error message kernelmanagerd: Kernel requested shutdown. Then it gives you the choice of trying again or booting into the recovery partition.

Over the next few days (and possibly weeks), I'm going to dig into kernelmanagerd and its friends, to find as much as I can about why they behave differently on physical hardware and in a Virtualization framework VM.