runner-images macOS arm64 runner doesn't allow access to system registers

Description

Hello,

Thanks for offering the precious macOS arm runners! However, as suggested in many places, the runners are virtualized with Apple Virtualization Framework and thus run jobs under EL1. This doesn't allow users to access many system registers, including SPRR. Our use case is to use this register to determine the current JIT state, as Apple doesn't offer an API to do so. This works pretty well on bare metal machines as SPRR is accessible.

From the post here, probably Github Action needs to add com.apple.private.hypervisor.vmapple entitlement to the hypervisor to enable the access. I understand this use case might be minor but I would appreciate if Github Action can add this so that we can test and distribute pre-built python wheels.

Platforms affected

[ ] Azure DevOps
[X] GitHub Actions - Standard Runners
[X] GitHub Actions - Larger Runners

Runner images affected

[ ] Ubuntu 20.04
[ ] Ubuntu 22.04
[ ] Ubuntu 24.04
[ ] macOS 12
[ ] macOS 13
[ ] macOS 13 Arm64
[ ] macOS 14
[X] macOS 14 Arm64
[ ] macOS 15
[X] macOS 15 Arm64
[ ] Windows Server 2019
[ ] Windows Server 2022

Image version and build link

Image: macos-14-arm64 Version: 20241202.580 Included Software: https://github.com/actions/runner-images/blob/macos-14-arm64/20241202.580/images/macos/macos-14-arm64-Readme.md Image Release: https://github.com/actions/runner-images/releases/tag/macos-14-arm64%2F20241202.580

Is it regression?

No

Expected behavior

Allow acess to system registers.

Actual behavior

Receive Illegal instruction: 4

Repro steps

We have a code snippet to reproduce this:

#include "stdint.h"
#include "stdio.h"
#include "pthread.h"
static inline uint64_t read_sprr_perm_mrs(void)
{
    uint64_t v;
    __asm__ __volatile__("isb sy\n"
                         "mrs %0, S3_6_c15_c1_5\n"
                         : "=r"(v)::"memory");
    return v;
}
static void print_all() {
    uint64_t v1 = read_sprr_perm_mrs();

    fprintf(stderr, "SPRR=%llx\n", v1);
}

int main() {
     print_all();
    fprintf(stderr, "Enable\n");
    pthread_jit_write_protect_np(1);
    print_all();
    fprintf(stderr, "Disable\n");
    pthread_jit_write_protect_np(0);
    print_all();
    fprintf(stderr, "Enable\n");
    pthread_jit_write_protect_np(1);
    print_all();
    return 0;
}

Run it by:

cc test.c --o ./test
./test

This works well on bare metal machines while crashes on Github Action runners.

Dec 08 '24 06:12 wtdcode

As a side note, the JIT protection is not enabled at all on the macos arm64 runners. For instance, we can use the following code to confirm:

#include "sys/mman.h"
#include "pthread.h"
#include "stdio.h"
#include "stdlib.h"
int main() {
        uint64_t* p = mmap(NULL, 0x8192, PROT_WRITE | PROT_READ | PROT_EXEC, MAP_JIT | MAP_PRIVATE | MAP_ANONYMOUS, -1, 0);
        if (p == MAP_FAILED) {
                perror("allocate");
                return -1;
        }

        pthread_jit_write_protect_np(0);
        p[0] = 0xff;
        fprintf(stderr, "Initial %llx\n", p[0]);
        pthread_jit_write_protect_np(1);
        p[1] = 0xee; // This shall be denied
        fprintf(stderr, "Enable write protection %llx\n", p[1]);
        return 0;
}

On my m2 pro, this gives:

Initial ff
fish: Job 1, './test' terminated by signal SIGBUS (Misaligned address error)

But on Github Action, this gives:

Initial ff
Enable write protection ee

Dec 08 '24 06:12 wtdcode

HI @wtdcode , We will look into the issue and keep you posted with updates.

Dec 09 '24 05:12 prasanjitsahoo

Hey guys, can I have some updates? @prasanjitsahoo @erik-bershel

Dec 19 '24 16:12 wtdcode

no. the hypervisor already has com.apple.private.hypervisor because the hypervisor is in a system process whose executable is in the virtualization framework. (/System/Library/Frameworks/Virtualization.framework/XPCServices/com.apple.Virtualization.VirtualMachine.xpc/Contents/MacOS/com.apple.Virtualization.VirtualMachine). The vmapple kernel simply does not support the sprr and gxf features. The only easy-ish way to run virtualized a kernel that supports SPRR and GXF is to run a PCC research guest on macOS 15.1 and later, but that's a different environment altogether.

TL;DR: This issue can't be fixed for as long as a virtual machine is used, unless Apple changes how the virtual machines work in the future.

Dec 23 '24 03:12 asdfugil

no. the hypervisor already has com.apple.private.hypervisor because the hypervisor is in a system process whose executable is in the virtualization framework. (/System/Library/Frameworks/Virtualization.framework/XPCServices/com.apple.Virtualization.VirtualMachine.xpc/Contents/MacOS/com.apple.Virtualization.VirtualMachine). The vmapple kernel simply does not support the sprr and gxf features. The only easy-ish way to run virtualized a kernel that supports SPRR and GXF is to run a PCC research guest on macOS 15.1 and later, but that's a different environment altogether.

TL;DR: This issue can't be fixed for as long as a virtual machine is used, unless Apple changes how the virtual machines work in the future.

How do you confirm the entitlement is there?

Dec 23 '24 03:12 wtdcode

How do you confirm the entitlement is there?

codesign -d --entitlements - /System/Library/Frameworks/Virtualization.framework/XPCServices/com.apple.Virtualization.VirtualMachine.xpc/Contents/MacOS/com.apple.Virtualization.VirtualMachine | grep com.apple.private.hypervisor

Dec 23 '24 03:12 asdfugil

How do you confirm the entitlement is there?

codesign -d --entitlements - /System/Library/Frameworks/Virtualization.framework/XPCServices/com.apple.Virtualization.VirtualMachine.xpc/Contents/MacOS/com.apple.Virtualization.VirtualMachine | grep com.apple.private.hypervisor

I think the entitlement I'm referring is com.apple.private.hypervisor.vmapple, does com.apple.private.hypervisor imply that?

Dec 23 '24 03:12 wtdcode

I think the entitlement I'm referring is com.apple.private.hypervisor.vmapple, does com.apple.private.hypervisor imply that?

com.apple.security.hypervisor and com.apple.vm.hypervisor are the most restrictive entitlements com.apple.private.hypervisor.vmapple is more permissive com.apple.private.hypervisor is the most permissive

Dec 23 '24 04:12 asdfugil

How do you confirm the entitlement is there?

codesign -d --entitlements - /System/Library/Frameworks/Virtualization.framework/XPCServices/com.apple.Virtualization.VirtualMachine.xpc/Contents/MacOS/com.apple.Virtualization.VirtualMachine | grep com.apple.private.hypervisor

I think the entitlement I'm referring is com.apple.private.hypervisor.vmapple, does com.apple.private.hypervisor imply that?

Thank you for raising this issue. As mentioned, the restrictions on macOS arm64 runners related to system registers stem from entitlement limitations (com.apple.security.hypervisor and com.apple.vm.hypervisor). Unfortunately, these constraints make it infeasible for us to provide prebuilt wheels on arm64 macOS. If you require further assistance or encounter related challenges, please feel free to reopen the issue or reach out to us.

Dec 31 '24 05:12 prasanjitsahoo