asmjit+macos+x86: ephemeral "Verifying shm-id-XXXXXXXXXXXXXXXX" security popups appear during each invocation of erlang

[zerth]

Describe the bug Ephemeral "Verifying shm-id-XXXXXXXXXXXXXXXX" security popups on x86 macos sonoma (14.2.1) appear due to use of asmjit:

Opening this issue here because I'm not sure if it's an issue with how otp is configuring/using asmjit on this platform or something upstream.

Recompiling with --disable-jit to disable use of asmjit resolves the issue.

To Reproduce

while true; do
    erl -noinput -eval "io:format(\".\"),init:stop()."
done

(reproduction consists of security popups as in the above image repeatedly appearing for ~0.03s during each invocation)

Expected behavior No security popups appear.

Affected versions Tested with OTP-26.0.

Additional context Others using erlang also seem to be affected.

[jhogberg]

Unfortunately there's not much we can do about this other than to have the JIT disabled by default on that configuration, and/or force people to (partially) disable the hardened runtime.

The gist of it is that x86 macs lack a certain feature that lets us safely toggle pages between being writable and executable (pthread_jit_write_protect_np), forcing us to rely on a technique known as dual-mapping where we have two different views into memory, one executable and one writable.

There's nothing wrong with that per se, but the only way to do that on a mac is to literally create a file and mmap(2) it twice, which I'm guessing causes the popups. :-(

[RobinMorisset]

JavaScriptCore appears to be doing it slightly differently, with a call to mach_vm_remap (https://github.com/WebKit/WebKit/blob/1e4b445227ae8d5e3c9ef8d13d0acd90e1c1366b/Source/JavaScriptCore/jit/ExecutableAllocator.cpp#L324), and I don't remember it having these pop-ups. Is there a reason this function cannot be used by asmjit?

[kobalicek]

@RobinMorisset That's a great find!

I think the only reason it's not used at the moment is that I didn't even know it existed. I will take a look into that, but I'm not sure I can reliably test this as I don't have x86 mac.

Interesting docs:

• https://developer.apple.com/documentation/kernel/1402218-mach_vm_remap

[navinpeiris]

FWIW, If you're using asdf, you can workaround this issue by disabling jit by adding: export KERL_CONFIGURE_OPTIONS="--disable-jit" to your .bashrc/.zshrc before installing erlang.

Not ideal, but so much better than the popup stealing focus every few minutes for me.

[kobalicek]

I created a branch in asmjit repo, which blindly implements the support - I used WebKit as a documentation in this case.

If anyone wants to try it (you would have to compile erlang with this asmjit branch):

• https://github.com/asmjit/asmjit/tree/macos_dual_mapping

Tests passed, but I have no idea whether it works on a real x86 mac (I don't have any).

[richcarl]

@kobalicek Tried it on latest OTP master on my Intel Mac (Sonoma 14.2.1) . Compiled, but failed with a Bus Error as soon as it runs the executable during bootstrap compilation. The asmjit master branch works though, so it's only that commit that fails. @jhogberg Do you think you guys could help debug this soon-ish?

[kobalicek]

I think there may be missing MemoryFlags::kMapShared on this line (function allocDualMappingUsingMachVmRemap):

ASMJIT_PROPAGATE(mapMemory(&rxPtr, size, MemoryFlags::kAccessReadWrite));

Should be:

ASMJIT_PROPAGATE(mapMemory(&rxPtr, size, MemoryFlags::kAccessReadWrite | MemoryFlags::kMapShared));

I think this needs more tests on CI, I will look into this in the evening. Or could you change that one line and report back whether that was the issue?

[richcarl]

Tried changing it, but same Bus Error.

[kobalicek]

I have looked into it again and updated the branch. I actually borrowed a real x86 mac to try it and it worked on that machine. Wondering whether you can retry?

[jvdvleuten]

How would you compile erlang with the new branch in ASMJIT? I can compile erlang with other flags (the --disable-jit flag solves my problem for now), but am not sure on how to use your new branch?

[jhogberg]

asmjit is vendored into the source tree for the time being, the easiest way is to grab the branch and copy the contents of its src folder into erts/emulator/asmjit.

[kobalicek]

A fix has been merged to asmjit:

https://github.com/asmjit/asmjit/commit/a63d41e80ba72f652c450d3bd6614327cde8531c

It now uses the mentioned mach_vm_remap to create dual mapping.

[dimitarvp]

Awesome. It would be great if we can get an Erlang patch release after the asmjit change is upstreamed.

[richcarl]

Confirmed working!

[richcarl]

Awesome. It would be great if we can get an Erlang patch release after the asmjit change is upstreamed.

Yeah, there have been some changes to asmjit's API recently, and the current OTP master is up to date and works with this fix, but I don't know how to backport the changes to OTP 26.

[jhogberg]

We’re unlikely to backport these changes to 26. We will include them in 27 to see how things go, but this functionality is undocumented and not part of the public API so it doesn’t feel right to change it in a patch release.

Even if annoying, the old way is the only supported way to do this as far as we’re aware.

[richcarl]

For now then, maybe disable jit by default on Mac OS Intel.

[jhogberg]

Yes, that's the plan. :)

[zerth]

🙌 Confirming that the upstream fix resolves the issue for me on master; thanks for the quick responses and fix!

[kobalicek]

@jhogberg

I would probably not worry about it being undocumented as I have found the use of the function in other projects, including DoxBox and V8:

https://github.com/v8/v8/blob/main/src/base/platform/platform-darwin.cc#L198

So, most likely it's not going to disappear. But I agree that I would be happier if it was documented by Apple, because it still feels like a hack.