JEP 391 Investigate macOS AArch64 support

[0xdaryl]

This issue tracks the work to investigate an OpenJ9 AArch64 port to macOS.

Relevant OpenJDK JEP -> https://openjdk.java.net/jeps/391

[0xdaryl]

@knn-k

[0xdaryl]

This statement from the JEP will require some investigation:

macOS/AArch64 forbids memory segments from being executable and writeable at the same time, a policy known as write-xor-execute (W^X). The HotSpot VM routinely creates and modifies executable code, so this JEP will implement W^X support in HotSpot for macOS/AArch64.

Perhaps some form of read-only code cache may be required in this configuration.

[knn-k]

"Porting Just-In-Time Compilers to Apple Silicon" https://developer.apple.com/documentation/apple_silicon/porting_just-in-time_compilers_to_apple_silicon

[0xdaryl]

Sizing: 2 months of work

Initial OpenJ9 and OMR tasks:

• (2 pw) Build and configure work

  • Notarized macOS applications must enable the Hardened Runtime capability. For applications with JITs this further requires the enablement of the MAP_JIT entitlement on the main Java executable (not the shared libraries). See https://developer.apple.com/documentation/apple-silicon/porting-just-in-time-compilers-to-apple-silicon . This is mainly build work to figure out where/how to do this. @pshipton's comment below suggests this work may already be done in the ibmruntimes repo, but may require backporting to older JDK releases.
  • We will need to modify the configure process for distinguishing AArch64 macOS platform from Intel macOS platform. We will also need to add new files in openj9/runtime/cmake/caches/.

• (1 pw) XCode is moving to the Clang integrated assembler as the default assembler and is deprecating the GNU assembler. Clang assembly is expected to be syntax compatible with the GNU assembler, but I don't think that is strictly guaranteed. Ensure all JVM assembly code builds, links, and runs on Apple Silicon. @knn-k will verify on his personal M1 Mac with the XCode toolchain if there are any issues assembling our assembler files to give us an idea if more work will be involved here.

  • Update: @knn's initial sniff testing on macOS suggests that we'll have trouble using the .size and .type GNU as directives directly. These are used in the VM assembler code. More investigation needed.

• (1 pw) Review #ifdefs for OSX and AARCH64/ARM64. OSX guarded code should not necessarily assume x86 architecture. AARCH64/ARM64 guarded code should not assume Linux.

• (1 pw) Implement Apple variations to the ARM64 standard ABI for JIT SystemLinkage. See https://developer.apple.com/documentation/xcode/writing_arm64_code_for_apple_platforms

• (< 1 pw) Allocate executable code memory region on macOS with mmap MAP_JIT option. When you call mmap to create the memory region, the system initially configures it as readable and executable (R-X) for all threads.

• (< 1 pw) Add port library calls for pthread_jit_write_protect_supported_np and pthread_jit_write_protect_np, and sys_icache_invalidate

• (3 pw) Ensure appropriate W^X permissions are set on JIT executable pages. This involves identifying all places where dynamically generated code could be written to or executed by some thread, and ensure the appropriate protections are relaxed and restored. This will need to be called from the JIT and from the JIT runtime:

  • To write newly JITed code or patch existing code, the patching thread must first call pthread_jit_write_protect_np with the value false to disable JIT write protections for the memory region (thereby giving it RW- permission). The thread may then proceed to update executable code allocated with MAP_JIT.
  • Before that thread can execute any of the patched code it must call pthread_jit_write_protect_np with the value true to restore JIT write protection for the memory region (thereby giving it R-X permission). The thread should then call sys_icache_invalidate(_:_:) to invalidate the processor memory caches before executing any of the code.

[pshipton]

For applications with JITs this further requires the enablement of the MAP_JIT entitlement on the main Java executable

FYI https://github.com/ibmruntimes/openj9-openjdk-jdk/commit/e5ed92f29af6833890c1d511b5120e7e71bbf0c5

I see -f added in the code signing. There is a default entitlements file which didn't change but already contains allow-jit https://github.com/ibmruntimes/openj9-openjdk-jdk/blob/openj9/make/data/macosxsigning/default.plist#L5

Some of this will need to be backported in order to support older Java versions with OpenJ9.

[knn-k]

We will need to modify the configure process for distinguishing AArch64 macOS platform from Intel macOS platform. We will also need to add new files in openj9/runtime/cmake/caches/.

[knn-k]

I opened the following PRs as the initial effort:

• https://github.com/eclipse/omr/pull/6204
• https://github.com/eclipse-openj9/openj9/pull/13700
• https://github.com/ibmruntimes/openj9-openjdk-jdk17/pull/37

Current restrictions:

• JDK 17 only
• Interpreter mode only: JIT compiler is not enabled
• Large heap mode only: The VM fails to start in the compressed refs mode -- (Edit) See https://github.com/eclipse-openj9/openj9/issues/13767
• DDR is not enabled -- (Edit) Enabled now
• (Edit) Build with CMake only
• (Edit) omrosdump.c and omrsignal_context.c are incomplete

[knn-k]

Issue with building DDR: #13699

[knn-k]

The three PRs above can be merged to the repositories independently, without breaking builds of other platforms.

[knn-k]

Temporary build instructions

Install the following prerequisites on Apple Silicon macOS:

• Command line tools for Xcode
• HomeBrew (autoconf, bash, binutils, cmake, freetype, git, gnu-tar, pkg-config, wget)
• JDK 16 or 17 for macOS (AArch64 or x86-64)

Set the following environment variable:

% export OPENJ9_JAVA_OPTIONS=-Xnocompressedrefs

Run the "configure" script as follows:

% /opt/homebrew/bin/bash configure --with-boot-jdk=/your/path/to/jdk-17+35/Contents/Home --disable-warnings-as-errors

Build, and run the generated runtime:

% make all % cd build/macosx-aarch64-server-release/images % jdk/bin/java -version openjdk version "17.0.1-internal" 2021-10-19 OpenJDK Runtime Environment (build 17.0.1-internal+0-adhoc.openj9.openj9-openjdk-jdk17) Eclipse OpenJ9 VM (build master-475346f25, JRE 17 Mac OS X aarch64-64-Bit 20211022_000000 (JIT disabled, AOT disabled) OpenJ9 - 475346f25 OMR - 285f5f5d7 JCL - 733ee853c89 based on jdk-17.0.1+12)

Note: Configuring with --with-noncompressedrefs will fail with an compilation error: unused function 'isGCPolicyMetronome' in jvminit.c.

[knn-k]

I opened #13760 for the compilation error with isGCPolicyMetronome.

[knn-k]

PR #13760 has been merged, and now you can build the runtime without setting the environment variable OPENJ9_JAVA_OPTIONS. Run the "configure" script as:

% /opt/homebrew/bin/bash configure --with-boot-jdk=/your/path/to/jdk-17+35/Contents/Home --disable-warnings-as-errors --with-noncompressedrefs

Other related PRs:

• https://github.com/eclipse/omr/pull/6215
• https://github.com/ibmruntimes/openj9-openjdk-jdk17/pull/40
• https://github.com/ibmruntimes/openj9-openjdk-jdk/pull/369
• https://github.com/ibmruntimes/openj9-openjdk-jdk11/pull/450

[knn-k]

omrosdump.c needs to be changed for AArch64 macOS.

[knn-k]

Status update: JIT compiler has started to work in my local environment. Recompilation also works. I need to do some more work for AOT.

[knn-k]

PR for omrosdump.c: https://github.com/eclipse/omr/pull/6275

[knn-k]

PRs for JNILinkage:

• https://github.com/eclipse/omr/pull/6278
• #14194
• https://github.com/eclipse/omr/pull/6293 (SystemLinkage)

[knn-k]

PRs for loading function pointers from memory (already merged):

• https://github.com/eclipse/omr/pull/6311
• https://github.com/eclipse-openj9/openj9/pull/14296

[knn-k]

Changes for the JIT compiler (already merged):

• https://github.com/eclipse/omr/pull/6270
• https://github.com/eclipse-openj9/openj9/pull/14213
• https://github.com/eclipse-openj9/openj9/pull/14388
• https://github.com/eclipse-openj9/openj9/pull/14359
• https://github.com/ibmruntimes/openj9-openjdk-jdk18/pull/2

[knn-k]

An OMR PR related to macOS:

• https://github.com/eclipse/omr/pull/6267

[knn-k]

#13937 has been merged, and now you can build the runtime and run some tests with Jenkins jobs.

Current restrictions:

• You cannot build for JDK 8 yet
• Large heap mode only: The VM fails to start in the compressed refs mode -- #13767
• DDR is not enabled -- #14387
• Build with CMake only
• There are some problems found in functional tests

[knn-k]

JDK 8 build for macOS depends on the "JavaNativeFoundation" framework, which targets x86_64 only. I am not going to spend time for supporting JDK 8 with OpenJ9 for AArch64 macOS.

[knn-k]

Need to implement stack backtracing support for AArch64 macOS: https://github.com/eclipse/omr/issues/3506

[knn-k]

DDR was enabled last month.

Remaining items:

• Implement compressed refs mode
• Fix known problems
• (JDK 8: not in plan for the short term)

[knn-k]

I opened Issue #15925 - AArch64 macOS: Implement native stack backtrace

[knn-k]

PR #15907 fixed a number of known intermittent failures at VM termination.

Remaining items:

• [ ] Compressed refs mode (#13767)
• [ ] Native stack backtrace (Not specific to AArch64, #15925)
• [ ] COMMANDLINE and Environment variables (Not specific to AArch64, #16082)

[knn-k]

Issue #19501: UMA build is not supported on AArch64 macOS