NWJS on Apple Silicon

[gpetrov]

As the new Apple Silicon M1 based Macs are shipping now, we would really like to have an NWJS version that runs natively on it.

Seems Electron already have a full working version for some time now:

https://www.electronjs.org/blog/apple-silicon

@rogerwang could you please make also an Apple Silicon binary available for the Mac?

[gpetrov]

Or maybe offer a "fat" build running on both intel and apple silicon, just as google is doing: https://bugs.chromium.org/p/chromium/issues/detail?id=1142017

node-gyp might also need some improvements to be able to build fat native modules

[gpetrov]

Here are also the build instructions for Chrome on Apple Silicon @rogerwang so it should be easy to incorporate:

https://chromium.googlesource.com/chromium/src.git/+/master/docs/mac_arm64.md

[rogerwang]

Thanks for the information. Will support it soon.

[gpetrov]

Chrome just released official native version for the Apple Silicon M1 chip Seems to be running fast and smooth. They do packed it as Universal "fat" build - including both intel and apple silicon executables.

Only the crash handler is still on intel - don't know why.

NodeJS should compile also fine already.

So @rogerwang hope we get NWJS also there soon. Everybody is waiting.

[hthetiot]

Related https://github.com/LeonardLaszlo/nw.js-armv7-binaries

[sbashyal]

Thanks for the information. Will support it soon.

What kind of timeline would this be?

[rogerwang]

Please try this build: http://dl.nwjs.io/live-build/nwjs-sdk-v0.50.1-osx-arm64.zip

[gpetrov]

great news @rogerwang ! The base seems to be working but we have trouble compiling the native modules with nw-gyp even if specifying the new architecture, still got some headers mixture, not sure if nw-gyp needs an update as well? Or are we not using the std=c++14 somehow.

/Users/george/.nw-gyp/0.50.0/deps/v8/include/v8-internal.h:418:38: error: no
      template named 'remove_cv_t' in namespace 'std'; did you mean 'remove_cv'?
            !std::is_same<Data, std::remove_cv_t<T>>::value>::Perform(data);
                                ~~~~~^~~~~~~~~~~
                                     remove_cv
/Applications/Xcode.app/Contents/Developer/Toolchains/XcodeDefault.xctoolchain/usr/bin/../include/c++/v1/type_traits:697:50: note: 
      'remove_cv' declared here
template <class _Tp> struct _LIBCPP_TEMPLATE_VIS remove_cv/Users/george/.nw-gyp/0.50.0/deps/v8/include/v8-internal.h:418:38: error: no
      template named 'remove_cv_t' in namespace 'std'; did you mean 'remove_cv'?
            !std::is_same<Data, std::remove_cv_t<T>>::value>::Perform(data);
                                ~~~~~^~~~~~~~~~~
                                     remove_cv
/Applications/Xcode.app/Contents/Developer/Toolchains/XcodeDefault.xctoolchain/usr/bin/../include/c++/v1/type_traits:697:50: note: 
      'remove_cv' declared here
template <class _Tp> struct _LIBCPP_TEMPLATE_VIS remove_cv

[rogerwang]

It works for me with this test: https://github.com/nwjs/nw.js/tree/nw50/test/sanity/native-module-buffer

[gpetrov]

well I tried nodegit and node-pty and both got the same error, see also: https://github.com/nwjs/nw-gyp/issues/145

[gpetrov]

@rogerwang well managed to compile both nodegit and node-pty for the arm64 now, but on require in NWJS I get a hard crash with:

[1125/113332.123338:WARNING:process_memory_mac.cc(93)] mach_vm_read(0x16b384000, 0x8000): (os/kern) protection failure (2)

They seems to be compiled now well for the arm64:

file node_modules/node-pty/build/Release/pty.node   
node_modules/node-pty/build/Release/pty.node: Mach-O 64-bit bundle arm64

Any ideas?

[rogerwang]

try patch common.gypi in ~/.nw-gyp as:

diff --git a/common.gypi b/common.gypi
index 2fdb2c5b36..1241ddb45f 100644
--- a/common.gypi
+++ b/common.gypi
@@ -119,7 +119,7 @@
         'obj_dir%': '<(PRODUCT_DIR)/obj.target',
         #'v8_base': '<(PRODUCT_DIR)/obj.target/tools/v8_gypfiles/libv8_snapshot.a',
       }],
-      ['target_arch=="x64"', {
+      ['target_arch=="x64" or target_arch=="arm64"', {
         'v8_enable_pointer_compression': 1,
         'v8_enable_31bit_smis_on_64bit_arch': 1,
       }],
 

[gpetrov]

Thanks will try that! For some miraculous reason the new compiled node modules loaded just fine in the subsequent launch of NWJS (without recompiling with the patch)

I'm trying to compile more modules like sharp - but seem they all have to be adjusted for bad xcode_settings in binding.gyp

all modules have something like this:

"xcode_settings": {
            "GCC_ENABLE_CPP_EXCEPTIONS": "YES",
            "MACOSX_DEPLOYMENT_TARGET": "10.9",
            'CLANG_CXX_LIBRARY': 'libc++',
            'CLANG_CXX_LANGUAGE_STANDARD': 'c++11',

            "WARNING_CFLAGS": [
              "-Wno-unused-variable",
              "-Wint-conversions",
              "-Wmissing-field-initializers",
              "-Wno-c++11-extensions"
            ]
          }

while they should be using c++14 and SDK 11 for arm64 on the Mac with arm64 Can you fix that in nw-gyp? so it is done automatically? I think the newer node-gyp does that.

[gpetrov]

@rogerwang next to your patch I added also in common.gypi at about line 291 new settings for Xcode_settings:

'xcode_settings': {
          'GCC_OPTIMIZATION_LEVEL': '3', # stop gyp from defaulting to -Os
          'conditions': [
            ['target_arch=="arm64"', {
              'CLANG_CXX_LANGUAGE_STANDARD': 'c++14',
              "OTHER_CPLUSPLUSFLAGS": [
                "-std=c++14",
                "-stdlib=libc++"
              ],
            }]
          ],
        },

I'm happy to report all native modules compile now fine on Mac M1 arm64! Congratulations!

[gpetrov]

@rogerwang still some modules like node-pty are crashing NWJS with: [1125/173514.526302:WARNING:process_memory_mac.cc(93)] mach_vm_read(0x16fa5c000, 0x8000): (os/kern) protection failure (2)

while all the patches are applied in common.gypi here are some dump files:

Archive.zip

Hope you can solve it and release the NWJS Mac arm64 build as official. Please do also include all debug symbols in the official Mac version as requested in: https://github.com/nwjs/nw.js/issues/7340

[gpetrov]

hmm seems this is more related to the requirements from Apple to have all native modules code signed .... after running:

/usr/bin/xcrun codesign --force --sign - --preserve-metadata=identifier,entitlements node_modules/node-pty/build/Release/pty.node

no more crashes occur for node-pty!

@rogerwang Maybe we should integrate default code signing in each nw-gyp native node module compilation, otherwise if not a crash - a nasty popup appears that the node module is not verified and can only be moved to the trash can. This is very annoying during development/

Of course when we package the final app we sign every binary file officially again.

[gpetrov]

As for the code signing @rogerwang , see https://github.com/golang/go/issues/38485#issuecomment-734927461 and the linked topic https://github.com/golang/go/issues/42684#issuecomment-731704900 for more info, internals and bypass. Might be useful for native node modules compilation flow.

[gpetrov]

any progress on the official build @rogerwang ? We are good to go!

[gpetrov]

Any news @rogerwang - we really like to get an official build, everything to run smoothly on the new M1.

The MacOS Homebrew is already up and running fine natively on M1 with almost all software available asp rebuild binaries "bottles"

So we really hope to see the official NWJS build for the M1 soon!

[corwin-of-amber]

Cheers @rogerwang — your build runs smoothly on my MacBook Air M1 and I was even able to build native modules with plain node-gyp.

I am trying to help with pushing this release forward, so far I have been struggling with the build instructions; I started with a Linux build, which I hoped would be more manageable, but I got stuck with an unmet dependency out/nw/libvulkan.so when generating the ninja files. Are the build instructions up-to-date?

[corwin-of-amber]

I got stuck with an unmet dependency out/nw/libvulkan.so when generating the ninja files. Are the build instructions up-to-date?

I managed to get past that (the instructions were a tad ambiguous, took me a while to realize that I need both gclient sync --with_branch_heads --nohooks and gclient sync --with_branch_heads, which in retrospect does make sense). Now I am facing some missing symbols in v8.

I was syncing with https://github.com/nwjs/chromium.src.git@origin/nw50 (I replaced the 17 from the instructions with 50), is that the right one to sync with? Also, when I manually cloned nwjs/nw.js, nwjs/node, nwjs/v8, I used master for all. Should I have used a tag?

ld.lld: error: undefined symbol: V8_Dcheck(char const*, int, char const*)
ld.lld: error: undefined symbol: v8::internal::Builtins::KindOf(int)
ld.lld: error: undefined symbol: V8_Fatal(char const*, int, char const*, ...)
ld.lld: error: undefined symbol: vtable for v8::internal::ArrayNArgumentsConstructorDescriptor
>>> the vtable symbol may be undefined because the class is missing its key function (see https://lld.llvm.org/missingkeyfunction)
ld.lld: error: undefined symbol: v8::internal::GetSequenceIndexFromFastElementsKind(v8::internal::ElementsKind)
ld.lld: error: undefined symbol: v8::internal::GetFastElementsKindFromSequenceIndex(int)
ld.lld: error: undefined symbol: std::__1::enable_if<(!(std::is_function<std::__1::remove_pointer<unsigned long>::type>::value)) && (has_output_operator<unsigned long>::value), std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > >::type v8::base::PrintCheckOperand<unsigned long>(unsigned long)
ld.lld: error: undefined symbol: std::__1::enable_if<(!(std::is_function<std::__1::remove_pointer<int>::type>::value)) && (has_output_operator<int>::value), std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > >::type v8::base::PrintCheckOperand<int>(int)
ld.lld: error: undefined symbol: v8::internal::Builtins::Call(v8::internal::ConvertReceiverMode)
ld.lld: error: undefined symbol: v8::internal::Builtins::CallFunction(v8::internal::ConvertReceiverMode)
ld.lld: error: undefined symbol: v8::internal::DebugCodegen::GenerateFrameDropperTrampoline(v8::internal::MacroAssembler*)
ld.lld: error: undefined symbol: v8::internal::DebugCodegen::GenerateHandleDebuggerStatement(v8::internal::MacroAssembler*)
ld.lld: error: undefined symbol: v8::internal::Marking::kWhiteBitPattern
ld.lld: error: undefined symbol: std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> >* v8::base::MakeCheckOpString<int, int>(int, int, char const*)
ld.lld: error: undefined symbol: vtable for v8::internal::CEntry1ArgvOnStackDescriptor
ld.lld: error: undefined symbol: v8::internal::ExternalReference::Create(v8::internal::IsolateAddressId, v8::internal::Isolate*)
ld.lld: error: undefined symbol: v8::internal::HandleScopeImplementer::kEnteredContextsOffset
ld.lld: error: undefined symbol: v8::internal::HandleScopeImplementer::kIsMicrotaskContextOffset
ld.lld: error: undefined symbol: v8::internal::ExternalReference v8::internal::ExternalReference::search_string_raw<unsigned char const, unsigned char const>()
ld.lld: error: undefined symbol: v8::internal::ExternalReference v8::internal::ExternalReference::search_string_raw<unsigned char const, unsigned short const>()

[corwin-of-amber]

Now I am facing some missing symbols in v8.

I rebuilt everything and now it links — HOLY SNACKS! This was just the Linux build, sorry for spamming this issue thread too much, I will have to get my hands on a fast enough Intel Mac connected to a fast enough network (according to the Chromium docs, building for ARM64 is only supported on Intel Macs, I wonder why that is).

[rogerwang]

For general rebuild questions please use our mailing list.

On Sun, Jan 3, 2021, 2:22 AM Shachar Itzhaky [email protected] wrote:

Now I am facing some missing symbols in v8.

I rebuilt everything and now it links — HOLY SNACKS! This was just the Linux build, sorry for spamming this issue thread too much, I will have to get my hands on a fast enough Intel Mac connected to a fast enough network (according to the Chromium docs, building for ARM64 is only supported on Intel Macs, I wonder why that is).

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub https://github.com/nwjs/nw.js/issues/7620#issuecomment-753511921, or unsubscribe https://github.com/notifications/unsubscribe-auth/AABIMGKDK2OKU5IMUIDPQ5LSX5P7PANCNFSM4TVN6M6Q .

[rogerwang]

That's pending on build worker system upgrade. Once that's done a M1 build worker will be setup.

On Tue, Dec 29, 2020, 2:51 AM George Petrov [email protected] wrote:

Any news @rogerwang https://github.com/rogerwang - we really like to get an official build, everything to run smoothly on the new M1.

The MacOS Homebrew is already up and running fine natively on M1 with almost all software available asp rebuild binaries "bottles"

So we really hope to see the official NWJS build for the M1 soon!

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub https://github.com/nwjs/nw.js/issues/7620#issuecomment-751827148, or unsubscribe https://github.com/notifications/unsubscribe-auth/AABIMGK46YW2CKXGGKD5BO3SXDHRLANCNFSM4TVN6M6Q .

[corwin-of-amber]

Re: the experimental 0.50.1 you posted earlier — I discovered that it crashes when trying to load a WebAssembly. This happened when the binary is streamed (i.e. loaded via a URL, rather than given as Uint8Array). I haven't had a chance to test other scenarios yet.

[gpetrov]

@rogerwang you do know that you can build mac arm on intel, so infrastructure change is not really needed. Google Chromium describes it well: https://chromium.googlesource.com/chromium/src/+/master/docs/mac_arm64.md or is your mac build worker just outdated and slow?

[gpetrov]

Any news @rogerwang ?

Another month passed and all major distributions already have a nice stable M1 native compilation. Hope you can include soon the Native M1 NWJS in the official distribution.

[rogerwang]

Thanks @gpetrov . As I described earlier, it's pending on the build worker system upgrade. If someone would contribute a new mac system to the project, that would definitely make this happen sooner.

[gpetrov]

but @rogerwang you don't need a new M1 mac to compile - it works just fine on intel macs, see https://chromium.googlesource.com/chromium/src/+/master/docs/mac_arm64.md

[rogerwang]

@gpetrov your point is well known here but the old intel macOS system in the build infrastructure doesn't run the latest Xcode, which is required by the port.