NWJS on Apple Silicon

[gpetrov]

As the new Apple Silicon M1 based Macs are shipping now, we would really like to have an NWJS version that runs natively on it.

Seems Electron already have a full working version for some time now:

https://www.electronjs.org/blog/apple-silicon

@rogerwang could you please make also an Apple Silicon binary available for the Mac?

[gpetrov]

Or maybe offer a "fat" build running on both intel and apple silicon, just as google is doing:
https://bugs.chromium.org/p/chromium/issues/detail?id=1142017

node-gyp might also need some improvements to be able to build fat native modules

[gpetrov]

Here are also the build instructions for Chrome on Apple Silicon @rogerwang so it should be easy to incorporate:

https://chromium.googlesource.com/chromium/src.git/+/master/docs/mac_arm64.md

[rogerwang]

Thanks for the information. Will support it soon.

[gpetrov]

Chrome just released official native version for the Apple Silicon M1 chip
Seems to be running fast and smooth.
They do packed it as Universal "fat" build - including both intel and apple silicon executables.

Only the crash handler is still on intel - don't know why.

NodeJS should compile also fine already.

So @rogerwang hope we get NWJS also there soon. Everybody is waiting.

[hthetiot]

Related https://github.com/LeonardLaszlo/nw.js-armv7-binaries

[sbashyal]

Thanks for the information. Will support it soon.

What kind of timeline would this be?

[rogerwang]

Please try this build: http://dl.nwjs.io/live-build/nwjs-sdk-v0.50.1-osx-arm64.zip

[gpetrov]

great news @rogerwang ! The base seems to be working but we have trouble compiling the native modules with nw-gyp
even if specifying the new architecture, still got some headers mixture, not sure if nw-gyp needs an update as well? Or are we not using the std=c++14 somehow.

/Users/george/.nw-gyp/0.50.0/deps/v8/include/v8-internal.h:418:38: error: no
      template named 'remove_cv_t' in namespace 'std'; did you mean 'remove_cv'?
            !std::is_same<Data, std::remove_cv_t<T>>::value>::Perform(data);
                                ~~~~~^~~~~~~~~~~
                                     remove_cv
/Applications/Xcode.app/Contents/Developer/Toolchains/XcodeDefault.xctoolchain/usr/bin/../include/c++/v1/type_traits:697:50: note: 
      'remove_cv' declared here
template <class _Tp> struct _LIBCPP_TEMPLATE_VIS remove_cv/Users/george/.nw-gyp/0.50.0/deps/v8/include/v8-internal.h:418:38: error: no
      template named 'remove_cv_t' in namespace 'std'; did you mean 'remove_cv'?
            !std::is_same<Data, std::remove_cv_t<T>>::value>::Perform(data);
                                ~~~~~^~~~~~~~~~~
                                     remove_cv
/Applications/Xcode.app/Contents/Developer/Toolchains/XcodeDefault.xctoolchain/usr/bin/../include/c++/v1/type_traits:697:50: note: 
      'remove_cv' declared here
template <class _Tp> struct _LIBCPP_TEMPLATE_VIS remove_cv

[rogerwang]

It works for me with this test: https://github.com/nwjs/nw.js/tree/nw50/test/sanity/native-module-buffer

[gpetrov]

well I tried nodegit and node-pty and both got the same error, see also:
nwjs/nw-gyp#145

[gpetrov]

@rogerwang well managed to compile both nodegit and node-pty for the arm64 now, but on require in NWJS I get a hard crash with:

[1125/113332.123338:WARNING:process_memory_mac.cc(93)] mach_vm_read(0x16b384000, 0x8000): (os/kern) protection failure (2)

They seems to be compiled now well for the arm64:

file node_modules/node-pty/build/Release/pty.node   
node_modules/node-pty/build/Release/pty.node: Mach-O 64-bit bundle arm64

Any ideas?

[rogerwang]

try patch common.gypi in ~/.nw-gyp as:

diff --git a/common.gypi b/common.gypi
index 2fdb2c5b36..1241ddb45f 100644
--- a/common.gypi
+++ b/common.gypi
@@ -119,7 +119,7 @@
         'obj_dir%': '<(PRODUCT_DIR)/obj.target',
         #'v8_base': '<(PRODUCT_DIR)/obj.target/tools/v8_gypfiles/libv8_snapshot.a',
       }],
-      ['target_arch=="x64"', {
+      ['target_arch=="x64" or target_arch=="arm64"', {
         'v8_enable_pointer_compression': 1,
         'v8_enable_31bit_smis_on_64bit_arch': 1,
       }],
 

[gpetrov]

Thanks will try that! For some miraculous reason the new compiled node modules loaded just fine in the subsequent
launch of NWJS (without recompiling with the patch)

I'm trying to compile more modules like sharp - but seem they all have to be adjusted for bad xcode_settings in binding.gyp

all modules have something like this:

"xcode_settings": {
            "GCC_ENABLE_CPP_EXCEPTIONS": "YES",
            "MACOSX_DEPLOYMENT_TARGET": "10.9",
            'CLANG_CXX_LIBRARY': 'libc++',
            'CLANG_CXX_LANGUAGE_STANDARD': 'c++11',

            "WARNING_CFLAGS": [
              "-Wno-unused-variable",
              "-Wint-conversions",
              "-Wmissing-field-initializers",
              "-Wno-c++11-extensions"
            ]
          }

while they should be using c++14 and SDK 11 for arm64 on the Mac with arm64
Can you fix that in nw-gyp? so it is done automatically? I think the newer node-gyp does that.

[gpetrov]

@rogerwang next to your patch I added also in common.gypi at about line 291 new settings for Xcode_settings:

'xcode_settings': {
          'GCC_OPTIMIZATION_LEVEL': '3', # stop gyp from defaulting to -Os
          'conditions': [
            ['target_arch=="arm64"', {
              'CLANG_CXX_LANGUAGE_STANDARD': 'c++14',
              "OTHER_CPLUSPLUSFLAGS": [
                "-std=c++14",
                "-stdlib=libc++"
              ],
            }]
          ],
        },

I'm happy to report all native modules compile now fine on Mac M1 arm64! Congratulations!

[gpetrov]

@rogerwang still some modules like node-pty are crashing NWJS with:
[1125/173514.526302:WARNING:process_memory_mac.cc(93)] mach_vm_read(0x16fa5c000, 0x8000): (os/kern) protection failure (2)

while all the patches are applied in common.gypi
here are some dump files:

Archive.zip

Hope you can solve it and release the NWJS Mac arm64 build as official.
Please do also include all debug symbols in the official Mac version as requested in: #7340

[gpetrov]

hmm seems this is more related to the requirements from Apple to have all native modules code signed ....
after running:

/usr/bin/xcrun codesign --force --sign - --preserve-metadata=identifier,entitlements node_modules/node-pty/build/Release/pty.node

no more crashes occur for node-pty!

@rogerwang Maybe we should integrate default code signing in each nw-gyp native node module compilation, otherwise if not a crash - a nasty popup appears that the node module is not verified and can only be moved to the trash can. This is very annoying during development/

Of course when we package the final app we sign every binary file officially again.

[gpetrov]

As for the code signing @rogerwang , see golang/go#38485 (comment) and the linked topic golang/go#42684 (comment) for more info, internals and bypass. Might be useful for native node modules compilation flow.

[gpetrov]

any progress on the official build @rogerwang ? We are good to go!

[gpetrov]

Any news @rogerwang - we really like to get an official build, everything to run smoothly on the new M1.

The MacOS Homebrew is already up and running fine natively on M1 with almost all software available asp rebuild binaries "bottles"

So we really hope to see the official NWJS build for the M1 soon!

[corwin-of-amber]

Cheers @rogerwang — your build runs smoothly on my MacBook Air M1 and I was even able to build native modules with plain node-gyp.

I am trying to help with pushing this release forward, so far I have been struggling with the build instructions; I started with a Linux build, which I hoped would be more manageable, but I got stuck with an unmet dependency out/nw/libvulkan.so when generating the ninja files. Are the build instructions up-to-date?

[corwin-of-amber]

Now I am facing some missing symbols in v8.

I rebuilt everything and now it links — HOLY SNACKS!
This was just the Linux build, sorry for spamming this issue thread too much, I will have to get my hands on a fast enough Intel Mac connected to a fast enough network (according to the Chromium docs, building for ARM64 is only supported on Intel Macs, I wonder why that is).

[rogerwang]

For general rebuild questions please use our mailing list.

…

On Sun, Jan 3, 2021, 2:22 AM Shachar Itzhaky ***@***.***> wrote: Now I am facing some missing symbols in v8. I rebuilt everything and now it links — HOLY SNACKS! This was just the Linux build, sorry for spamming this issue thread too much, I will have to get my hands on a fast enough Intel Mac connected to a fast enough network (according to the Chromium docs, building for ARM64 is only supported on Intel Macs, I wonder why that is). — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub <#7620 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AABIMGKDK2OKU5IMUIDPQ5LSX5P7PANCNFSM4TVN6M6Q> .

[rogerwang]

That's pending on build worker system upgrade. Once that's done a M1 build worker will be setup.

…

On Tue, Dec 29, 2020, 2:51 AM George Petrov ***@***.***> wrote: Any news @rogerwang <https://github.com/rogerwang> - we really like to get an official build, everything to run smoothly on the new M1. The MacOS Homebrew is already up and running fine natively on M1 with almost all software available asp rebuild binaries "bottles" So we really hope to see the official NWJS build for the M1 soon! — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub <#7620 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AABIMGK46YW2CKXGGKD5BO3SXDHRLANCNFSM4TVN6M6Q> .

[corwin-of-amber]

Re: the experimental 0.50.1 you posted earlier — I discovered that it crashes when trying to load a WebAssembly. This happened when the binary is streamed (i.e. loaded via a URL, rather than given as Uint8Array). I haven't had a chance to test other scenarios yet.

[gpetrov]

@rogerwang you do know that you can build mac arm on intel, so infrastructure change is not really needed. Google Chromium describes it well: https://chromium.googlesource.com/chromium/src/+/master/docs/mac_arm64.md
or is your mac build worker just outdated and slow?

[zkrige]

Is there any update? @dpage kindly provided a Mac to build on more than 3 months ago, and there has been no feedback since then? We have a number of clients begging for apple silicon version of our app and we're sitting with egg on our face because we're unable to help them.

I'd be more than happy to help if I'm able to - please just let me know what you need to have done

[corwin-of-amber]

@zkrige If you have a Mac available you can use these tips to build NWjs yourself. It is based on the official build instructions with my tips.

Also here are the exact actions I use in my quick-and-dirty CI (in JSON format).

I will resume my builds but right now I do not have access to my build machine due to a network outage 😱

[ayushmanchhabra]

I will resume my builds

I've added MacOS ARM support on nw-builder using the @corwin-of-amber's builds! It is available from v4.1.0-beta.3 onward. I have a very basic demo set up. I don't have access to a Mac and am looking for feedback. Hope this helps!

[zkrige]

@tharatau i have a Mac that I work on. I've been looking to convert our build process from nwjs-builder to your more modern builder anyways. I'll gladly test it and give you feedback

[ayushmanchhabra]

@zkrige Feel free to contact me via the nwutils Discord server or Gitter chatroom, the links are posted on the website:
https://nwutils.io/

[JeFawk]

Any news on this & a stable release featuring it?

[corwin-of-amber]

I have been trying to re-instate my unofficial M1 builds since there does not seem to be sufficient development. Over the last few weeks I was stumbling because of a resurfacing issue causing the non-SDK build to fail:
#6579
It seems that chrome/browser/nwjs_resources.grd is somehow out of sync. I am trying this workaround (thanks @llamasoft) and will report whether the result works.

If anyone has a clue on what's going on there, help is extremely welcome.

I am also not able to build the payload executable anymore, but it seems that it has been removed from the official SDK as well? Perhaps @nebular can confirm that it is no longer needed.

[nebular]

Hi @corwin-of-amber ! Sorry i have been absent quite some time - been using your version 0.66.0 that has been running flawlessly for me for this long time. Glad to hear you are reinstating the unofficial builds. Thanks so much for that, I appreciate the big effort you put on it.

I can confirm I am not using "payload", thanks for asking.

Have upgraded my software "Obscene Studio" (https://obscene.studio) to your latest version 0.75 - also to test it in a new machine with a M2 PRO + Ventura 13.3.1. Encountered some irritating issues but at the end it is more a Ventura / Chromium thing .-- if you are curious it is about a chrome bug and other related one that causes getUserMedia calls to take a very long time to resolve, or even freeze until you change cameras ... a disaster :(

[rogerwang]

Please try this testing build: https://dl.nwjs.io/live-build/nw76/20230516-150708/a88c13a6c/v0.76.1/

I'm setting up the buildbot and it is scheduled to be released with 0.77 in June.

[corwin-of-amber]

Amazing! Thank you so much @rogerwang!
And thanks to all the wonderful M1 activists in this thread :D

[dpage]

@rogerwang Thanks! However, it won't open at all on my machine as it doesn't appear to be signed. It probably will open on the build machine (as I assume that's where you tested it), as it knows you built it on that box.

[JeFawk]

Thank you so much!!! Looking forward to a stable build so that web apps like Scirra (construct.net) can use it so and so we can publish HTML5 games using nwjs on apple native ^^

🥳🎉🎊

[rogerwang]

@rogerwang Thanks! However, it won't open at all on my machine as it doesn't appear to be signed. It probably will open on the build machine (as I assume that's where you tested it), as it knows you built it on that box.

After removing the quarantine bit it works for me in another system. Will fix it in the next build.

sudo xattr -r -d com.apple.quarantine /path/to/MyApp.app

[nebular]

omg omg omg its xmas !! thanks so much @rogerwang
btw, could you elaborate a little if you have fixed specific things ? being for long a user of @corwin-of-amber builds that work pretty well - but in M2 + Ventura getUserMedia is quite unstable

[dpage]

@rogerwang Thanks! However, it won't open at all on my machine as it doesn't appear to be signed. It probably will open on the build machine (as I assume that's where you tested it), as it knows you built it on that box.

After removing the quarantine bit it works for me in another system. Will fix it in the next build.

sudo xattr -r -d com.apple.quarantine /path/to/MyApp.app

Oh, neat - didn't realise you could do that. The basic app runs now - I'll try it in a build of pgAdmin and report any issues.

Thanks!

[TongDaDa]

emm, does nwjs support the m1 chip now? @rogerwang

[JeFawk]

emm, does nwjs support the m1 chip now? @rogerwang

Yes, for about 2 months now, v 0.77 i think

[TongDaDa]

@rogerwang @JeFawk

Hi, Is it possible to provide a universal nwjs sdk? other than x64 & arm64

ref: https://developer.apple.com/documentation/apple-silicon/building-a-universal-macos-binary

A few later... I use the lipo -create -output command to combine all the binary of nwjs-sdk.app into a single universal binary. BUT, the v8_context_snapshot file seems to be incompatible with this command,

throwing error with this command: fatal error: /Applications/Xcode.app/Contents/Developer/Toolchains/XcodeDefault.xctoolchain/usr/bin/lipo: can't figure out the architecture type of: /Users/xxx/Desktop/build/nwjs_arm64.app/Contents/Frameworks/nwjs Framework.framework/Versions/115.0.5790.114/Resources/v8_context_snapshot.arm64.bin

lipo -create -output ./universal64_snapshot /Users/xxx/Desktop/build/nwjs_arm64.app/Contents/Frameworks/nwjs\ Framework.framework/Versions/115.0.5790.114/Resources/v8_context_snapshot.arm64.bin /Users/xxx/Desktop/build/nwjs_x64.app/Contents/Frameworks/nwjs\ Framework.framework/Versions/115.0.5790.114/Resources/v8_context_snapshot.x86_64.bin

[bluthen]

What if you just leave that as two separate files? I think the executable nwjs Framework will load each file specifically by name (including the arch), not a general library name without arch like the others.

[TongDaDa]

What if you just leave that as two separate files? I think the executable nwjs Framework will load each file specifically by name (including the arch), not a general library name without Archs like the others.

@bluthen You can download the two arch bundles in the URL: https://dl.nwjs.io/v0.77.0/. x64 & arm64. most files can combine a single file from two binaries, but the Contents/Frameworks/nwjs Framework.framework/Versions/115.0.5790.114/Resources/v8_context_snapshot.arm64.bin file cannot be unverified

[bluthen]

Sorry if I'm wasting your time. I thought those files are just snapshots of v8 heap when it first loads and not a real binary, so I don't know why you need to combine them.

[TongDaDa]

Sorry if I'm wasting your time. I thought those files are just snapshots of v8 heap when it first loads and not a real binary, so I don't know why you need to combine them.

@bluthen Soooorry, It does not need to combine them after looking over the chrome fat release, and placing the two arches in the directory.

Totally, the purpose I did that is to build a UNIVERSAL nwjs build by lipo macos command, this means running an app on the x64 & arm64 platforms.

It is realized by electron, expected for nwjs.

https://stackoverflow.com/questions/73569950/how-to-build-a-universal-bundle-for-macos-on-apple-silicon-electron-builder

[MikalDev]

FYI for the Construct 3 platform which exports to two different Mac versions separately (ARM + Intel), I created a CLI tool to make a universal nwjs app from the two versions. Very early release, testing ongoing, works w/ small C3 demo apps exports and also later C3 Greengrinds addon (and probably C3 greenworks too)

https://github.com/MikalDev/fattennwjs open to PRs

Basic idea is to go through the ARM dir find all executables and fatten them from the intel dir, some special cases needed which complicates the code a little. Also does the v8_context copy from intel to arm.

[MikalDev]

I noticed that permissions did not work with the fattened version of the nw.js, unless the app was signed again, so added an optional --sign parameter to the fattennwjs app to do a codesign --force --deep -s "${keyName}" "${appPath}" - which works well enough for adhoc. For app store release, etc. there may be more work involved and you may not want to do the --force --deep, instead walk through the executables and only sign what is needed.