Integration of Node.js core-modules (maybe as optional extension / feature?)

[drywolf]

@nakosung

By opening this issue I just to want to get a feeling what's your take on supporting the Node.js core modules in Unreal.js ? How feasable would it be to integrate them in Unreal.js ?

I'm asking this because whenever I want to try out something more serious with Unreal.js that involves any kind of external npm modules, it will never work because the required core Node.js modules are missing.

What do you think would the effort be to add support for some of the more fundamental Node.js modules (e.g. 'assert' 'buffer' 'fs' 'path' 'http' 'tty')

If it is not totally off the map, we could talk about how to achieve it. I'd certainly be willing to help out if I knew a bit better how this integration should get started.

Thanks

[nakosung]

@drywolf You can switch modules with pure javascript versions which are made for browsers. (eg. buffer, path, assert, ...)

https://www.npmjs.com/package/assert https://www.npmjs.com/package/buffer https://www.npmjs.com/package/util

But for 'fs', 'http', 'net' there aren't any appropriate versions, which should be implemented within Unreal.js.

I think that Unreal.js can be a kind of browsers, not a node.js compatible one. Integrating node.js core modules may be not off the map, but I have so little time to extend unreal.js to support the whole universe of npm modules. If you want to help, I'll really appreciate.

[getnamo]

I wonder if you could write platform specific binds which re-implement the api the modules have in C++. E.g. you bind fs to windows commands, same with http and net. Then for each platform you'd do the same. Seems like a lot of work though potentially.

Edit: Looking at the source a bit, you might be able to just drop in node.js c++ and .js file sources and with minimal modification have node.js core modules embedded in unreal.js, but there are a lot of inter-dependencies and you might just need to embed entire node.js.

link to node source: https://github.com/nodejs/node/tree/master/src

how to look at core module source: http://serebrov.github.io/html/2013-12-02-node-core-module-source.html

e.g. net is defined in https://github.com/nodejs/node/blob/master/lib/net.js

which depends on these c++ headers: https://github.com/nodejs/node/blob/master/src/pipe_wrap.h https://github.com/nodejs/node/blob/master/src/tcp_wrap.h https://github.com/nodejs/node/blob/master/src/tty_wrap.h https://github.com/nodejs/node/blob/master/src/stream_wrap.h

and cares/uv (which I'm not sure where they are atm)

Edit2: There is also this (embedding node into c++ app): https://github.com/nodekit-io/nodekit

and

https://github.com/ZECTBynmo/tacnode but it is massively out of date.

where you could write a wrapper to pass-through the fs, http, net calls to the embedded part.

[nakosung]

@getnamo I consider wrapping over UnrealEngine API (FSocket, FFileSystem, ...). It has quite different set of API but it would be easy to write API translation layer in javascript.

I think that 100% compatibility cannot be achieved by any means because unreal.js is not a node.js, but unreal.js could mimic node.js just as browserify does.

[drywolf]

I consider wrapping over UnrealEngine API (FSocket, FFileSystem, ...). It has quite different set of API but it would be easy to write API translation layer in javascript.

I think that 100% compatibility cannot be achieved by any means because unreal.js is not a node.js, but unreal.js could mimic node.js just as browserify does.

That's what I also thought would make the most sense and could be doable with a reasonable effort.

My current usecase is that I am trying to run Mocha.js inside Unreal.js to perform automated integration tests / unit tests for the u.js related project that I'm working on (react-umg).

But for this to work I think at least the fs module has to work properly with the UE FileSystem underneath, what the other dependencies for that usecase are I still have to figure out, but file-access is definitely the first on the list (for me).

[nakosung]

fs is definitely the first module to be supported properly.

BTW, I had implemented mocha-like test framework integrated within UE automated test. (https://github.com/ncsoft/Unreal.js/wiki/Automated-test)

[getnamo]

I believe that's the right approach. Just keep in mind the option of having the missing modules as passthroughs if you hit a roadblock. that's what I think https://github.com/nodekit-io/nodekit could be used for.

[drywolf]

I'm currently investigating this (in the scope of getting Mocha.js running for my usecase).

What I found is another problem, which seems to be related to the way that Unreal.js currently uses to resolve & load node modules. If I analyzed the problem correctly, then the current module loading code of Unreal.js runs into trouble if there is a circular dependency in the require() module dependency graph.

Such a problem occurs for example in the node-glob module, which Mocha.js also uses.

The circular require() dependency happens between those files/lines:

• https://github.com/isaacs/node-glob/blob/master/sync.js#L8
• https://github.com/isaacs/node-glob/blob/master/glob.js#L52

Which leads to the following error log in Unreal.js

< Invalid script for require: './glob.js'
    at path: c:\code\react-umg\examples\ReactUmgExamples\Content\Scripts\node_modules\glob\glob.js
Error: c:\code\react-umg\examples\ReactUmgExamples\Content\Scripts\node_modules\glob\sync.js:7: TypeError: Cannot read property 'Glob' of undefined
Error: TypeError: Cannot read property 'Glob' of undefined
Error:     at c:\code\react-umg\examples\ReactUmgExamples\Content\Scripts\node_modules\glob\sync.js:7:32
Error:     at c:\code\react-umg\examples\ReactUmgExamples\Content\Scripts\node_modules\glob\sync.js:462:4
Error:     at c:\code\react-umg\examples\ReactUmgExamples\Content\Scripts\node_modules\glob\sync.js:463:25
Error:     at c:\code\react-umg\examples\ReactUmgExamples\Content\Scripts\node_modules\glob\glob.js:51:16
Error:     at c:\code\react-umg\examples\ReactUmgExamples\Content\Scripts\node_modules\glob\glob.js:754:4
Error:     at c:\code\react-umg\examples\ReactUmgExamples\Content\Scripts\node_modules\glob\glob.js:755:25
Error:     at c:\code\react-umg\examples\ReactUmgExamples\Content\Scripts\node_modules\glob\sync.js:7:32
Error:     at c:\code\react-umg\examples\ReactUmgExamples\Content\Scripts\node_modules\glob\sync.js:462:4
Error:     at c:\code\react-umg\examples\ReactUmgExamples\Content\Scripts\node_modules\glob\sync.js:463:25
Error:     at c:\code\react-umg\examples\ReactUmgExamples\Content\Scripts\node_modules\glob\glob.js:51:16
< Invalid script for require: './sync.js'
    at path: c:\code\react-umg\examples\ReactUmgExamples\Content\Scripts\node_modules\glob\sync.js
Error: c:\code\react-umg\examples\ReactUmgExamples\Content\Scripts\node_modules\glob\glob.js:53: TypeError: Cannot read property 'alphasort' of undefined
Error: TypeError: Cannot read property 'alphasort' of undefined
Error:     at c:\code\react-umg\examples\ReactUmgExamples\Content\Scripts\node_modules\glob\glob.js:53:23
Error:     at c:\code\react-umg\examples\ReactUmgExamples\Content\Scripts\node_modules\glob\glob.js:754:4
Error:     at c:\code\react-umg\examples\ReactUmgExamples\Content\Scripts\node_modules\glob\glob.js:755:25
Error:     at c:\code\react-umg\examples\ReactUmgExamples\Content\Scripts\node_modules\glob\sync.js:7:32
Error:     at c:\code\react-umg\examples\ReactUmgExamples\Content\Scripts\node_modules\glob\sync.js:462:4
Error:     at c:\code\react-umg\examples\ReactUmgExamples\Content\Scripts\node_modules\glob\sync.js:463:25
Error:     at c:\code\react-umg\examples\ReactUmgExamples\Content\Scripts\node_modules\glob\glob.js:51:16
Error:     at c:\code\react-umg\examples\ReactUmgExamples\Content\Scripts\node_modules\glob\glob.js:754:4
Error:     at c:\code\react-umg\examples\ReactUmgExamples\Content\Scripts\node_modules\glob\glob.js:755:25
Error:     at c:\code\react-umg\examples\ReactUmgExamples\Content\Scripts\node_modules\glob\sync.js:7:32
< Invalid script for require: './glob.js'
    at path: c:\code\react-umg\examples\ReactUmgExamples\Content\Scripts\node_modules\glob\glob.js
Error: c:\code\react-umg\examples\ReactUmgExamples\Content\Scripts\node_modules\glob\sync.js:7: TypeError: Cannot read property 'Glob' of undefined
Error: TypeError: Cannot read property 'Glob' of undefined
Error:     at c:\code\react-umg\examples\ReactUmgExamples\Content\Scripts\node_modules\glob\sync.js:7:32
Error:     at c:\code\react-umg\examples\ReactUmgExamples\Content\Scripts\node_modules\glob\sync.js:462:4
Error:     at c:\code\react-umg\examples\ReactUmgExamples\Content\Scripts\node_modules\glob\sync.js:463:25
Error:     at c:\code\react-umg\examples\ReactUmgExamples\Content\Scripts\node_modules\glob\glob.js:51:16
Error:     at c:\code\react-umg\examples\ReactUmgExamples\Content\Scripts\node_modules\glob\glob.js:754:4
Error:     at c:\code\react-umg\examples\ReactUmgExamples\Content\Scripts\node_modules\glob\glob.js:755:25
Error:     at c:\code\react-umg\examples\ReactUmgExamples\Content\Scripts\node_modules\glob\sync.js:7:32
Error:     at c:\code\react-umg\examples\ReactUmgExamples\Content\Scripts\node_modules\glob\sync.js:462:4
Error:     at c:\code\react-umg\examples\ReactUmgExamples\Content\Scripts\node_modules\glob\sync.js:463:25
Error:     at c:\code\react-umg\examples\ReactUmgExamples\Content\Scripts\node_modules\glob\glob.js:51:16
< Invalid script for require: './sync.js'
    at path: c:\code\react-umg\examples\ReactUmgExamples\Content\Scripts\node_modules\glob\sync.js
Error: c:\code\react-umg\examples\ReactUmgExamples\Content\Scripts\node_modules\glob\glob.js:53: TypeError: Cannot read property 'alphasort' of undefined
Error: TypeError: Cannot read property 'alphasort' of undefined
Error:     at c:\code\react-umg\examples\ReactUmgExamples\Content\Scripts\node_modules\glob\glob.js:53:23
Error:     at c:\code\react-umg\examples\ReactUmgExamples\Content\Scripts\node_modules\glob\glob.js:754:4
Error:     at c:\code\react-umg\examples\ReactUmgExamples\Content\Scripts\node_modules\glob\glob.js:755:25
Error:     at c:\code\react-umg\examples\ReactUmgExamples\Content\Scripts\node_modules\glob\sync.js:7:32
Error:     at c:\code\react-umg\examples\ReactUmgExamples\Content\Scripts\node_modules\glob\sync.js:462:4
Error:     at c:\code\react-umg\examples\ReactUmgExamples\Content\Scripts\node_modules\glob\sync.js:463:25
Error:     at c:\code\react-umg\examples\ReactUmgExamples\Content\Scripts\node_modules\glob\glob.js:51:16
Error:     at c:\code\react-umg\examples\ReactUmgExamples\Content\Scripts\node_modules\glob\glob.js:754:4
Error:     at c:\code\react-umg\examples\ReactUmgExamples\Content\Scripts\node_modules\glob\glob.js:755:25
Error:     at c:\code\react-umg\examples\ReactUmgExamples\Content\Scripts\node_modules\glob\sync.js:7:32
...
...
...

... those are just two iterations of the error, this is actually repeated 300+ times ... I think at some point just an internal stack overlow in V8 happens and breaks the infinite loop (no visible error is shown about that though).

@nakosung Can you confirm that circular usage of require() in node modules will cause this behavior ?

I would need to find a fix for this issue before I can start looking into what core-modules would need to be ported for Mocha.js to run.

[drywolf]

PS: I was able to create a quite minimal reproducible test case for this behavior...

https://gist.github.com/drywolf/7e5e4517c4edbd9e55b5883a30b923a6

If you run the code in regular node via node run.js the code executes just fine and prints rnd 4 In Unreal.js it leads to the infinite-loop in the module loading code as described above.

[nakosung]

@drywolf Actually there was an issue related this problem. (https://github.com/ncsoft/Unreal.js/issues/85) Node.js discourages use of circular dependencies, but it seems mocha.js doesn't bother with those guide lines. (sad)

[nakosung]

module needs to be registered before executing .js code. https://github.com/ncsoft/Unreal.js-core/blob/master/Source/V8/Private/JavascriptContext_Private.cpp#L1263 https://github.com/ncsoft/Unreal.js-core/blob/master/Source/V8/Private/JavascriptContext_Private.cpp#L1269

To fix,

1. Build a module object in C++ first, (module = { exports : {}, dirname : string })
2. Register it to module cache in C++,
3. Call js with the module created before.

[drywolf]

Thanks @nakosung for the hints.

For a fast & probably cleaner fix for my case I started to integrate a bundled version of mocha.js which effectively works arround the whole node.js module resolution issue. (https://cdnjs.com/libraries/mocha)

But unrelated, I'm still interested in finding a solution so we can start using UE4 with full Node.js compatibility, JavaScript as a cross-platform development target becomes way more powerful in the form of Node.js What do you think, wouldn't it make Unreal.js a much more viable development tool if it came with all the features that Node.js provides and modern JavaScript developers have come to expect ?

I've read through the following issue on the Node.js project which makes it seem like embedding Node.js is pretty achievable nowadays anyway... https://github.com/nodejs/node/issues/5265

I might have a look into it over the weekend if I find some time.

[njibhu]

I don't think building Unreal.js on top of node is a good idea.. I would prefer having a wrapper between Unreal API's and node, than having two different way of accessing the same ressources. That will lead to chaos if you start to access ressources with both node and unreal, and I'm quite sure integrating them together must be a nightmare because both node and unreal expect to provide a highlevel API of lowlevel ressources.. I'm scared that it would lead to a lot of bugs.

I know many people that are interested by Unreal.js to work with Unreal API with Javascript, and I'm feeling that way too, even though node is great, people are expecting Unreal.js to be about integrating js into unreal, not Unreal into node..

Don't get me wrong, exposing Node modules through a wrapper with Unreal seems to me a good idea.

[HeadClot]

Personally speaking having an optional NodeJS Module for UnrealJS would be a major time saver.

+1 from me on this. :)

[nakosung]

@HeadClot You can build and publish your own unreal.js node modules on npm, just like react-native modules which work only on specific platforms. :)

[drywolf]

In my opinion the two approaches are not mutually exclusive, but could serve two different perspectives of abstraction & integration when it comes to JavaScript inside UE:

• Unreal.JS-core
  • lightweight & performance oriented V8 runtime integration
  • for doing simple game-logic JS scripting in UE
  • with minimal memory/CPU footprint
  • only wraps the required UE APIs for scripting
  • does not aim to support most of the NPM ecosystem out of the box
• Unreal.JS-node
  • Node.JS compatible V8 runtime + including all Node.JS supported core APIs
  • aiming for compatibility with NPM modules from npmjs.org
  • also integrates/wraps UE APIs that are needed for scripting in UE
  • needs to try and reuse/wrap the existing UE APIs that intersect with the ones that Node.JS needs to provide whereever possible, to not create two conflicting worlds in the system which would lead to a bad developer experience (this is targeting the point @njibhu brought up)

PS: when I looked into the approaches for embedding Node.JS I found electron/node which might be the most stable and best supported project out there for doing just that

[sascha1337]

+1

[speedpacer]

+1

[neurogamedev]

+1