Edge.Func eventually stops working and all calls to it hang

[moodmosaic]

After some unknown period of time, Edge.Func eventually stops working and all calls to it hang. We're using version 0.9.2 of Edge.js, and 0.12.7 of Node.js, in the context of a Web API application.

This breakpoint, as well as this one and this one are all hit.

---

(I'm aware of #215, though the links there point always to the latest version of Edge and I'm not sure if they still make sense.)

---

This call succeeds if called asynchronously 1000000 times from a Console Application, though eventually can stop working if called from a ASP.NET Web API controller's asynchronous method:

var incrementTwice = Edge.Func(@"
        return function (data, callback) {
            var q = require('Q');
            var f = function() {
                var deferred = q.defer();
                deferred.resolve(data + 1);
                return deferred.promise;
            };
            f().then(
                function(result) { 
                    callback(null, result + 1);
                });                            
        }
    ");

if (Convert.ToInt32(await incrementTwice(1)) == 3)
{
    Console.WriteLine("[Passed] EdgeJs: " + sw.Elapsed.TotalSeconds);
}

Any help or feedback on this critical issue would be much appreciated.

[tjanczuk]

Are you calling Edge.Func for every HTTP request in your app? I don't see the JS code having any dependency on the request. In that case I'd recommend calling Edge.Func just once in the lifetime of the web application and store the result in some global variable, e.g. incrementTwice. You can then call incrementTwice whenever a request arrives. It can be called any number of times, concurrently or not.

Calling Edge.Func with literal C# makes Edge invoke the C# compiler and then load the resulting assembly into memory. Not only is this slow (compared to just calling into already pre-compiled function), but I'd not be surprised if it exhausts available memory when call a very large number of times.

[moodmosaic]

The code I've posted is a sample, showing how Edge.js can be used in this particular scenario.

Code similar to this comes from the clients (that issue HTTP requests on the server) so there isn't really something to store on a global variable.

It's really arbitrary code, coming from clients doing HTTP requests on the server and having Edge.js execute that code and then having the server return the result back to the clients with an HTTP response.

[kevinsbennett]

tjanczuk,

I work with moodmosaic... so I thought I'd jump in here too.

I'm not sure it was clear, but we're executing javascript code strings from C#. Are you saying that the javascript somehow turns into compiled C#?

[tjanczuk]

No, I apologize, I misread the original description. My comment above does not make sense in the context of your scenario.

What you are saying here:

This call succeeds if called asynchronously 1000000 times from a Console Application, though eventually can stop working if called from a ASP.NET Web API controller's asynchronous method

I am trying to understand the difference in your test code for console app vs web app. I imagine in the context of a web app the Edge.Func may end up being called concurrently as a result of concurrent HTTP requests? Did you console app replicate this situation or was Edge.Func called sequentially?

[kevinsbennett]

Yes, our use of Edge.Func may end up being called concurrently as a result of concurrent HTTP requests. Our test above does not test this aspect, but we will try it.

What we can tell you is that once the calls start hanging in production, all subsequent calls will hang as well until we reboot the server.

Basically, we have a sort of plugin system where users can write their own customizations to our software. So we don't know in advance the javascript that will be running. I can tell you that some of them can be quite large. We don't really see any performance degradation over time though.

We're willing to put in the work to figure this out, we're just hoping you might have some ideas on where to focus our investigation.

In addition to finding the root cause, just as important for us would be a way to sense that Edge might have entered this state (a timeout is difficult because we don't know how long the code should take... some of them involve http calls, calls to a database, etc.). If we can sense that this has occurred, the next step would be to correct it... like restarting Edge (node) if possible without having to reboot the server.

Thanks in advance for the help!

[tjanczuk]

just as important for us would be a way to sense that Edge might have entered this state (a timeout is difficult because we don't know how long the code should take... some of them involve http calls, calls to a database, etc.)

Let me make sure I understand: is it the Edge.Func call that appears to be hanging, or the call to the JS function around which Edge.Func created a CLR wrapper?

The time it takes to execute Edge.Func should be relatively quick and deterministic in most situations. The only aspect that may prolong Edge.Func execution is the JavaScript code that runs prior to return function .... In an extreme case, code of the following form:

var evilCode = Edge.Func(@"
    while (true);
    return function () {}; // this is never reached
");

will in fact cause Edge.Func to hang forever. Moreover, any subsequent calls to Edge.Func will also appear to hang, because the singleton node.js event loop is blocked by the while (true); that executed before.

where users can write their own customizations to our software

Generally speaking executing JavaScript code you don't control and therefore cannot trust in the address space of your web server and without any isolation between your users is major security issue. Interestingly I have spent the last year of my life developing a system that securely supports platform extensibility through untrusted Node.js code. It is key part of our identity platform at Auth0. You can read more about it at http://tomasz.janczuk.org/2015/07/extensibility-through-http-with-webtasks.html, and play with it at https://webtask.io.

[kevinsbennett]

Actually we don't know which is hanging, but I would assume it the execution of the javascript function that is hanging and not the Func call itself.

Our users cannot write raw javascript (as you are right, this would be a security issue), and there is isolation between users. They actually use a system of blocks that generates javascript. So we control everything they can and cannot do in this code. We just don't know exactly how they'll put the pieces together. With that said, webtasks look pretty awesome, and could be useful for another aspect of our project.

I can think of a couple ways that our block set could be put together in a way that produces an infinite loop, so this is definitely a helpful thought. We'll add handling for this situation so that it's not possible. Although it's possible this is the cause of the issue, I also think it is unlikely (we're in a private beta with a low number of users, and it would be a strange use of our block set to create this situation).

Any other thoughts on what could lead to this state that we should investigate as well?
And is there a way to restart Edge without rebooting the server?

[tjanczuk]

I also find infinite loops unlikely to be the root cause of the issue; I suspect there is a stress issue of some kind. Isolated repro would be very helpful in diagnosing the problem, ideally console app based. The first step would be to determine if it is Edge.Func or the actual function that hangs. Once you have a console.app repro, you can run it with EDGE_DEBUG=1 and EDGE_CS_DEBUG=1 environment variables set, that will generate plentiful debug info to stdout that may be helpful in tracking down the root cause.

There is no way of restarting Edge without bringing the process down.

[moodmosaic]

AFAICT, Edge runs on its own CLR thread, from where it starts the Node process. If it hangs, wouldn't it be possible to start a new CLR thread and continue from there?

[tjanczuk]

Edge does not start an external Node process. Edge starts the Node.js event loop and V8 within the CLR process and provides an in-memory interop mechanism between CLR and V8. While ripping out the CLR thread that Edge uses would not be a problem, shutting down and restarting the Node.js machinery is. The way Node.js is implemented it assumes it "is" the process, and does not offer any provisions for recycling itself other than just terminating the owning process.

[moodmosaic]

FWIW, on Edge.js 0.10.*, 4.*, and 5.*, the provided repro makes all calls to Edge.js to hang always:

1. Create a Console Application
2. Install Edge.js via NuGet
3. Install q library via npm in bin\Debug (npm install q)
4. Run the provided repro

---

It looks like q library is not working after Edge.js 0.9.*, or something related to it.

[ljani]

I had better luck with promises on 5.x if I append something like this at the end:

setInterval(function () {}, 100);

It felt too ugly and I decided to ditch Edge.js all together.

[moodmosaic]

It felt too ugly and I decided to ditch Edge.js all together.

What did you use instead?

[ljani]

@moodmosaic

What did you use instead?

IIRC some kind of homebrew solution where I directly run node.exe via the Process API and parse the StandardOutput. Not the most elegant solution, but it works and I know why it works instead of relying on black magic.

[moodmosaic]

Hmm... I guess that might easily work for trivial scenarios. Not sure what happens when you want to pass complex .NET types and/or delegates.

[ljani]

@moodmosaic I agree. I'd prefer Edge.js if it worked, since my solution doesn't really scale and it introduces some other pitfalls you need to be aware of.

Anyway, in my case I'm using nodejs to do some on-demand processing (ie. running tasks/jobs). I serialize my job description to JSON and launch the process/script and the process outputs JSON which I parse again. I could use some other transport than stdout/stdin, but I didn't need it. The case is simple enough for this to work.

[dpolivy]

I think the hang is related to #325. I'm working on a fix; you can follow along in that issue for details.

[dpolivy]

A fix was just released which should address this. See if the latest release ([email protected]) fixes the problem and report back!

[Ilair]

Olá! Estou recebendo este erro abaixo nesta chamada:

var imprimir = edge.func('DYMO/NewERP.DymoPrinter.dll');

Fatal error in c:\projects\libchromiumcontent\src\v8\src\objects.cc, line 3230

Check failed: receiver->IsJSFunction().

Error initializing symbols (87). Dumping unresolved backtrace:

Gostaria de saber como posso resolver isso. Grato pela Atenção!