Using DashJS (which requires grpc) in AWS Lambda function

[riongull]

Expected Behavior

Expected to be able to use DashJS without issue, regardless of platform.

Current Behavior

I got a {"message": "Internal server error"} when my DashJS-using lambda function fired (which was set up to fire upon an http API request, but that's probably irrelevant). The original error on AWS showed:

{
    "errorType": "Error",
    "errorMessage": "Failed to load gRPC binary module because it was not installed for the current system\nExpected directory: node-v64-linux-x64-glibc\nFound: [node-v64-darwin-x64-unknown]\nThis problem can often be fixed by running \"npm rebuild\" on the current system\nOriginal error: Cannot find module '/var/task/node_modules/grpc/src/node/extension_binary/node-v64-linux-x64-glibc/grpc_node.node'",
    "code": "MODULE_NOT_FOUND",
    "stack": [
        "Error: Failed to load gRPC binary module because it was not installed for the current system",
        "Expected directory: node-v64-linux-x64-glibc",
        "Found: [node-v64-darwin-x64-unknown]",
        "This problem can often be fixed by running \"npm rebuild\" on the current system",
        "Original error: Cannot find module '/var/task/node_modules/grpc/src/node/extension_binary/node-v64-linux-x64-glibc/grpc_node.node'",
        "    at Object.<anonymous> (/var/task/node_modules/grpc/src/grpc_extension.js:53:17)",
        "    at Module._compile (internal/modules/cjs/loader.js:778:30)",
        "    at Object.Module._extensions..js (internal/modules/cjs/loader.js:789:10)",
        "    at Module.load (internal/modules/cjs/loader.js:653:32)",
        "    at tryModuleLoad (internal/modules/cjs/loader.js:593:12)",
        "    at Function.Module._load (internal/modules/cjs/loader.js:585:3)",
        "    at Module.require (internal/modules/cjs/loader.js:692:17)",
        "    at require (internal/modules/cjs/helpers.js:25:18)",
        "    at Object.<anonymous> (/var/task/node_modules/grpc/src/client_interceptors.js:144:12)",
        "    at Module._compile (internal/modules/cjs/loader.js:778:30)"
    ]
}

Problem & Solution

DashJS uses grpc under the hood. The problem seems to be that the grpc package you download depends on the platform you run npm install on. If you are developing on MacOS (darwin), the downloaded package will include a directory named node-v64-darwin-x64-unknown. If your function ultimately executes on a different platform than the one you packaged your lambda on there's going to be a problem, because the runtime will look for a folder (e.g. node-v64-linux-x64-glibc for the executing platform (typically Linux).

After debugging this for longer than I'd like to admit I figured out the (or at least one possible) solution:

1. Add the following to your package.json:

"scripts": {
  "postinstall": "npm rebuild grpc --target_arch=x64 --target_platform=linux --target_libc=glibc"
}

I'm pretty sure this adds a folder to the package (rather than replacing it), so it should work in both your local and remote environments. I found this out by first naively trying other solutions like this and this, where they include a specific --target=x.y.z in that postinstall script above. This would add a path to the "Found: [node-v64-darwin-x64-unknown]" directory array, but still resulted in an error. Simply removing the --target=x.y.z part did the trick because it seems to default to process.version, which should always work.

2. Run rm -rf ./node_modules/ package-lock.json && npm install on your development machine.

This removes your cached packages and reinstalls them fresh, now including the target environment's grpc version due to the postinstall script.

Recommendation

I'm not sure if there's anything to do to fix this issue other than to know about how to address it (as explained above), but it might be helpful if there's something that can be changed in the packages DashJS requires for grpc so that the issue doesn't arise.

[Alex-Werner]

Thank you so much ! I wonder if not committing package-lock.json be enough ? Did you try with default script but with the lock file removed ?

Mentioning @antouhou @shumkov too, they might be interested.

[thephez]

Thanks for the info. I have bumped into this when using repl.it (e.g. name script) recently, but thought it was something related to changes on their site and did not debug any further.

[shumkov]

@riongull Thank you very much for your report and the provided solution. It seems like we need to compile it manually for rare platforms. I think it's fair for everything.

I'm afraid, we can't add this post-install script 'coz it's too specific (not universal).

@Alex-Werner Removing package-lock.json is not a case. We won't be able to guarantee that things are works properly with different deps.

What if we just document this solution and ask people to solve it manually?

P.S. We could use pure JS implementation when it ready https://github.com/grpc/grpc-node/tree/master/packages/grpc-js

[shumkov]

Pure JS implementation is ready to use https://github.com/grpc/grpc-node/releases/tag/%40grpc%2Fgrpc-js%401.0.0

[riongull]

Happy to help, men. @shumkov, I think it would be good to look into whether the pure JS implementation will work for us. It seems to have pretty good feature parity.

[thephez]

@Alex-Werner @shumkov Can we switch to this solution soon? Would be nice to get this issue closed.

[shumkov]

Yeah, but we need to ask @dalibran about priority.

[riongull]

In terms of priority, this issue is no longer blocking me (or anyone else who reads this for the solution). Issue dashevo/js-dash-sdk#46 on the other hand is blocking anyone who wants to consume DashJS in a typescript file. Solving that first would not only unblock them, it should also provide IntelliSense to normal JS devs so that they can see documentation in their IDE, which lowers the urgency of keeping text docs up to date.

[cloudwheels]

Longshot as there's no direct error message, but might this also explain why dash can't be used in runkit, which gives the error Cannot find module 'grpc' ... in @dashevo/dapi-grpc/clients/nodejs/CorePromiseClient.js ? ( https://npm.runkit.com/dashand and run to recreate)

Pure js version might at least solve that problem too since https://npm.runkit.com/@grpc/grpc-js is a thing, whereas https://npm.runkit.com/grpc does indeed show it as missing.

[cloudwheels]

@shumkov Is it possible to clarify whether the additional considerations for migration below are likely to exist, or whether this should be 'a drop-in replacement'?

Migrating from grpc @grpc/grpc-js is almost a drop-in replacement for grpc, but you may need to make a few code changes to use it:

If you are currently loading .proto files using grpc.load, that function is not available in this library. You should instead load your .proto files using @grpc/proto-loader and load the resulting package definition objects into @grpc/grpc-js using grpc.loadPackageDefinition.

If you are currently loading packages generated by grpc-tools, you should instead generate your files using the --generate_package_definitions option in grpc-tools, then load the object exported by the generated file into @grpc/grpc-js using grpc.loadPackageDefinition.

If you have a server and you are using Server#bind to bind ports, you will need to use Server#bindAsync instead.

[shumkov]

@cloudwheels yeah, we already use proto-loader, so It should be easy. Please check https://github.com/dashevo/dapi-grpc

P.S. Your PR as usual will be appreciated 😉

[dashameter]

@cloudwheels @riongull have you had success running the pure js implementation on aws or is there additional work required that we could bounty out to the incubator?

[shumkov]

In v0.21 we migrated to the pure JS implementation of the gRPC library, so you can use this version for your test.

[SamKirby22]

Discussed in Triage - Merged and ready for retest in v21.

[dashameter]

Testing pure js implementation:

using sdk on runkit: https://runkit.com/embed/4hfnp3oa7eki

throws: Error: Cannot find module 'protobufjs' even though protobufjs is on runkit: https://npm.runkit.com/protobufjs

[SamKirby22]

Thanks @dashameter i've marked this as Failed Retest on the triage sheet.

[thephez]

@dashameter It's probably related to dapi-grpc (see https://npm.runkit.com/@dashevo/dapi-grpc) which uses this: https://github.com/dashevo/dapi-grpc/blob/v0.22-dev/package-lock.json#L16. I frequently have issues due to a git+ssh reference in package-lock.json for this specific dependency (which I've gotten around by borrowing this).

[SamKirby22]

@dashameter please verify workaround. Thanks

[dashameter]

@dashameter It's probably related to dapi-grpc (see https://npm.runkit.com/@dashevo/dapi-grpc) which uses this: https://github.com/dashevo/dapi-grpc/blob/v0.22-dev/package-lock.json#L16. I frequently have issues due to a git+ssh reference in package-lock.json for this specific dependency (which I've gotten around by borrowing this).

Thanks for the trick, how would I use git config --global url."https://github.com/".insteadOf ssh://[email protected]/ with runkit, though ? The whole issue to be resolved is being able to use the sdk with js only, e.g. runkit or aws lambda functions.

[dash-maverick]

@thephez can you refer to that please ?

[thephez]

@dash-maverick I was using the library in a different way and the workaround I used isn't applicable for this case. Someone else (probably a dev) will need to look at this. It appears this is still an issue with the latest dev version (0.22-dev.7).

[thephez]

@shumkov Some interesting discussion in a GH issue that appears related: https://github.com/npm/cli/issues/2610#issuecomment-914610832. Perhaps updating this would fix it: https://github.com/dashevo/platform/blob/v0.22-dev/packages/dapi-grpc/package.json#L41? And https://github.com/dashevo/platform/blob/v0.22-dev/packages/js-grpc-common/package.json#L34 and https://github.com/dashevo/platform/blob/v0.22-dev/package.json#L68?

[shumkov]

Interesting. @jawid-h @dash-maverick shall we just release your branch as NPM module in dashevo organisation?

[jawid-h]

@shumkov yeah, probably we should, let me check if a 7 version is released

[jawid-h]

@shumkov it is not, let's release our version maybe. I would merge latest updates from v6 of the original repo as well

[dashameter]

retest failed with sdk v3.21.8

[dash-maverick]

try it out with v22 please

[dashameter]

retest failed on v22.1

https://runkit.com/622b3b64e1ba8c000a6090da/62403edffa69e900094bb6ee

[SamKirby22]

As per triage, unable to reproduce, please retest.

[dash-maverick]

Hey. The investigation/fix is being currently in progress. You can expect some updates here quite soon.