feat(devops): Bundle used node_modules in the server compilation

[mattkrick]

Signed-off-by: Matt Krick [email protected]

Description

Removes the dependency on the node_modules directory in production builds.

TEST

• [ ] after running yarn build delete node_modules & run node dist/web.js and node gqlExecutor.js. They both startup
• [ ] create a new user with a password to make sure bcrypt still works
• [ ] upload a large avatar image to make sure sharp still works

Notes:

• pm2 isn't included in the bundle because i think we're just gonna call the files directly (see test 1 above) @rafaelromcar-parabol
• gqlExecutor needs sharp & web needs uWebSockets.js. when putting these in pods, you can separate them out accordingly @adaniels-parabol

[mattkrick]

somehow moving sharp out of node modules means we have to install libvips on its own:

apt-get update && apt-get install -y libvips-dev --no-install-recommends

[nickoferrall]

Hey @mattkrick, is this PR ready for review? I see you requested a review from me, but then added some commits and left comments for Aaron & Rafa. Just want to check that the request for review was intentional before starting

[mattkrick]

@nickoferrall not yet.... still running out of memory on circleci, hopefully will fix today

[adaniels-parabol]

@nickoferrall not yet.... still running out of memory on circleci, hopefully will fix today

@mattkrick I did some testing, if you give it 10GB it'll build (8-9GB it would sometimes OOM) 10GB passed every time.

[mattkrick]

circleci only gives us 8GiB. The error I'm getting is from a worker thread, which means it's not running on a single thread, which explains the memory explosion. Going to try to figure out what is creating child processes & hopefully bring it under control

[mattkrick]

@nickoferrall ready for review! (don't worry about the 45 commits :sweat_smile: )

[nickoferrall]

@mattkrick I'm having difficulties with the sharp module when running node dist/web.js. I get the following:

nickoferrall@MacBook-Pro parabol % node dist/web.js                                  
/Users/nickoferrall/parabol/dist/web.js:594613
  throw new Error(help.join('\n'));
  ^

Error: 
Something went wrong installing the "sharp" module

node-loader:
Error: dlopen(/Users/nickoferrall/parabol/dist/sharp-darwin-arm64v8.node, 0x0001): Library not loaded: @rpath/libvips-cpp.42.dylib
  Referenced from: /Users/nickoferrall/parabol/dist/sharp-darwin-arm64v8.node
  Reason: tried: '/Users/nickoferrall/parabol/dist/../.././vendor/8.12.2/darwin-arm64v8/lib/libvips-cpp.42.dylib' (no such file), '/Users/nickoferrall/parabol/dist/../.././vendor/8.12.2/darwin-arm64v8/lib/libvips-cpp.42.dylib' (no such file), '/usr/local/lib/libvips-cpp.42.dylib' (no such file), '/usr/lib/libvips-cpp.42.dylib' (no such file)

Possible solutions:
- Install with verbose logging and look for errors: "npm install --ignore-scripts=false --foreground-scripts --verbose sharp"
- Install for the current darwin-arm64v8 runtime: "npm install --platform=darwin --arch=arm64v8 sharp"
- Consult the installation documentation: https://sharp.pixelplumbing.com/install
    at Object.65939 (/Users/nickoferrall/parabol/dist/web.js:594613:9)
    at __webpack_require__ (/Users/nickoferrall/parabol/dist/web.js:854362:42)
    at Object.26955 (/Users/nickoferrall/parabol/dist/web.js:590803:1)
    at __webpack_require__ (/Users/nickoferrall/parabol/dist/web.js:854362:42)
    at Object.66456 (/Users/nickoferrall/parabol/dist/web.js:591192:15)
    at __webpack_require__ (/Users/nickoferrall/parabol/dist/web.js:854362:42)
    at Object.53798 (/Users/nickoferrall/parabol/dist/web.js:92828:17)
    at __webpack_require__ (/Users/nickoferrall/parabol/dist/web.js:854362:42)
    at Object.89779 (/Users/nickoferrall/parabol/dist/web.js:118605:18)
    at __webpack_require__ (/Users/nickoferrall/parabol/dist/web.js:854362:42)
    at /Users/nickoferrall/parabol/dist/web.js:854455:37
    at Object.<anonymous> (/Users/nickoferrall/parabol/dist/web.js:854457:12)
    at Module._compile (node:internal/modules/cjs/loader:1103:14)
    at Object.Module._extensions..js (node:internal/modules/cjs/loader:1155:10)
    at Module.load (node:internal/modules/cjs/loader:981:32)
    at Function.Module._load (node:internal/modules/cjs/loader:822:12)

I've:

• followed the steps it suggests
• nuked and reinstalled node modules
• tried the suggested solutions here: https://sharp.pixelplumbing.com/install#apple-m1

I just want to check whether you recognise the issue before I spend more time digging into a solution?

EDIT: I tried again this morning, saw this comment, brew installed libvips but got the same issue

[mattkrick]

hmmm, you'll need the libvips-dev package, i'm not sure that's available on homebrew. sharp isn't build to support this usecase & i'm without a macbook so I can't help much :-( https://github.com/lovell/sharp/issues/3171#issuecomment-1085780209

could be as easy an a reinstall? https://github.com/lovell/sharp/issues/2029#issuecomment-573301209

ultimately it's pointing to the wrong location, so you'll have to instruct it to look for vips outside of node_modules

[mattkrick]

@nickoferrall you could also try building from source! that should do it

[mattkrick]

docker commands look like they work, i think the only holdup is getting this working on macOS. Since this is devops specific & we won't be bundling on macOS I think it's safe to keep the merge train going