Provide ability to bundle dev dependencies for Node adapter

[Conduitry]

Describe the problem

Currently, the Node adapter will not bundle any external server side dependencies. This means that you need to make all of the dependencies you might need at runtime be prod dependencies. This differs from its (recent) previous behavior, and from Sapper's behavior, and from (I would argue) expected behavior.

Until now, it's been standard operating procedure to bundle as many of the server-side dependencies as possible, to reduce the deployed app size, parse time, and startup time. If there's some good reason to stop doing this, it should certainly at least be documented.

This issue may also well apply to other adapters (other than the static one), but I haven't tried to validate that.

Describe the proposed solution

I'm not entirely sure. It would be nice to be able to provide sensible default Vite configuration so that we have Vite do it, rather than do this in a separate esbuild step afterwards (either automatically or manually).

I don't know whether some different behavior from Vite would make this easier on our end, or whether this is something that Vite ought to change.

Below, I describe what I'm currently doing in userland to deal with this, but I'm not positive that this would have an equivalent in-framework analogue. (Do we have convenient access to the user's project's package.json?)

Alternatives considered

Currently, I have a vite.ssr.noExternal setting of process.env.NODE_ENV === "development" ? undefined : Object.keys(pkg.devDependencies) (where pkg is the parsed JSON from package.json) to force Vite to bundle all of my dev dependencies.

The check for development vs prod mode is there because I was having an issue using Object.keys(pkg.devDependencies) in dev mode that I haven't fully tracked down or tried to reproduce in a minimal repo, but which seem to stem from having a CJS-only dependency that Vite doesn't try to convert to ESM during dev mode.

Importance

would make my life easier

Additional Information

No response

[Karlinator]

So there used to be an esbuild step in adapter-node, which would bundle everything. It was removed in #2931.

@Rich-Harris was this an unintended side-effect of that? I can't see it mentioned in the PR or discussion.

[rpf3]

I ran into a similar issue to @Conduitry since upgrading and taking inspiration from this thread I added my package dependencies manually to the vite.ssr.noExternal config and got things working again. This definitely feels like a breaking change in how the Node adapter previously worked though.

A path forward would be a welcome discussion.

EDIT:

This was actually an issue with my build pipeline. We were copying the output directory of the Node adapter into a Docker image thinking it would provide a fully working application. I think what was going on is because certain packages were being externalized I needed to also copy the node_modules folder into the image.

[SomaticIT]

Hello,

I agree with @Conduitry that it could be interesting to have an option to bundle dev dependencies when using node adapter. The workaround is working well but an option would be simpler to use.

I think it should not create a single entry file (like before) but bundles only dev dependencies as suggested by Conduitry. An option directly on svelte-kit that apply the above workaround may also be ok.

[Conduitry]

As of the latest versions of SvelteKit - with a separate vite.config.js - I now need to use conditional Vite config (https://vitejs.dev/config/#conditional-config) and specify an ssr.noExternal value of command === 'build' ? Object.keys(pkg.devDependencies) : undefined.

[benmccann]

Doing this makes the system harder to understand in my opinion because it makes dev different than prod (though perhaps that ship has sailed) and SvelteKit different from Vite. It also adds another way in which adapter-node is different from the rest (though the others are also different because they use esbuild).

If the adapters were Vite plugins (or could provide Vite plugins) then we could at least make this functionality available with a single option like bundleDependencies: boolean.

[loxs]

Yeah, this is really annoying. Currently my docker image is 1.5G and could be 150-200M if everything was bundled

[sidvishnoi]

FWIW, we presently use a postbuild step using esbuild. This reduces our docker image size from 1GB to 20MB.

// File: scripts/postbuild.js
// In package.json: scripts.postbuild = "node scripts/postbuild.js"
require('esbuild').buildSync({
  entryPoints: ['./build/index.js'],
  outfile: './build/index.bundle.js',
  format: 'esm',
  target: 'esnext',
  platform: 'node',
  // When outputting ESM, any require calls will be replaced with the __require
  // shim, since require will not be available. However, since external paths
  // will not be bundled, they will generate a runtime error. This matters only
  // for built-in node modules.
  // https://github.com/evanw/esbuild/pull/2067#issuecomment-1152399288
  banner: {
    js: `
    import { createRequire } from 'module';
    import { fileURLToPath as urlESMPluginFileURLToPath } from "url";
    import { dirname as pathESMPluginDirname} from "path";
    const __filename = urlESMPluginFileURLToPath(import.meta.url);
    const __dirname = pathESMPluginDirname(urlESMPluginFileURLToPath(import.meta.url));
    const require = createRequire(import.meta.url);
    `,
  },
  bundle: true,
});

[Conduitry]

With the now-latest version of SvelteKit, @sveltejs/kit itself now has prod dependencies that need to be bundled at app build time - it used to be that these dependencies were bundled in @sveltejs/kit itself - so my above workaround doesn't work anymore. The same would presumably happen with other dev dependencies that have prod dependencies.

My new workaround is something that I think did not used to be possible in Vite 2 but which is now possible in Vite 3. Rather than telling Vite which dependencies to bundle, I tell it to bundle all dependencies, but exclude the ones that are prod dependencies of the app:

import { sveltekit } from '@sveltejs/kit/vite';
import { readFileSync } from 'fs';

const pkg = JSON.parse(readFileSync('package.json' ,'utf8'));

export default ({ command }) => ({
	plugins: [sveltekit()],
	ssr: command === 'build' && {
		external: Object.keys(pkg.dependencies),
		noExternal: true,
	},
});

[SylvainGarrigues]

Issue https://github.com/sveltejs/kit/issues/5944 was marked as duplicate, yet I found out recent version of sveltekit and adapter-node makes the build dependent on the devalue package. Installing it solves the issue.

[kevpye-fabdata]

With the now-latest version of SvelteKit, @sveltejs/kit itself now has prod dependencies that need to be bundled at app build time - it used to be that these dependencies were bundled in @sveltejs/kit itself - so my above workaround doesn't work anymore. The same would presumably happen with other dev dependencies that have prod dependencies.

My new workaround is something that I think did not used to be possible in Vite 2 but which is now possible in Vite 3. Rather than telling Vite which dependencies to bundle, I tell it to bundle all dependencies, but exclude the ones that are prod dependencies of the app:

export default ({ command }) => ({
	plugins: [sveltekit()],
	ssr: command === 'build' && {
		external: Object.keys(pkg.dependencies),
		noExternal: true,
	},
})

Hi @Conduitry in your work around you are getting the keys of pkg.dependencies but how are you defining or importing pkg? Are you just manually reading the dependencies from package.json or is this a special vite / node import to list them?

[Conduitry]

Nope, I'm just manually reading the file. I've updated my comment to give a more complete example.

[kevpye-fabdata]

Awesome, thank you for the quick reply @Conduitry. I assumed that was the case - but wanted to double check just in case something else was happening.

Do we have any idea when this workaround may be able to be reverted and sveltekit / adapter-node will either bundle these required dependencies or some alternative solution?

For now I have tested this solution and it works well - well, once I moved some other dependencies that were breaking this workaround from dev to production in my package.json. Which dependencies go in which in package.json section is a whole other topic :)

[fmaclen]

I tried @Conduitry's suggestion but I'm still getting Cannot find package 'devaule', 'cookie', 'set-cookie-parser' and so on.

Also, I have some /foo.json/+server.ts endpoints where I'm importing { json } from '@sveltejs/kit' and when I hit those it also complains that it Cannot find package '@sveltejs/kit'.

My current versions:

vite: 3.0.9
@sveltejs/kit: 1.0.0-next.427
@sveltejs/adapter-node: 1.0.0-next.86

Edit: As others pointed out, adding devalue, cookie and set-cookie-parser as production dependencies and running npm ci --prod fixes the issues, including the import { json } from '@sveltejs/kit' error I mentioned above.

[moisesbites]

Edit: As others pointed out, adding devalue, cookie and set-cookie-parser as production dependencies and running npm ci --prod fixes the issues, including the import { json } from '@sveltejs/kit' error I mentioned above.

I think the process of build need to add adapter node dependencies in production. It' very confusing if every time this dependencies was changed, we would need to start the app in production to discover that some packages is not there.

[Rich-Harris]

Opened a PR that adds @Conduitry's solution to the default Vite config https://github.com/sveltejs/kit/pull/6301

[benmccann]

@fmaclen can you share a reproduction on the latest SvelteKit? I just tried @Conduitry's suggestion and it works fine for me

[fmaclen]

@benmccann just tried @Conduitry's solution on my project again (with the latest version and the one I was using when I posted my comment) and both work as expected so the error was probably caused by some other issue specific to my repo 👍

[moisesbites]

Edit: As others pointed out, adding devalue, cookie and set-cookie-parser as production dependencies and running npm ci --prod fixes the issues, including the import { json } from '@sveltejs/kit' error I mentioned above.

I think the process of build need to add adapter node dependencies in production. It' very confusing if every time this dependencies was changed, we would need to start the app in production to discover that some packages is not there.

If I install devalue, when run builded occurs:

file:///home/node/app/build/server/chunks/index.js:3
import devalue from "devalue";
       ^^^^^^^
SyntaxError: The requested module 'devalue' does not provide an export named 'default'

If I not install, the App don't work because it don't find devalue.

Would someone help me?

[vboulaye]

@moisesbites you should pin the devalue version to 2.0.1 (last version is 3.0.0 but the upgrade as not yet been done in kit)

here are the version that I used with a successful build yesterday :

    "cookie": "^0.5.0",
    "devalue": "2.0.1",
    "set-cookie-parser": "^2.5.1",

[moisesbites]

"devalue": "2.0.1",

@vboulaye Thank you!

It would be nice for the adapter to keep the libraries up to date with the latest versions, as for now the build installation has been delegated to the developer at the end. That would be less confusing, at least for now.

[Conduitry]

This should be good to go now in 1.0.0-next.88 of the Node adapter. When you update, you should remove any workarounds you added for this issue, including my ssr Vite config and including adding SvelteKit's runtime dependencies as direct prod dependencies of your app.