Include non-imported assets in deployments

[Rich-Harris]

Describe the problem

A reasonably common pattern is for an endpoint to depend on files in the project directory that aren't visible to Vite because there's no import declaration involved — for example database.sqlite or documentation/getting-started.md. These endpoints work during development and in local preview, but as soon as the app is deployed they break, because the files are not included in the build.

Describe the proposed solution

Something like this:

import marked from 'marked';

// src/routes/documentation/[slug].js
export async function get({ params }) {
  const markdown = await import.meta.load(`documentation/${params.slug}.md`);

  return {
    body: marked(markdown.toString())
  };
}

This uses import.meta.load because it's easy to statically analyse and replace with an environment-appropriate implementation. In this case, we see that it's called with the expression `documentation/${params.slug}.md`, and we can easily turn that into a list of files in the project directory that match /documentation\/.+\.md/. An adapter creating lambdas can simply include those files in the function directory and provide a promisified fs.readFile implementation; an adapter creating a Cloudflare Worker could put the file in a KV store and provide an implementation that reads from it.

In some cases, an adapter might not have a way to read those files, and would be able to throw an error at build time rather than failing cryptically at runtime.

We can restrict the expressions that import.meta.load accepts to template literals and string concatenations.

Slightly related: #3850

Alternatives considered

Next.js solves this problem with the (unfortunately-named!) nft package. It statically analyses the build output and figures out which files are needed, including dependencies in node_modules that weren't included in the bundle and assets that are read from the filesystem.

Since static analysis is inherently limited, code needs to be written a certain way for files to be correctly located. nft's static analysis is very good — you can do this...

const cwd = process.cwd();
const file = path.resolve(cwd, `documentation/${slug}.md`);

...and get a list of dependencies that includes all the .md files in documentation. But you can't do this...

const contents = fs.readFileSync(`documentation/${slug}.md`);

...or this...

import { cwd } from './shared.js';
const file = path.resolve(cwd, `documentation/${slug}.md`);

...or this:

import { read } from './utils.js';
const contents = read(`documentation/${slug}.md`);

In addition, since we're using things like path and process, this can only work in Node-like environments, when ideally a SvelteKit app should happily render in an edge function.

This leads me to the conclusion that we're better off using something less idiomatic that will only work if you adhere to the 'rules', and which could theoretically be used across different environments.

It might still be useful to use nft to find external node_modules (in particular native dependencies that can't be bundled by Vite), so that they can be included in lambdas without further configuration. But that's a separate issue.

---

Another possibility is to use some sort of explicit configuration. Aside from being something of a burden (that doesn't solve the cross-environment problem), it would be difficult to come up with an API that works with function splitting (and doesn't result in all your documentation/*.md files being bundled with unrelated routes).

---

Finally, because someone is going to suggest it, I don't think we should do this:

import { load } from '$app/fs';

const db = await load('database.sqlite');

This takes us into an uncanny valley where load is sort of a normal function but not really — it looks like you should be able to memoize it or pass it somewhere as an argument, but you can't (at least not without breaking the static analysis guarantees we need).

[Rich-Harris]

One thought: if Vite were to start using import.meta.load for something else, it could cause a problem. We could namespace it as import.meta.sveltekit.load but that feels slightly excessive.

[styfle]

How about new URL('./asset.txt', import.meta.url) which is supported by @vercel/nft and other static analysis tools?

[Rich-Harris]

Another suggestion in Discord was to use Vite's existing import.meta.globEager:

+import { load } from '$app/fs';
import marked from 'marked';

+const files = import.meta.globEager('../../documentation/*.md?url');
+
// src/routes/documentation/[slug].js
export async function get({ params }) {
-  const markdown = await import.meta.load(`documentation/${params.slug}.md`);
+  const file = files[`../../documentation/${params.slug}.md?url`];
+  const markdown = await load(file);

  return {
    body: marked(markdown.toString())
  };
}

Pros:

• Files can be transformed by Vite plugins
• Uses existing idioms
• Relative paths are nicer in some situations

Cons:

• More verbose
• Relative paths are less nice in some situations

[bluwy]

To add on the cons, a glob having ?url probably won't work as Vite treats it as a glob pattern naively. This could be improved in Vite though. Also, aliases should work too, like import.meta.glob('$lib/docs/*.md')

[shinokada]

I just wanted to mention that Vite features provides importing different type of files in different format.

For example, you can import JSON files directly:

// import the entire object
import json from './example.json'
// import a root field as named exports - helps with tree-shaking!
import { field } from './example.json'

Load assets as strings:

// Load assets as strings
import assetAsString from './shader.glsl?raw'

Vite also supports importing multiple modules from the file system via import.meta.glob function:

const modules = import.meta.glob('./dir/*.js')

And all work on the current SvelteKit.

[janosh]

Would be great if assets in endpoint dirs were automatically added to the bundle. If I have different banner.svgs for all routes in a directory and the +layout.svelte has a

<img src="{slug}/banner.svg" alt="banner" />

it works in dev but errors in build.

[olmohake]

Has there been any progress on this? I am currently generating docs and news from markdown files. Works perfectly in dev but ran into the same issues once deploying to Vercel.

[PH4NTOMiki]

yeah