Cannot open file `app_engine_key.proto` after build with esbuild

[giangm9]

I was making a Google AppEngine project. I use the google app datastore as the following code

import { Datastore } from "@google-cloud/datastore";

then bundle using https://esbuild.github.io/ After that, an error occurred

  + node --enable-source-maps server.js dev
Error: ENOENT: no such file or directory, open '/home/gl/protos/app_engine_key.proto'
    at Object.openSync (fs.js:476:3)
    at Object.readFileSync (fs.js:377:35)

Environment details

• OS: Ubuntu 20.0 (wsl)
• Node.js version: 14
• npm version: 6.14.10
• @google-cloud/datastore version: 6.3.1

Steps to reproduce

1. esbuild src/server/index.tsx --minify --bundle --jsx-factory=h --log-level=error --color=true --sourcemap --platform=node --outfile=server.js
2. node --enable-source-maps server.js dev

[stephenplusplus]

Hi @giangm9. Thank you for reporting, and sorry for the delay. This module, and others that we use in our client libraries, require .proto files. These are included with the module in the published npm package. From the module's source files, the proto files are located with Node's built-in directory traversing mechanisms, such as the magic variable __dirname. That becomes an issue with esbuild, because when it bundles all of the source code files into one, it removes the directory hierarchy that our modules expect to exist.

Unfortunately, I'm not very familiar with esbuild-- do you know if there's a way to work around this type of problem?

[stephenplusplus]

@bcoe do you know if we are able to support this type of use?

[bcoe]

@alexander-fenster @summer-ji-eng is this a use case that's supported now, by virtue of us shipping JSON versions of protos files (am I imagining that we made this change?).

[agektmr]

I'm bumping into this exact issue. I'm using webpack + typescript. I copied /protos/app_engine_key.proto separately to the project root folder, but no luck. I imagine my webpack configuration lacks something but I'm not too familiar with how it works. Any advice?

[agektmr]

I just realized I don't need to bundle server side code. Please ignore my question.

[crwilcox]

@bcoe @alexander-fenster ?

[bcoe]

@crwilcox @giangm9 while it can be possible to get things working, with some workarounds, we do not currently support bundlers in our gRPC libraries.

There's a feature tracking thread here it's something we very much would like to support.

[bcoe]

In the meantime, you might have better luck with the REST client.

[shane-lamb]

Hi all, wanted to share the solution I'm using, which is to exclude the @google-cloud/datastore package from bundling altogether: https://esbuild.github.io/api/#external

Then, you just need to include the @google-cloud/datastore dependency in your package.json that you deploy to the cloud function (as it's obviously not part of the bundle anymore)

This way you don't have to disable bundling for your other packages, so it's a nice middle ground.

Cheers!

[joosttk]

Same issue when using NextJS (v13). Anyone any idea how to resolve? Looks like now I just cannot use Datastore with NextJS.

[b4lk0n]

hey folks - I've faced such issue in Next.js 13.4 The solution is to update the next.config.js:

const nextConfig = {
  experimental: {
    // don't bundle the package
    serverComponentsExternalPackages: ['@google-cloud/datastore'],
    // enable alpha server actions
    serverActions: true,
  },  
}

And explicitly specify that a page or a component are server-side by adding 'use server'

cc @joosttk

[anestesya]

@b4lk0n works for me!

[meredithslota]

Closing this out as the workaround seems to be effective. Re: web support, we are considering this across our Node.js portfolio of libraries and are unlikely to do it with just one library. Please follow this issue for the overall project: https://github.com/googleapis/google-cloud-node/issues/2933