Doesn't work with next.js Edge runtime

[etodanik]

I'm writing an app on next.js which I plan to deploy to CloudFlare Pages + Workers. I know there's CloudFlare support, but when running the local development server of next.js, I get the following errors:

Learn More: https://nextjs.org/docs/messages/node-module-in-edge-runtime
    at <unknown> (webpack-internal:///(middleware)/./node_modules/next/dist/esm/server/web/globals.js:33)
    at Object.get (webpack-internal:///(middleware)/./node_modules/next/dist/esm/server/web/globals.js:33:19)
    at eval (webpack-internal:///(middleware)/./node_modules/pg/lib/crypto/utils-webcrypto.js:19:36)
    at (middleware)/./node_modules/pg/lib/crypto/utils-webcrypto.js (file:///home/danny/src/project/.next/server/pages/api/manageUser.js:3097:1)
    at __webpack_require__ (file:///home/danny/src/project/.next/server/edge-runtime-webpack.js:37:33)
    at fn (file:///home/danny/src/project/.next/server/edge-runtime-webpack.js:286:21)
    at eval (webpack-internal:///(middleware)/./node_modules/pg/lib/crypto/utils.js:7:22)
    at (middleware)/./node_modules/pg/lib/crypto/utils.js (file:///home/danny/src/project/.next/server/pages/api/manageUser.js:3108:1)
    at __webpack_require__ (file:///home/danny/src/project/.next/server/edge-runtime-webpack.js:37:33)
    at fn (file:///home/danny/src/project/.next/server/edge-runtime-webpack.js:286:21)
    at eval (webpack-internal:///(middleware)/./node_modules/pg/lib/crypto/sasl.js:3:16)

Would it make sense to generalize CloudFlare Workers detection into a broader Edge detection that also includes node edge runtime?

[brianc]

Yeah it would probably...how does one reproduce this issue locally? If running in node then pg should be using the node pieces.

[etodanik]

Set up a basic next.js app, create an API endpoint, set its' runtime to edge by specifying: export const runtime = 'edge';

And then assume to use node-postgres from within that route.

CloudFlare Pages requires a next.js app to specify that all backend endpoints will be using the edge runtime.

There's a further complication that NodeEdge runtime isn't the same as CloudFlare workers. I suppose next.js' NodeEdge is emulating what Vercel's edge runtime would look like. But one important detail is that it won't have the ability to run cloudflare:sockets

[muhammedaksam]

I am trying to use Prisma, PrismaPg and Pg.

Error: The edge runtime does not support Node.js 'crypto' module.

Call Stack

Next.js
eval
node_modules\.pnpm\[email protected]\node_modules\pg\lib\crypto\utils-webcrypto.js (21:1)
(middleware)/./node_modules/.pnpm/[email protected]/node_modules/pg/lib/crypto/utils-webcrypto.js
file:///C:/Code/React/nextplate/.next/server/middleware.js (1292:1)

Next.js
eval
node_modules\.pnpm\[email protected]\node_modules\pg\lib\crypto\utils.js (8:3)
(middleware)/./node_modules/.pnpm/[email protected]/node_modules/pg/lib/crypto/utils.js
file:///C:/Code/React/nextplate/.next/server/middleware.js (1303:1)

Next.js
eval
node_modules\.pnpm\[email protected]\node_modules\pg\lib\crypto\sasl.js (2:16)

https://vercel.com/docs/functions/runtimes/edge-runtime#unsupported-apis

I know it's not really an issue related to Pg but I wanted to inform here anyway.

[mlntr]

I am using Next.js and Drizzle, and get the same: Error: The edge runtime does not support Node.js 'crypto' module. Learn More: https://nextjs.org/docs/messages/node-module-in-edge-runtime

[andyjy]

This patch fixes for me. Specifically: prevents the error Error: The edge runtime does not support Node.js 'crypto' module. when using pg under the Next.js Edge runtime locally.

pg+8.11.5.patch:

diff --git a/node_modules/pg/lib/crypto/utils-webcrypto.js b/node_modules/pg/lib/crypto/utils-webcrypto.js
index 0433f01..588066e 100644
--- a/node_modules/pg/lib/crypto/utils-webcrypto.js
+++ b/node_modules/pg/lib/crypto/utils-webcrypto.js
@@ -13,7 +13,10 @@ module.exports = {
  * The Web Crypto API - grabbed from the Node.js library or the global
  * @type Crypto
  */
-const webCrypto = nodeCrypto.webcrypto || globalThis.crypto
+
+// PATCH: Don't break under Next.js Edge runtime production build
+const webCrypto = ("webcrypto" in nodeCrypto ? nodeCrypto.webcrypto : undefined) ?? globalThis.crypto
+
 /**
  * The SubtleCrypto API for low level crypto operations.
  * @type SubtleCrypto

(I opted to use runtime-agnostic Feature Detection for webcrypto on the crypto import rather than try detect the Next.js Edge runtime specifically via typeof EdgeRuntime === "string", which I assume could be a valid alternative - haven't verified)

[janpio]

Quick note from @prisma that this affects some of our users and is being reported to us: https://github.com/prisma/prisma/issues/24430

(This issue has a reproduction, but that includes Prisma, Next.js and next-auth - that is probably a bit more than what you are asking for @brianc. I am pretty sure just trying to use pg in a Next.js middleware should have the same result though.)

We would love to see this fixed and released, so that Prisma users wanting to use pg in Next.js projects with middleware can do so. Let us know if we can help in any way.

[brianc]

k i'll get this looked at today if I have time!

[brianc]

@janpio @andyjy @etodanik would one of y'all be able to provide me w/ more concrete steps to follow to set up a local reproduction? I did npx create-next-app@latest and did NOT use the API router (so the legacy or older form of nextjs). It created a hello api route:

So i tested it at http://localhost:3000/api/hello and it worked. great. then I went into the route file and added

export const runtime = 'edge'

and now its erroring when I hit the route:

So I'm doing something wrong, but not sure what.

This is the full contents of my hello.ts api route file:

// Next.js API route support: https://nextjs.org/docs/api-routes/introduction
import type { NextApiRequest, NextApiResponse } from "next";

type Data = {
  name: string;
};

export const runtime = 'edge'

export default function handler(
  req: NextApiRequest,
  res: NextApiResponse<Data>,
) {
  res.status(200).json({ name: "John Doe" });
}

[brianc]

update: I tried a different next-js app using the app router instead of the old pages/* approach

I just did same steps as above except said "yes" when the setup asked me to use an app router.

Setup looks like this:

my route.ts file:

import * as pg from 'pg'

export const runtime = 'edge'
const pool = new pg.Pool()

export async function GET() {
  const { rows } = await pool.query('SELECT 1 as one')

  return Response.json({ rows })
}

Immediately when I run 'yarn next dev' to start the project it errors out with being unable to import pg-native which is annoying....I've never messed w/ the edge runtime before - it's quite a bit different...so any thoughts on this?

Import trace for requested module:
./node_modules/pg/lib/connection-parameters.js
./node_modules/pg/lib/client.js
./node_modules/pg/lib/index.js
./src/app/api/route.ts
./node_modules/next/dist/build/webpack/loaders/next-edge-app-route-loader/index.js?absolutePagePath=%2FUsers%2Fbmc%2Fsrc%2Fpg-edge-test%2Fsrc%2Fapp%2Fapi%2Froute.ts&page=%2Fapi%2Froute&appDirLoader=bmV4dC1hcHAtbG9hZGVyP25hbWU9YXBwJTJGYXBpJTJGcm91dGUmcGFnZT0lMkZhcGklMkZyb3V0ZSZhcHBQYXRocz0mcGFnZVBhdGg9cHJpdmF0ZS1uZXh0LWFwcC1kaXIlMkZhcGklMkZyb3V0ZS50cyZhcHBEaXI9JTJGVXNlcnMlMkZibWMlMkZzcmMlMkZwZy1lZGdlLXRlc3QlMkZzcmMlMkZhcHAmcGFnZUV4dGVuc2lvbnM9dHN4JnBhZ2VFeHRlbnNpb25zPXRzJnBhZ2VFeHRlbnNpb25zPWpzeCZwYWdlRXh0ZW5zaW9ucz1qcyZyb290RGlyPSUyRlVzZXJzJTJGYm1jJTJGc3JjJTJGcGctZWRnZS10ZXN0JmlzRGV2PXRydWUmdHNjb25maWdQYXRoPXRzY29uZmlnLmpzb24mYmFzZVBhdGg9JmFzc2V0UHJlZml4PSZuZXh0Q29uZmlnT3V0cHV0PSZwcmVmZXJyZWRSZWdpb249Jm1pZGRsZXdhcmVDb25maWc9ZTMwJTNEIQ%3D%3D&nextConfigOutput=&preferredRegion=&middlewareConfig=e30%3D!
 ○ Compiling /api ...
 ⚠ ./node_modules/pg/lib/native/client.js
Module not found: Can't resolve 'pg-native' in '/Users/bmc/src/pg-edge-test/node_modules/pg/lib/native'

I know its trying to webpack the server...which is fun.

[andyjy]

@brianc here you go - reproduction for pages router: https://github.com/andyjy/mre-pg-nextjs-edge

[brianc]

here you go - reproduction for pages router:

amazing, ty

[brianc]

@andyjy when i applied your patch to the crypto file I now get this:

⨯ Error: The edge runtime does not support Node.js 'net' module.
Learn More: https://nextjs.org/docs/messages/node-module-in-edge-runtime
    at Object.construct (webpack-internal:///(middleware)/./node_modules/next/dist/esm/server/web/globals.js:37:19)
    at getStream (webpack-internal:///(middleware)/./node_modules/pg/lib/stream.js:8:12)
    at new Connection (webpack-internal:///(middleware)/./node_modules/pg/lib/connection.js:19:36)
    at new Client (webpack-internal:///(middleware)/./node_modules/pg/lib/client.js:48:7)
    at handler (webpack-internal:///(middleware)/./src/pages/api/hello.ts:13:16)
    at eval (webpack-internal:///(middleware)/./node_modules/next/dist/esm/server/api-utils/index.js:32:16)
    at eval (webpack-internal:///(middleware)/./node_modules/next/dist/esm/server/lib/trace/tracer.js:115:36)
    at NoopContextManager.with (webpack-internal:///(middleware)/./node_modules/next/dist/compiled/@opentelemetry/api/index.js:2:7062)
    at ContextAPI.with (webpack-internal:///(middleware)/./node_modules/next/dist/compiled/@opentelemetry/api/index.js:2:518)
    at NoopTracer.startActiveSpan (webpack-internal:///(middleware)/./node_modules/next/dist/compiled/@opentelemetry/api/index.js:2:18108)
 ✓ Compiled in 107ms (263 modules)

[andyjy]

@andyjy when i applied your patch to the crypto file I now get this:

~ah ok, turns out I am using import { Pool } from "pg" in my app, which the patch above solves for - whereas in the reproduction repo I happened to write import { Client } from "pg" which exposes the new net incompatibility.~

Update: 🤦‍♂️ sorry, I realised that while the patch I shared above indeed prevents the Next.js local edge runtime compilation error, I'm not actually executing the pg Client under the Edge runtime locally. So the patch only fixes import, not usage.. (yes I know, facepalm..)

(I remembered I'm actually serving my API routes that use pg under the Node runtime, although my code that references pg is imported - albeit never executed - within some of my Edge runtime routes, which is why I produced the patch.)

Sorry for the incorrect/misleading post previously!

Thinking ahead to whether/how it's possible to get things actually working under the Next.js Edge runtime locally - it would probably(?) require breaking out of Next.js' local Edge Runtime "sandbox" to access node:net locally (i.e. via the Next.js Webpack config). So if this is the case, some tweaks to pg may be necessary but not sufficient alone.

If I get an actually-working prototype then I'll share back.

[janpio]

Tiny PR to @andyjy's repro to show the same problem with a Next.js middleware, the common case where Prisma users run into this: https://github.com/andyjy/mre-pg-nextjs-edge/pull/1

Thinking ahead to whether/how it's possible to get things actually working under the Next.js Edge runtime locally - it would probably(?) require breaking out of Next.js' local Edge Runtime "sandbox" to access node:net locally (i.e. via the Next.js Webpack config). So if this is the case, some tweaks to pg may be necessary but not sufficient alone.

Semi informed comment from the sidelines: Afaik the Cloudflare Workers support for pg had similar challenges, that parts of the internal pg code had to be disabled or skipped to not use APIs that are not available in the CfW environment.

[brianc]

Sorry for the incorrect/misleading post previously!

absolutely no problem! I'll continue to noodle on this....might take a bit of time to do a similar retrofit that the CF stuff did. I'm also about to go out of town for a bit so...hang tight - i'll get there eventually!

[janpio]

🤔 Can node-postgres even work in the edge runtime by Vercel and Next.js? Isn't it missing TCP socket support, and Vercel does not offer a connect() like Cloudflare does, that could be used instead. That would be a limitation of Next.js functionality that uses the edge runtime (middleware, app routes) to not be able to support pg at all.

[andyjy]

🤔 Can node-postgres even work in the edge runtime by Vercel and Next.js? Isn't it missing TCP socket support, and Vercel does not offer a connect() like Cloudflare does, that could be used instead. That would be a limitation of Next.js functionality that uses the edge runtime (middleware, app routes) to not be able to support pg at all.

Yes - having looked into it more, I also realised you're essentially correct - that lack of TCP socket support is a blocker for the Next.js/Vercel Edge Runtime.

Despite running on Cloudflare Workers behind the scenes (still ?) - when deploying to Vercel, Edge runtime code that references require("cloudflare:sockets") fails to deploy, with error The Edge Function "middleware" is referencing unsupported modules: cloudflare:sockets.

What I assume is possible is deploying a Next.js app using the Edge runtime directly to Cloudflare. With no Vercel / Edge Runtime wrapper, the bundled code should just work on Cloudflare Workers.

And if someone wishes to do that, it is technically possible to:

1. Patch pg to compile/bundle successfully under Next.js' Webpack config. This requires a bunch of patching.. and additional next.config.js config.

2. Patch Next.js to successfully pg under the Edge runtime (middleware, Pages API routes + App router) under local development (next dev), by breaking through the local Edge sandbox to expose net.Socket(!). Requires a patch and additional next.config.js config.

Potentially handy if anyone really wants to develop using Next.js + pg under Edge before deploying to Cloudflare.. but I can't imagine this ever being supported by Next.js "officially" unless the Edge runtime is somehow updated to incorporate TCP socket support...(!?)

The reproduction repo is now updated with a proof of concept for both (1) and (2). I haven't deployed to Cloudflare directly but see no reason why this wouldn't work (potentially with a bit more tweaking to get the output bundle correct)

/nerd-snipe over

[janpio]

Thanks for the validation of my thought, I was not sure if I was really mixing things up now.

Note, that will probably make this a lot more confusing instead of helping, as it makes it more complex: For Cloudflare Pages there is an additional package next-on-pages so one can deploy Next.js to Cloudflare Pages, and that has its own problem with pg: https://github.com/cloudflare/next-on-pages/issues/605 Not sure if the "Edge Runtime" (from Next.js and Vercel) is even involved there anymore though.

Probably means no easy way to use pg in Next.js middlewares though 😢

[andyjy]

@janpio here's the reproduction repo deployed directly to cloudflare via @cloudflare/next-on-pages

.. it works(!) - connecting to a Neon database using node-postgres:

• Pages router API route: https://44d82908.mre-pg-nextjs-edge.pages.dev/api/hello
• App router: https://44d82908.mre-pg-nextjs-edge.pages.dev/api/hello-app
• Edge middleware (you contributed): https://44d82908.mre-pg-nextjs-edge.pages.dev/api/foo

The (currently super ugly..) patches to pg I mentioned above are what fixes https://github.com/cloudflare/next-on-pages/issues/605

🎉

[andyjy]

@janpio ..and here it is deployed to Cloudflare with Prisma added (via @prisma/adapter-pg):

Pages router API route: https://d2ff1a42.mre-pg-nextjs-edge.pages.dev/api/hello-app App router: https://d2ff1a42.mre-pg-nextjs-edge.pages.dev/api/hello-app Edge middleware: https://d2ff1a42.mre-pg-nextjs-edge.pages.dev/api/foo

[janpio]

Ok wow, that is pretty wild @andyjy - amazing!

Do I understand correctly that you seem a way to make pg work on Cloudflare Pages via next-on-pages via some changes? But deploying to Vercel is still completely out of the picture, correct?

[andyjy]

Do I understand correctly that you seem a way to make pg work on Cloudflare Pages via next-on-pages via some changes

Exactly! And specifically those changes are:

1. Patch pg to compile/bundle successfully under Next.js' Webpack config. This requires a bunch of patching.. and additional next.config.js config.

I believe this fixes https://github.com/cloudflare/next-on-pages/issues/605

(The patches linked above are still a bit of a messy proof-of-concept - in particular there are some "unhappy paths" in pg that execute during handling of e.g. connection errors/exceptions that use other unsupported Node APIs e.g. process.nextTick(), that I believe will crash currently, but could probably(?) be switched out for the cross-platform queueMicrotask()(?).

The changes in my patches would also need a bit more love before they can be merged into pg in a clean way and without breaking anything for non-Edge runtimes and non-Next.js bundler configs. Should be very doable but I expect quite a chore unfortunately..)

Re: Vercel:

But deploying to Vercel is still completely out of the picture, correct

Yes - at least, I observe that when pushing bundled code to Vercel that includes require("cloudflare:sockets"), Vercel's deployment process objects with error The Edge Function "...." is referencing unsupported modules: cloudflare:sockets

(It's the same error if I push any code referencing e.g. require("this_module_does_not_exist"))

As I understand it - Vercel's Edge runtime product specifically supports their Edge Runtime API that is designed to be runnable under various V8-powered environments - AFAIK they don't particualrly publicise the fact it's Cloudflare Workers under the hood (and could host their Vercel Edge functions elsewhere under non-Cloudflare at any time - for all I know they may do already!).

So at this point while it may be worth asking a friendly contact at Vercel about the possibility of bypassing the unsupported module: cloudflare:sockets error and running this code on Vercel, even if there is some way to make it work today I still wouldn't be too optimistic about it not breaking in future. The only practical way I can imagine would be if Vercel add support for a TCP socket library to their official Edge Runtime spec.

[janpio]

Shared that workaround over at next-on-pages: https://github.com/cloudflare/next-on-pages/issues/605#issuecomment-2154450280

[brianc]

Hi! I'm back from my trip - will be looking into this tomorrow morning! Thanks for all the help & research...I'll dive in tomorrow with some 👀 and 🔨

[brianc]

ugg...reading up on the edge runtime vm trying to come up w/ a way to write some test for this within the repo. A bit complicated. Definitely open to ideas & help on this or a PR w/ the patch (though yeah I looked at it, it needs a bit more massaging). Thanks for the work so far @andyjy!

[andresgutgon]

Same issues here using Drizzle with node-postgres and next-auth which use as DB adapter drizzle/pg. What I did is to do a dummy next-auth adapter stripping DB

import { AUTH_MAX_AGE } from '$/auth/constants';
import Google from 'next-auth/providers/google'
import NextAuth from 'next-auth'
import { NextResponse } from 'next/server';
import { NextAuthRequest } from 'node_modules/next-auth/lib';

// Make here the dummny next-auth without PostgreSQL dependency
const { auth } = NextAuth({
  session: { strategy: 'jwt', maxAge: AUTH_MAX_AGE },
  providers: [Google]
})

export default auth((_req: NextAuthRequest) => {
  return NextResponse.next();
})

export const config = {
  matcher: ["/((?!api|_next/static|_next/image|favicon.ico).*)"],
}

Not sure if this works but at least in development the page load

[muhammedaksam]

Same issues here using Drizzle with node-postgres and next-auth which use as DB adapter drizzle/pg. What I did is to do a dummy next-auth adapter stripping DB

import { AUTH_MAX_AGE } from '$/auth/constants';
import Google from 'next-auth/providers/google'
import NextAuth from 'next-auth'
import { NextResponse } from 'next/server';
import { NextAuthRequest } from 'node_modules/next-auth/lib';

// Make here the dummny next-auth without PostgreSQL dependency
const { auth } = NextAuth({
  session: { strategy: 'jwt', maxAge: AUTH_MAX_AGE },
  providers: [Google]
})

export default auth((_req: NextAuthRequest) => {
  return NextResponse.next();
})

export const config = {
  matcher: ["/((?!api|_next/static|_next/image|favicon.ico).*)"],
}

Not sure if this works but at least in development the page load

If you are going to use JWT as strategy you probably don't need to use an adapter for your Next Auth so you probably don't need to use Pg either.

[andresgutgon]

If you are going to use JWT as strategy you probably don't need to use an adapter for your Next Auth so you probably don't need to use Pg either.

I still need to use PG adapter for storing users and accounts

[vnce]

For those hitting this issue as part of a next-auth / auth js 5 integration, this document helped me understand the situation better: https://authjs.dev/guides/edge-compatibility

""" PostgreSQL is a database that uses a message-based protocol for communication between a the client and server that is transported via TCP (or Unix) sockets. Raw TCP sockets are one of those Node.js features that are generally not available to edge runtimes. Therefore, on the surface, it seems like it’s not possible to communicate with a PostgreSQL database from JavaScript running on edge runtime. The same goes for many other databases and their respective communication protocols.

As edge runtimes have matured and become more popular, however, people have gotten creative and implemented various solutions to this problem. One such common solution is to put some sort of API server in front of the database whose goal is to translate database queries sent to it via HTTP into a protocol the database can understand. This allows the client side to only have to make HTTP requests to the API server, which is something that every edge runtime supports. """

My workaround was to expose an API to pg that an auth.ts callback can hit from edge. I think I'd still prefer using pg in edge runtime (sounds like Brian is close to a solution), but maybe this is working as intended if node-postgres is designed for the node.js runtime, which Vercel's Edge runtime doesn't (or only partially) uses

[andresgutgon]

My workaround was to expose an API to pg that an auth.ts callback can hit from edge. I think I'd still prefer using pg in edge runtime (sounds like Brian is close to a solution), but maybe this is working as intended if node-postgres is designed for the node.js runtime, which Vercel's Edge runtime doesn't (or only partially) uses

But you still can have middleware if you use jwt and node-postgresql. This worked for me and is explained in the docs here