prisma icon indicating copy to clipboard operation
prisma copied to clipboard

Spike in `Unable to start a transaction in the given time.` for new `prisma-client`

Open iamnafets opened this issue 4 months ago • 30 comments

Bug description

We're experiencing a spike in errors like this after migrating to the new prisma-client and driverAdapters:

Invalid `prisma.pageView.createMany()` invocation: Transaction API error: Unable to start a transaction in the given time.

It doesn't effect 99% of our requests but crashes those that do get this error. We were not experiencing this error with prisma-client-js.

Severity

🚨 Critical: Data loss, app crash, security issue

Reproduction

Do not have a stable reproduction.

Expected vs. Actual Behavior

No errors.

Frequency

Intermittent / Random

Does this occur in development or production?

Only in development (e.g., CLI tools, migrations, Prisma Studio)

Is this a regression?

Worked in 6.9.0, broken in 6.14.0.

Workaround

No workaround ATM.

Prisma Schema & Queries

model PageView {
  id        String   @id @default(cuid())
  userId    String?
  sessionId String?
  pathName  String
  referrer  String
  timestamp DateTime

  @@index([userId])
  @@index([sessionId])
  @@index([timestamp])
  @@index([pathName])
}
                        pageViews.push({
                            userId: typedPayload.user.userID,
                            sessionId: typedPayload.user.customIDs?.stableID,
                            pathName: typedPayload.metadata.pathname,
                            referrer: typedPayload.metadata.referrer,
                            timestamp: DateTime.fromMillis(typedPayload.timestamp).toJSDate(),
                        });
...

                    await prisma().pageView.createMany({
                        data: pageViews
                    });

Prisma Config

No prisma config.

Logs & Debug Info

// Debug logs here

Environment & Setup

  • OS: Debian
  • Database: Postgres / Aurora Serverless RDS
  • Node.js version: 22.18.0

Prisma Version

6.14.0

iamnafets avatar Aug 28 '25 14:08 iamnafets

Hi @iamnafets, can you please post your generator and datasource blocks from your schema.prisma file, as well as the snippet you're currently using to initialize your PrismaClient instance?

Also, does the same problem arise when using prisma-client without driverAdapters?

Thanks.

jkomyno avatar Aug 29 '25 10:08 jkomyno

Can confirm massive transaction timeout spikes (both, starting transactions and executing within timeout) since using the new adapters without Rust binary. Just reverted to previous setup, as not enough time to evaluate things currently.

generator client {
// changed from prisma-client-js
  provider        = "prisma-client"
// changed from Rust binary
  previewFeatures = ["relationJoins", "queryCompiler", "driverAdapters"]
// changed from node_modules
  output          = "./src/generated"
}

datasource db {
  provider = "postgresql"
  url      = env("DATABASE_URL")
}

Init:

export function makePrismaClient({
  clientOptions,
  databaseUrl,
}: MakePrismaClientOptions) {
  const adapter = new PrismaPg({
    connectionString: databaseUrl,
  });
  return new PrismaClient({
    ...clientOptions,
    adapter,
  });
}

nickluger avatar Sep 07 '25 06:09 nickluger

Hi @iamnafets, can you please post your generator and datasource blocks from your schema.prisma file, as well as the snippet you're currently using to initialize your PrismaClient instance?

Also, does the same problem arise when using prisma-client without driverAdapters?

Sure thing!

generator client {
  provider        = "prisma-client"
  output          = "../client"
  previewFeatures = ["relationJoins", "queryCompiler", "driverAdapters"]
  // previewFeatures = ["relationJoins", "postgresqlExtensions"]
  runtime         = "nodejs"
  moduleFormat    = "esm"
}

datasource db {
  provider = "postgresql"
  url      = env("DATABASE_URL")
  // Extension handled in migrations; no automatic creation
}
function createPrismaClient(url?: string) {
  const p = new RawPrismaClient({
    // datasources: {
    //   db: {
    //     url: url ?? "", // This will be undefined during builds
    //   },
    // },
    adapter: new PrismaPg({
      connectionString: url ?? env.DATABASE_URL,
      min: 2,
      idleTimeoutMillis: 5 * 60 * 1000, // 5 minutes to reconnect
      connectionTimeoutMillis: 10 * 1000, // 10 seconds to connect
      maxLifetimeSeconds: 15 * 60, // 15 minutes
      max: 10,
    }),
    log:
      // ["error","warn","query"] // For query logging
      (env.NODE_ENV === "development" ? ["error", "warn"] : ["error"]) as Array<Prisma.LogLevel>,
  }).$extends({
    name: "Extensions",
    result: {
      completion: {
        ast: {
          needs: { ast: true },
          compute(completion) {
            return completion.ast as PromptASTNode[];
          }
        },
        inputs: {
          needs: { inputs: true },
          compute(completion) {
            return completion.inputs as PromptRecord;
          }
        }
      },
      mockSatisfactionSurvey: {
        variableQuestions: {
          needs: { variableQuestions: true },
          compute(survey) {
            return survey.variableQuestions as SessionSatisfactionSurveyVariableQuestions;
          }
        }
      },
      // ... More extensions ...
    }
  });

  return p;
}


export type PrismaClient = ReturnType<typeof createPrismaClient>;

The problem indeed happens regardless of whether we're using driver adapters, switched to driver adapters to see if we could fix the issue.

iamnafets avatar Sep 07 '25 15:09 iamnafets

The problem indeed happens regardless of whether we're using driver adapters

So, to properly understand:

  • You're experiencing transaction errors on Postgres RDS even when using the "old" Rust query engine
  • You're only experiencing this problem with prisma-client, not prisma-client-js
  • You've only noticed this problem in 9.14.0

This seems quite odd.

Can you please try again with the following configurations, and report back with the result?

  1. [email protected] + prisma-client-js + previewFeatures = ["relationJoins"]
// ./schema.prisma

generator client {
  provider        = "prisma-client-js"
  previewFeatures = ["relationJoins"]
  output          = "./src/generated/prisma"
}
// ./src/db.ts
import { PrismaClient } from './generated/prisma/client';

export function makePrismaClient({
  clientOptions,
}: MakePrismaClientOptions) {
  return new PrismaClient({
    ...clientOptions,
  });
}
pnpm prisma generate
  1. [email protected] + prisma-client + previewFeatures = ["relationJoins"]
// ./schema.prisma

generator client {
  provider        = "prisma-client"
  previewFeatures = ["relationJoins"]
  output          = "./src/generated/prisma"
}
// ./src/db.ts
import { PrismaClient } from './generated/prisma/client';

export function makePrismaClient({
  clientOptions,
}: MakePrismaClientOptions) {
  return new PrismaClient({
    ...clientOptions,
  });
}

Can you also try again with Driver Adapters without specifying custom input parameters to PrismaPg, besides the connection string?

const adapter: new PrismaPg({
  connectionString: url ?? env.DATABASE_URL,
})

Thanks!

jkomyno avatar Sep 08 '25 08:09 jkomyno

I can confirm the issue persists without any custom parameters to the PrismaPg adapter. I ran that before attempting to solve the problem with these parameters.

Is there a sensible way for me to hook telemetry to these specific errors? I want to make sure I'm giving you something data-driven so I'd love to put a prometheus metric on these transaction-level events so I can track before and after.

iamnafets avatar Sep 08 '25 16:09 iamnafets

The same issue prisma 6.16.2

Kila14 avatar Sep 17 '25 04:09 Kila14

same issue here. only started happening once we switched to the new rustfree prisma engine

aroman avatar Sep 23 '25 06:09 aroman

Also seeing this issue with new Rust-free engine, we saw a massive spike in Unable to start a transaction in the given time after moving to the client.

We pass in the same options regardless of engine:

const options: Prisma.PrismaClientOptions = {
    transactionOptions: {
      maxWait: db.transactionMaxWait, // set to 4 seconds in our environment
      timeout: db.transactionTimeout, // set to 30 seconds in our environment
    }
};

const unextendedClient = new PrismaClient(options);

We rolled back to the Rust engine and this issue disappears. We have no issue with other transaction timeouts in either engine. (that is, timeout does not appear to be an issue, but maxWait is an issue) Prisma 6.16.2 with Node 22.20.

sosafe-robert-mcauley avatar Sep 30 '25 09:09 sosafe-robert-mcauley

I am experiencing the same problem. I have rolled back too.

sepiaflux avatar Oct 05 '25 17:10 sepiaflux

has anyone managed to find a mitigation/workaround other than downgrading?

aroman avatar Oct 05 '25 17:10 aroman

Here to pile into the "rust free is slower" train as the Discord Chatbot sent me here. 🚂🤖

On a local Postgres database I was getting a massive number of transaction timeout messages running out seed script. Ended up with this config to avoid those.

new PrismaClient({
      transactionOptions: {
        maxWait: 300000,
        timeout: 300000,
      },
      log: process.env.HIDE_PRISMA_QUERY === "true" ? [] : ["query"],
      adapter: new PrismaPg({
        connectionString: getDBUrl(),
        idle_in_transaction_session_timeout: 15000,
        statement_timeout: 15000,
        query_timeout: 15000,
        connectionTimeoutMillis: 15000,
      }),
    })

Rust + Non-ESM - Seeding Complete in 166.956seconds Rust + ESM - Seeding Complete in 149.892 seconds Rust Free + ESM - Seeding Complete in 557.726 seconds

Thats from just changing

generator client {
  provider        = "prisma-client"
  output          = "../../prisma-generated"
  previewFeatures = ["views"]
  binaryTargets   = ["native", "rhel-openssl-3.0.x"]
}

to this

generator client {
  provider        = "prisma-client"
  output          = "../../prisma-generated"
  previewFeatures = ["views"]
  engineType      = "client"
}

and running npx prisma generate before the seed script.

Our seed script uses functions from our application code, with seed data and some use of fakerjs. That is a lot of prisma upsert statements with deep nesting, and deep returns. That we then run synchronously with Promise.all([..]);

Excited for the DX improvement from switching to the ESM, but doesn't look like the Rust-Free engine is ready for us. We would be happy to provide to the Prisma whatever could help profile this issue.

gschechter avatar Oct 06 '25 01:10 gschechter

Here to pile into the "rust free is slower" train as the Discord Chatbot sent me here. 🚂🤖

I am also seeing this in my team's unit test suite. Switching from Rust to ESM, similar small change as yours in the generator, our team is seeing a 20-30% increase in time taken to run our suite. (run variance does happen, but the percentage increase is in the same ballpark across various Mac and Windows laptops)

We are definitely not switching to Rust-free anytime soon as a result. We simply cannot trust what the Prisma team has released here. :(

I am not sure if this is the Github issue for "ESM is slower in general" though - I hope this bug stays focused on the 'unable to start a transaction' issue.

sosafe-robert-mcauley avatar Oct 06 '25 07:10 sosafe-robert-mcauley

Not moving anytime soon neither due to issues like this, plus queries seems slower contrary to reported benchmarks. Seems not as production-ready as the Rust version.

nickluger avatar Oct 06 '25 08:10 nickluger

Is there a fix for that? It happens to us as well

dikaioai avatar Oct 06 '25 15:10 dikaioai

Confirming this is happening to me as well.

TaylorFacen avatar Oct 07 '25 01:10 TaylorFacen

  const pool = new Pool({
    connectionString: process.env.DATABASE_URL,
    max: Number(process.env.DATABASE_CONNECTION_LIMIT) || 1,
    ssl: { rejectUnauthorized: false },
  });

  const adapter = new PrismaPg(pool, {
    schema: getSchema(),
  });

  return new PrismaClient({
    transactionOptions: {
      maxWait: 300000,
      timeout: 300000,
    },
    adapter,
    log: ['info', 'error'],
  });

maybe this solves the issue, not the best solution though and very hard to test it as it does not happen frequently, only in production because of concurrent requests

dikaioai avatar Oct 07 '25 09:10 dikaioai

const pool = new Pool({ connectionString: process.env.DATABASE_URL, max: Number(process.env.DATABASE_CONNECTION_LIMIT) || 1, ssl: { rejectUnauthorized: false }, });

const adapter = new PrismaPg(pool, { schema: getSchema(), });

return new PrismaClient({ transactionOptions: { maxWait: 300000, timeout: 300000, }, adapter, log: ['info', 'error'], }); maybe this solves the issue, not the best solution though and very hard to test it as it does not happen frequently, only in production because of concurrent requests

would this not run the risk of deadlocking the client, as all connections could be 'stuck' in those long-running transactions? i don't really understand what the underlying cause of this regression is (besides that it's clearly an issue with the rustfree engine)

aroman avatar Oct 07 '25 09:10 aroman

same issue :/

ozgurozalp avatar Oct 15 '25 06:10 ozgurozalp

Same issue over here on 6.17.1

HubbardJacob avatar Oct 20 '25 15:10 HubbardJacob

Any solution from prisma team? Not just increase timeouts

Kila14 avatar Oct 21 '25 12:10 Kila14

We are also experiencing this issue with 6.17.1 using the Rust-free engine.

yharaskrik avatar Oct 21 '25 22:10 yharaskrik

Related: https://github.com/prisma/prisma/issues/28152

yharaskrik avatar Oct 21 '25 22:10 yharaskrik

We are also experiencing this issue in 6.18.0 and node 24.

Sriram-52 avatar Nov 11 '25 16:11 Sriram-52

apparently it should've been fixed in 6.18

https://x.com/prisma/status/1988292899243782489

SupertigerDev avatar Nov 12 '25 09:11 SupertigerDev

Hi @iamnafets @yharaskrik @Kila14 @HubbardJacob @ozgurozalp @TaylorFacen @sosafe-robert-mcauley , We've released a fix that should resolve some cases of this error in version 6.18.0. Can you let us know if you're still seeing this issue on Prisma 6.18.0 or newer? Thanks

jacek-prisma avatar Nov 12 '25 09:11 jacek-prisma

FWIW we are also seeing this in 6.18 / Node 24 (using the Rust-free engine + pg adapter)

egoldschmidt avatar Nov 13 '25 18:11 egoldschmidt

I don't have load testing tools in my staging environment to test a potential fix for this issue, so I am hesitant to deploy a new version that may or may not fix the issue. I do not want to risk pushing to production and having another outage incident. I'll have to wait until others have tested and confirmed that it has been fixed before we try the new engine.

sosafe-robert-mcauley avatar Nov 14 '25 12:11 sosafe-robert-mcauley

We are also seeing this in 7.0.1 / Node 22 (Rust-free engine + pg adapter)

ihadabs avatar Nov 30 '25 10:11 ihadabs

maybe unrelated as we are using Azure SQL Db, but after migrating to 6.19 and "rust free" we get weird connection errors, after some time looks like connection pooler gets stuck and we see lots of:

2025-12-02T13:32:25.6619785Z Unhandled promise rejection at: Promise {
2025-12-02T13:32:25.6621007Z   <rejected> TransactionError: Can't rollback transaction. There is a request in progress.

2025-12-02T13:32:49.7728766Z TimeoutError: operation timed out for an unknown reason
2025-12-02T13:32:49.7729695Z     at /usr/app/dist/node_modules/tarn/dist/PendingOperation.js:17:27 {
2025-12-02T13:32:49.7729774Z   clientVersion: '6.19.0'
2025-12-02T13:32:49.7729851Z }

errors.. Reverting back to rust engine for now.

jsmircic avatar Dec 02 '25 14:12 jsmircic

We attempted to implement the Rust-free upgrade in a staging environment for a client, but it proved to be entirely ineffective. The upgrade introduced numerous issues, including a persistent spike in database connections reaching approximately 170. This spike, coupled with slow read and write operations to the database, rendered the upgrade unusable. Consequently, we have decided to downgrade the system to version 6.5 for the time being.

Sriram-52 avatar Dec 09 '25 07:12 Sriram-52