bun icon indicating copy to clipboard operation
bun copied to clipboard

Segmentation Fault Crash when using Prisma

Open monkfromearth opened this issue 6 months ago • 8 comments

What version of Bun is running?

1.0.20+09d51486e

What platform is your computer?

Linux 5.15.133.1-microsoft-standard-WSL2 x86_64 x86_64

What steps can reproduce the bug?

  • Perform a CPU-intensive task in the same method as where the database insertion can happen
  • Use sharp to transform the image, for example, and try to insert some data

What is the expected behavior?

The insertion should happen normally.

What do you see instead?

error: script was terminated by signal SIGSEGV (Address boundary error) Segmentation fault

Additional information

We tried to transform an image through sharp and store the S3 path to where they were uploaded. We refactored the functions so that the transformation happens in another method. The transformations and uploads would happen normally, but during Prisma insertion, the script crashes directly.

This works on 1.0.15, but not since.

Output from bunx prisma -v:

prisma                  : 5.7.1
@prisma/client          : 5.7.1
Computed binaryTarget   : debian-openssl-1.1.x
Operating System        : linux
Architecture            : x64
Node.js                 : v18.17.0
Query Engine (Node-API) : libquery-engine 0ca5ccbcfa6bdc81c003cf549abe4269f59c41e5 (at node_modules/@prisma/engines/libquery_engine-debian-openssl-1.1.x.so.node)
Schema Engine           : schema-engine-cli 0ca5ccbcfa6bdc81c003cf549abe4269f59c41e5 (at node_modules/@prisma/engines/schema-engine-debian-openssl-1.1.x)
Schema Wasm             : @prisma/prisma-schema-wasm 5.7.1-1.0ca5ccbcfa6bdc81c003cf549abe4269f59c41e5
Default Engines Hash    : 0ca5ccbcfa6bdc81c003cf549abe4269f59c41e5
Studio                  : 0.495.0

monkfromearth avatar Dec 27 '23 10:12 monkfromearth

sorry

to any contributors: without seeing the stack trace, my guess is it's an async context thing. Prisma uses async hooks + napi. Sharp & Prisma uses threadsafe functions. Threadsafe functions sometimes attempt to read from hooks. It's possible something is going wrong there.

Another thought: maybe it's because we don't implement escapable handle scopes and the inner function is async. This would be reproducible for any async NAPI function (which is truly async, not just microtasks). Something like await Bun.sleep(1) inside a NAPI threadsafe function (such as Sharp or Prisma) would reproduce this.

Jarred-Sumner avatar Dec 27 '23 10:12 Jarred-Sumner

is there documentation for getting a stack trace for these segfaults? i'm running into this issue as well and would like to help if i can

JosephClay avatar Jan 12 '24 15:01 JosephClay

I also experienced this issue without any clues. Similar to the OP, after processing the image using Sharp and uploading it to S3, then the data insertion process is carried out using Prisma and a segvault occurs.

What I can do now to keep my service running is to use bun v1.0.18

Hrdtr avatar Jan 12 '24 18:01 Hrdtr

I can confirm this bug still present on latest bun 1.0.23

rizrmd avatar Jan 16 '24 13:01 rizrmd

This bug still exists on ~~1.0.25~~ 1.0.26. If you want to use prisma in bun, v1.0.18 is your best bet.

rizrmd avatar Jan 21 '24 03:01 rizrmd

I can confirm I was seeing this in bun v1.0.26. Downgrading to v1.0.18 with the following command (on Linux) seems to have made the problem go away for now...

curl -fsSL https://bun.sh/install | bash -s "bun-v1.0.18"

jontybrook avatar Feb 06 '24 20:02 jontybrook

I upgraded from bun v1.0.0 to v1.0.26 and I started getting this error intermittently when starting my server. Downgrading to 1.0.18 made it go away.

Not using prisma. My server code has the following dependencies:

  • bun:sqlite
  • node:fs/promises
  • uuid
  • redis
  • cookie
  • cookie-signatures
  • pizzip
  • docxtemplater

AdrianoFerrari avatar Feb 11 '24 05:02 AdrianoFerrari

Something must gone very wrong in 1.0.19 upwards. I really want to dig it deeper, but... yeah...

rizrmd avatar Feb 11 '24 07:02 rizrmd

1.0.27 released, yay!

Still error, but It is throwing different message than previous versions!!

thread 'tokio-runtime-worker' panicked at /Users/runner/.cargo/registry/src/index.crates.io-6f17d22bba15001f/tokio-1.30.0/src/sync/rwlock.rs:427:17:
internal error: entered unreachable code
note: run with `RUST_BACKTRACE=1` environment variable to display a backtrace 


...minified-js-code-truncated...

                                                                                                                                                                                                                ^
PrismaClientRustPanicError: 
Invalid `prisma.site.findFirst()` invocation:


PANIC: internal error: entered unreachable code

This is a non-recoverable error which probably happens when the Prisma Query Engine has a panic.

....minified-js-code-truncated....


If you want the Prisma team to look into it, please open the link above 🙏
To increase the chance of success, please post your schema and a snippet of
how you used Prisma Client in the issue. 

EDIT: After I ran it several times, it throws usual segmentation fault. Back to 1.0.18 i guess...

rizrmd avatar Feb 17 '24 12:02 rizrmd

1.0.27 released, yay!

Still error, but It is throwing different message than previous versions!!

This is the same error I was getting in 1.0.26 as well when running oven/bun:1.0.26-slim in Docker (confirmed also in oven/bun:1.0.27-slim) with Postgres. I'll try to make a minimum example repo that exhibits the behavior in a bit

camero2734 avatar Feb 17 '24 14:02 camero2734

Repro repo is here: https://github.com/camero2734/prisma-bun-issue (run w/ docker-compose up)

With FROM oven/bun:1.0.27-slim:

prisma-bun-app  | PrismaClientRustPanicError:
prisma-bun-app  | Invalid `prisma.$executeRaw()` invocation:
prisma-bun-app  |
prisma-bun-app  |
prisma-bun-app  | PANIC: internal error: entered unreachable code
prisma-bun-app  |
prisma-bun-app  | This is a non-recoverable error which probably happens when the Prisma Query Engine has a panic.
prisma-bun-app  |
prisma-bun-app  | https://github.com/prisma/prisma/issues/new?body=SNIPPED_OUT
prisma-bun-app  |
prisma-bun-app  | If you want the Prisma team to look into it, please open the link above 🙏
prisma-bun-app  | To increase the chance of success, please post your schema and a snippet of
prisma-bun-app  | how you used Prisma Client in the issue.
prisma-bun-app  |
prisma-bun-app  |       at new be (/code/node_modules/@prisma/client/runtime/library.js:26:2141)
prisma-bun-app  |       at handleRequestError (/code/node_modules/@prisma/client/runtime/library.js:126:7136)
prisma-bun-app  |       at handleAndLogRequestError (/code/node_modules/@prisma/client/runtime/library.js:126:6104)
prisma-bun-app  |       at /code/node_modules/@prisma/client/runtime/library.js:126:5812
prisma-bun-app  |
prisma-bun-app  | Encountered error after 14 iterations

thread 'tokio-runtime-worker' panicked at /root/.cargo/registry/src/index.crates.io-6f17d22bba15001f/tokio-1.30.0/src/sync/rwlock.rs:427:17:
prisma-bun-app  | internal error: entered unreachable code
prisma-bun-app  | stack backtrace:
prisma-bun-app  |    0:     0x14a9b786f34f - <unknown>
prisma-bun-app  |    1:     0x14a9b70a5480 - <unknown>
prisma-bun-app  |    2:     0x14a9b78463be - <unknown>
prisma-bun-app  |    3:     0x14a9b7870d5e - <unknown>
prisma-bun-app  |    4:     0x14a9b7870665 - <unknown>
prisma-bun-app  |    5:     0x14a9b78715c5 - <unknown>
prisma-bun-app  |    6:     0x14a9b7871098 - <unknown>
prisma-bun-app  |    7:     0x14a9b7871026 - <unknown>
prisma-bun-app  |    8:     0x14a9b7871011 - <unknown>
prisma-bun-app  |    9:     0x14a9b6f5a184 - <unknown>
prisma-bun-app  |   10:     0x14a9b6f5a352 - <unknown>
prisma-bun-app  |   11:     0x14a9b6fee9c9 - <unknown>
prisma-bun-app  |   12:     0x14a9b700d71e - <unknown>
prisma-bun-app  |   13:     0x14a9b6fe1f24 - <unknown>
prisma-bun-app  |   14:     0x14a9b6ff4517 - <unknown>
prisma-bun-app  |   15:     0x14a9b7886619 - <unknown>
prisma-bun-app  |   16:     0x14a9b788fde1 - <unknown>
prisma-bun-app  |   17:     0x14a9b7884afd - <unknown>
prisma-bun-app  |   18:     0x14a9b78845a6 - <unknown>
prisma-bun-app  |   19:     0x14a9b7889b57 - <unknown>
prisma-bun-app  |   20:     0x14a9b7872186 - <unknown>
prisma-bun-app  |   21:     0x14aa18327ea7 - start_thread
prisma-bun-app  |   22:     0x14aa1843da6f - clone
prisma-bun-app  |   23:                0x0 - <unknown>

With FROM oven/bun:1.0.18-slim:

prisma-bun-app  |
prisma-bun-app  | No error

The repro code is:

import { createCanvas } from "@napi-rs/canvas";
import { PrismaClient } from '@prisma/client';

const client = new PrismaClient();

async function causeError() {
  // 1. Some DB query
  const tx = client.$executeRaw`SELECT pg_sleep(0.01)`;
  // 2. Some napi-based operation
  createCanvas(1000, 1000);
  // 3. Wait for the DB query to finish
  await tx;
}

let success = true;
for (let i = 0; i < 50; i++) {
  try {
    await causeError();
  } catch (e) {
    console.error(e);
    console.log(`Encountered error after ${i + 1} iterations`);
    success = false;
    break;
  }
}

if (success) console.log("No error");

If I remove 1, 2, or 3, the issue does not reproduce in the latest version of Bun.

Edit: Also found that the problem is gone if I build main with this commit reverted https://github.com/oven-sh/bun/commit/a93f467a7428d62a3ca38244b69b080e9d85d1be

camero2734 avatar Feb 17 '24 17:02 camero2734

Okay, I've narrowed down that set of changes to just a single line in napi_delete_reference:

// src/bun.js/bindings/napi.cpp
extern "C" napi_status napi_delete_reference(napi_env env, napi_ref ref)
{
    NAPI_PREMABLE
    NapiRef* napiRef = toJS(ref);
    
    napiRef->~NapiRef(); // PREVIOUS -- reverting to this fixes the issues with Prisma
    delete napiRef       // CURRENT -- breaks Prisma

    return napi_ok;
}

camero2734 avatar Feb 17 '24 22:02 camero2734

Pinging @Jarred-Sumner, can we just revert this line to solve this issue ?

rizrmd avatar Feb 19 '24 00:02 rizrmd