[BUG]: RangeError: Maximum call stack size exceeded on insert of large amount of rows

[jakeleventhal]

trafficstars

What version of drizzle-orm are you using?

0.29.2

What version of drizzle-kit are you using?

0.20.9

Describe the Bug

Reopening https://github.com/drizzle-team/drizzle-orm/issues/269 I am trying to seed a test database by running db.insert(someTable).values(newValues).

If newValues.length is >=5958, I get an error message saying MAX_PARAMETERS_EXCEEDED: Max number of parameters (65534) exceeded.

Expected behavior

It should not have this error at any size. If anything, under the hood it should be doing something like I described here: https://github.com/drizzle-team/drizzle-orm/issues/797#issuecomment-1874159066

The ORM should be handling this, not application code.

Environment & setup

pg

[jakeleventhal]

I also experience issues with this when doing a large inArray query. If the list passed as the second parameter is too large, it says maximum callstack exceeded. Seems like there is probably some underlying recursion that doesn't need to happen.

[konhi]

happens to me too, and it'd be great if it could be handled by drizzle as @jakeleventhal suggested

[jakeleventhal]

A related issue:

This should be handled in the same case/way

[varugasu]

Having the same issue when inserting ~21k rows

version: ^0.30.1

This started happening when I decided to use cuid2 for id:

import { sql } from 'drizzle-orm'
import { text, integer, sqliteTable } from 'drizzle-orm/sqlite-core'
import { createId } from '@paralleldrive/cuid2'

export const animes = sqliteTable('animes', {
  id: text('id')
    .primaryKey()
    .$defaultFn(() => createId()),
  mal_id: integer('mal_id').unique(),
  title: text('title').notNull(),
  created_at: text('created_at')
    .notNull()
    .default(sql`CURRENT_TIMESTAMP`),
})

If I use id: integer('id').primaryKey({ autoIncrement: true }), I can insert

The error when bulking inserting:

12 | }
13 | function mergeQueries(queries) {
14 |   const result = { sql: "", params: [] };
15 |   for (const query of queries) {
16 |     result.sql += query.sql;
17 |     result.params.push(...query.params);
         ^
RangeError: Maximum call stack size exceeded.
      at mergeQueries (/Users/vargasmesh/Projects/Git/animes/node_modules/drizzle-orm/sql/sql.js:17:5)
      at /Users/vargasmesh/Projects/Git/animes/node_modules/drizzle-orm/sql/sql.js:51:21
      at _prepare (/Users/vargasmesh/Projects/Git/animes/node_modules/drizzle-orm/sqlite-core/query-builders/insert.js:125:7)
      at /Users/vargasmesh/Projects/Git/animes/node_modules/drizzle-orm/sqlite-core/query-builders/insert.js:134:12
      at execute (/Users/vargasmesh/Projects/Git/animes/node_modules/drizzle-orm/sqlite-core/query-builders/insert.js:145:19)
      at then (/Users/vargasmesh/Projects/Git/animes/node_modules/drizzle-orm/query-promise.js:21:12)

[Redm4x]

I am also getting both errors when trying to insert large amount of rows in Postgres.

Version: 0.30.10

MAX_PARAMETERS_EXCEEDED: Max number of parameters (65534) exceeded and Maximum call stack size exceeded

[goors]

I am also getting same Error: MAX_PARAMETERS_EXCEEDED: Max number of parameters (65534) exceeded on 2000 rows.

[latinrev]

I'm not sure if it has something to do with row amount, I had the Maximum call stack size error when I was executing this query with only 4 rows

  const insertedImages = await db.insert(images).values(mappedImages).onConflictDoUpdate({
    target: images.id,
    set: { url: sql` excluded.url`, pending: false }
  }).returning({ id: images.id + 'generated', url: images.url }).execute();

removing the returning clause made it go away

  const insertedImages = await db.insert(images).values(mappedImages).onConflictDoUpdate({
    target: images.id,
    set: { url: sql` excluded.url`, pending: false }
  }).execute();

[candidia]

Same happening with driver: "turso" and dialect: "sqlite" with about 160k rows. I have to do batches of just 1k for it to work...