`RangeError: Invalid string length`

[sangwoo-joh]

I have a bunch of tests (4,500+) and Python bespoke asserts scripts. When I run npx promptfoo eval -j16, it runs all jobs in parallel, but after they have finished, I got the following error:

/home/sangwoo-joh/type/work/promptfoo/node_modules/drizzle-orm/sqlite-core/columns/text.cjs:75
    return JSON.stringify(value);
                ^

RangeError: Invalid string length
    at JSON.stringify (<anonymous>)
    at SQLiteTextJson.mapToDriverValue (/home/sangwoo-joh/type/work/promptfoo/node_modules/drizzle-orm/sqlite-core/columns/text.cjs:75:17)
    at /home/sangwoo-joh/type/work/promptfoo/node_modules/drizzle-orm/sql/sql.cjs:157:73
    at Array.map (<anonymous>)
    at SQL.buildQueryFromSourceParams (/home/sangwoo-joh/type/work/promptfoo/node_modules/drizzle-orm/sql/sql.cjs:110:32)
    at /home/sangwoo-joh/type/work/promptfoo/node_modules/drizzle-orm/sql/sql.cjs:129:21
    at Array.map (<anonymous>)
    at SQL.buildQueryFromSourceParams (/home/sangwoo-joh/type/work/promptfoo/node_modules/drizzle-orm/sql/sql.cjs:110:32)
    at /home/sangwoo-joh/type/work/promptfoo/node_modules/drizzle-orm/sql/sql.cjs:132:21
    at Array.map (<anonymous>)

Node.js v19.9.0

• Ran on 16GB RAM, 16GB Swap, 8 Core Machine
• Node 19.9.0

[sangwoo-joh]

Hmm... I reduced the number of tests to about 1k and it works fine, which seems like this error occurs when we have too many tests.

[typpo]

Looks like we'd have to do something like this: https://stackoverflow.com/questions/29175877/json-stringify-throws-rangeerror-invalid-string-length-for-huge-objects

[mldangelo]

Hey @sangwoo-joh, thanks for your patience and for bringing this up! We’ve improved the data model in 0.92.0, which should help Promptfoo handle larger datasets more smoothly. Please give the new version a try when you have a chance. I’ll go ahead and close this issue, but if anything still isn’t quite right, feel free to reopen it.