[FEATURE]: bigint mode in SQLite integer column

[hyunbinseo]

Describe what you want

• SQLite's maximum integer value is 9223372036854775807
• JavaScript's maximum safe integer is 9007199254740991

9223372036854775807 // SQLite's largest possible integer
9007199254740991 // JavaScript Number.MAX_SAFE_INTEGER

Allow SQLite integer columns to be inferred as JavaScript BigInt.

integer('id', { mode: 'number' })
integer('id', { mode: 'bigint' }) // support this

PostgreSQL and MySQL support this feature in the bigint column.

// will be inferred as `number`
bigint: bigint('bigint', { mode: 'number' })

// will be inferred as `bigint`
bigint: bigint('bigint', { mode: 'bigint' })

This will be useful for Autoincrement columns that can become large.

Related issues:

• https://github.com/drizzle-team/drizzle-orm/pull/739
• https://github.com/drizzle-team/drizzle-orm/pull/558 - implemented with blob

[ksmithut]

One complication with this will be supporting all the different sqlite drivers. For example, better-sqlite3 has some documentation on how to set up BigInt, but it seems to be sort of an all-or-nothing thing, not column specific. So turning on BigInt for everything would mean that all the other int types would need to convert bigints to normal javascript numbers in order for the types to work.

[MrRahulRamkumar]

Will try and create a PR for this this weekend, but not sure what to do about the complication @ksmithut mentioned.

[MrRahulRamkumar]

Did some digging into this and it looks likes most drivers have a configuration option on how SQLite integers are converted to JavaScript values for the entire database. libsql-client-ts allows you to return integers as numbers, big ints or as strings. (See https://github.com/tursodatabase/libsql-client-ts/issues/32 and https://github.com/tursodatabase/libsql-client-ts/pull/51) Whereas bun:sqlite only supports conversion of all SQLite integers to big ints. (See https://bun.sh/docs/api/sqlite#safeintegers-true).

Since we do not have the ability to configure this per column and not all drivers support conversion to big ints, I suggest we add a new bigint column type which would be used instead of the number column. This column would have the same functionality and modes as number with the only difference being that it is set up to handle the underlying driver returning JavaScript big ints instead of numbers.

So for example, with the libsql driver you would do this:

import { drizzle } from 'drizzle-orm/libsql';
import { createClient } from '@libsql/client';
const client = createClient({
  url: "DATABASE_URL",
  authToken: "DATABASE_AUTH_TOKEN",
  intMode: "bigint"
});
const db = drizzle(client);

// all integer columns would use the bigint column
const table = sqliteTable('table', {
  id: bigint('id')
});

// you can customize integer mode to be number, boolean, timestamp, timestamp_ms
bigint('id', { mode: 'number' })
bigint('id', { mode: 'boolean' })
bigint('id', { mode: 'timestamp_ms' })
bigint('id', { mode: 'timestamp' }) // Date

@dankochetov I can create a PR if you are okay with this aproach

[hyunbinseo]

@MrRahulRamkumar If the int mode is set to BigInt, are bigint columns forced?

What would happen to the (existing) integer columns? Would they throw errors?

const client = createClient({ intMode: 'bigint' });

const db = drizzle(client);
const db = drizzle(client, { schema }); // Check for the schema here?

If BigInt → Number conversion has minimal performance impacts, this could be possible:

// Always returns a Number
// If the driver returns Number, keep
// In the driver returns BigInt, covnert
integer('id', { mode: 'number' });

// Always returns a BigInt
// If the driver returns Number, convert
// In the driver returns BigInt, keep
integer('id', { mode: 'bigint' });

Values larger than the Number.MAX_SAFE_INTEGER should be handled by the developer.

[MrRahulRamkumar]

@hyunbinseo if the int mode is set to bigint, all integer columns would be returned as Javascript big ints by the driver. Existing integer columns would possibly break as drizzle expects the driver to return Javascript number and not bigint.

Your suggestion would work but we would need to make sure that all integer modes (number, boolean, timestamp, timestamp_ms) are also updated to handle big ints being returned by the driver.

There might also be some cases where the developer would have to manually handle big ints being returned by drizzle.

I will create a PR that implements what you suggested. Let's see what the maintainers think.

[xray-robot]

+1

[GustavoOS]

Any news on the PR?

[kotsoft]

Honestly, I would almost consider this a BUG, not just an enhancement (as this issue has been labeled). Could lead to a security vulnerability or critical system failure later down the line.

[GustavoOS]

Honestly, I would almost consider this a BUG, not just an enhancement (as this issue has been labeled). Could lead to a security vulnerability or critical system failure later down the line.

This is the main reason I can't use something like Snowflake Id in SQLite with turso. If a table holds the primary key as blob, it would be unqueriable (see #2902 ), but Drizzle won't let us set it as a integer, despite SQLite supporting up to 8 bytes integers

[eslym]

my current workaround is having custom types for those involved integer

// shema.ts
import { sqliteTable, customType, primaryKey } from 'drizzle-orm/sqlite-core';

const bignum = customType<{ data: bigint; driverData: bigint }>({
    dataType: () => 'INTEGER'
});

const integer = customType<{ data: number; driverData: bigint }>({
    dataType: () => 'INTEGER',
    toDriver: (value) => BigInt(value),
    fromDriver: (value) => Number(value)
});

const boolean = customType<{ data: boolean; driverData: bigint }>({
    dataType: () => 'INTEGER',
    toDriver: (value) => BigInt(value ? 1 : 0),
    fromDriver: (value) => Boolean(value)
});

const autoIncrement = customType<{ data: bigint; driverData: bigint; default: true }>({
    dataType: () => 'INTEGER PRIMARY KEY AUTOINCREMENT'
});

const timestamp = customType<{ data: Date; driverData: bigint }>({
    dataType: () => 'INTEGER',
    toDriver: (value) => BigInt(value.getTime()),
    fromDriver: (value) => new Date(Number(value))
});

[GustavoOS]

my current workaround is having custom types for those involved integer

// shema.ts
import { sqliteTable, customType, primaryKey } from 'drizzle-orm/sqlite-core';

const bignum = customType<{ data: bigint; driverData: bigint }>({
    dataType: () => 'INTEGER'
});

const integer = customType<{ data: number; driverData: bigint }>({
    dataType: () => 'INTEGER',
    toDriver: (value) => BigInt(value),
    fromDriver: (value) => Number(value)
});

const boolean = customType<{ data: boolean; driverData: bigint }>({
    dataType: () => 'INTEGER',
    toDriver: (value) => BigInt(value ? 1 : 0),
    fromDriver: (value) => Boolean(value)
});

const autoIncrement = customType<{ data: bigint; driverData: bigint; default: true }>({
    dataType: () => 'INTEGER PRIMARY KEY AUTOINCREMENT'
});

const timestamp = customType<{ data: Date; driverData: bigint }>({
    dataType: () => 'INTEGER',
    toDriver: (value) => BigInt(value.getTime()),
    fromDriver: (value) => new Date(Number(value))
});

Thanks so much. However, this disables auto generated values for me. Does it happen to you as well?

[hyunbinseo]

Is this implemented in 0.41.0?

bigint, number modes for SQLite, MySQL, PostgreSQL, SingleStore decimal & numeric column types

[Download]

May I offer a suggestion?

BigInt in Javascript is a variable-width integer type. It is NOT an Int64. I saw issues mention that BLOB columns are used to store bigint values (making them unsearchable) and that is essentially the only way you can store all possible Javascript bigint values, precisely because it is variable.

This is mostly NOT what developers want/need/ask for. It is thinking in the wrong direction. Starting at Javascript bigint and then picking the database column suitable to store it. We should think in the other direction: What number type does the database support and what can we use in Javascript to support it? Then, we arrive at the conclusion that most developers want 64-bit number support. And Javascript number simply does not hack it.

Most databases DO support 64-bit numbers. So personally, I feel it makes a lot more sense to do something like this:

integer('id', { mode: 'bigint', precision: 64 });

Or just

integer('id', { mode: 'bigint' });

and document that this ONLY supports numbers up to 64 bits.

Then when devs truly need variable width numbers, they can go the Blob('id', { mode: 'bigint' }) route.

Making the column function have name BigInt is doubly-confusing imho since it seems it ends up mapped as a int64 on the column side and NOT as a bigint so....

Thoughts?

[AndriiSherman]

Hey everyone!

I've created this message to send in a batch to all opened issues we have, just because there are a lot of them and I want to update all of you with our current work, why issues are not responded to, and the amount of work that has been done by our team over ~8 months.

I saw a lot of issues with suggestions on how to fix something while we were not responding – so thanks everyone. Also, thanks to everyone patiently waiting for a response from us and continuing to use Drizzle!

We currently have 4 major branches with a lot of work done. Each branch was handled by different devs and teams to make sure we could make all the changes in parallel.

---

First branch is drizzle-kit rewrite

All of the work can be found on the alternation-engine branch. Here is a PR with the work done: https://github.com/drizzle-team/drizzle-orm/pull/4439

As you can see, it has 167k added lines of code and 67k removed, which means we've completely rewritten the drizzle-kit alternation engine, the way we handle diffs for each dialect, together with expanding our test suite from 600 tests to ~9k test units for all different types of actions you can do with kit. More importantly, we changed the migration folder structure and made commutative migrations, so you won't face complex conflicts on migrations when working in a team.

What's left here:

• We are finishing handling defaults for Postgres, the last being geometry (yes, we fixed the srid issue here as well).
• We are finishing commutative migrations for all dialects.
• We are finishing up the command, so the migration flow will be as simple as drizzle-kit up for you.

Where it brings us:

• We are getting drizzle-kit into a new good shape where we can call it [email protected]!

Timeline:

• We need ~2 weeks to finish all of the above and send this branch to beta for testing.

Second big branch is a complex one with several HUGE updates

• Bringing Relational Queries v2 finally live. We've done a lot of work here to actually make it faster than RQBv1 and much better from a DX point of view. But in implementing it, we had to make another big rewrite, so we completely rewrote the drizzle-orm type system, which made it much simpler and improved type performance by ~21.4x:

(types instantiations for 3300 lines production drizzle schema + 990 lines relations)

TS v5.8.3: 728.8k -> 34.1k
TS v5.9.2: 553.7k -> 25.4k

You can read more about it here.

What's left here:

• We have 1 issue with TS that is already in progress of being fixed. The issue and Post about fixing.

Where it brings us:

• We are getting drizzle-orm into a new good shape where we can call it [email protected]!

Breaking changes:

• We will have them, but we will have open channels for everyone building on top of drizzle types, so we can guide you through all the changes.

Third branch is adding support for CockroachDB and MSSQL dialects

Support for them is already in the alternation-engine branch and will be available together with the drizzle-kit rewrite.

Summary

All of the work we are doing is crucial and should be done sooner rather than later. We've received a lot of feedback and worked really hard to find the best strategies and decisions for API, DX, architecture, etc., so we can confidently mark it as v1 and be sure we can improve it and remain flexible for all the features you are asking for, while becoming even better for everyone building on top of the drizzle API as well.

We didn't want to stay with some legacy decisions and solutions we had, and instead wanted to shape Drizzle in a way that will be best looking ahead to 2025–2026 trends (v1 will get proper effect support, etc.).

We believe that all of the effort we've put in will boost Drizzle and benefit everyone using it.

Thanks everyone, as we said, we are here to stay for a long time to build a great tool together!

Timelines

We are hoping to get v1 for drizzle in beta this fall and same timeline for latest. Right after that we can go through all of the issues and PRs and resond everyone. v1 for drizzle should close ~70% of all the bug tickets we have, so on beta release we will start marking them as closed!