[BUG]: migrations do not work - table already exists - ER_TABLE_EXISTS_ERROR - mysql

[renhyl]

What version of drizzle-orm are you using?

0.30.10

What version of drizzle-kit are you using?

0.20.18

Describe the Bug

whenever I run my package.json script: "db:migrate": "bun run db/migrate.js",

my migrate.js:

import { drizzle } from 'drizzle-orm/mysql2'
import { migrate } from 'drizzle-orm/mysql2/migrator'
import { createConnection } from 'mysql2'

const test = process.env.BUN_ENV === 'test'

const connection = createConnection({
    host: test ? process.env.TEST_DB_HOST : process.env.DB_HOST,
    user: test ? process.env.TEST_DB_USERNAME : process.env.DB_USERNAME,
    database: test ? process.env.TEST_DB_DATABASE : process.env.DB_DATABASE,
    port: test ? process.env.TEST_DB_PORT : process.env.DB_PORT,
    password: test ? process.env.TEST_DB_PASSWORD : process.env.DB_PASSWORD,
    multipleStatements: true
})

const db = drizzle(connection)

async function main() {
    await migrate(db, { migrationsFolder: 'drizzle' })
    await connection.end()

    console.info('Migrations complete!')
    process.exit() // default success
}

main().catch((e) => {
    console.error(e)
    process.exit(1) // 1 indicates error
})

I get

$ bun run db/migrate.js
89 |     this.connection.release();
90 |   }
91 | 
92 |   query(query, params) {
93 |     const c = this.connection;
94 |     const localErr = new Error();
                          ^
error: Table 'game_room_chat' already exists
 errno: 1050
  code: "ER_TABLE_EXISTS_ERROR"

      at query (/home/damian/work/spark-graphql/node_modules/mysql2/promise.js:94:22)
      at execute (/home/damian/work/spark-graphql/node_modules/drizzle-orm/mysql2/session.js:43:17)
      at /home/damian/work/spark-graphql/node_modules/drizzle-orm/mysql-core/dialect.js:39:22
      at /home/damian/work/spark-graphql/node_modules/drizzle-orm/mysql-core/dialect.js:35:38
      at /home/damian/work/spark-graphql/node_modules/drizzle-orm/mysql2/session.js:170:28

error: script "db:migrate" exited with code 1

Expected behavior

According to documentation the migrations should be skipped if these were already applied, this does not seem to be happening.

ref: https://orm.drizzle.team/docs/migrations

import 'dotenv/config';
import { migrate } from 'drizzle-orm/mysql2/migrator';
import { db, connection } from './db';
// This will run migrations on the database, skipping the ones already applied
await migrate(db, { migrationsFolder: './drizzle' });
// Don't forget to close the connection, otherwise the script will hang
await connection.end();

my migrations should be able to run and apply new changes to the schema without complaining about table already existing.

I am very keen on solution to this bug and or workaround if such exists.

Environment & setup

No response

[renchris]

I'm getting this issue too with drizzle-kit generate and drizzle-kit migrate when adding one row to a table ie. ALTER TABLE `item` ADD `count` integer DEFAULT 0 NOT NULL; and it resulting an error for another existing table LibsqlError: SQLITE_ERROR: table `credential` already exists

[gokhantaskan]

I'm new to drizzle and ORMs, but I figured out my issue (which is similar to renchris issue):

• My first migration includes "CREATE TABLE reports" not "CREATE TABLE IF NOT EXISTS reports"
• My second migration includes "ALTER TABLE reports ADD submitter_id text NOT NULL" which does not set a default value for NOT NULL.
  • I added "...NOT NULL DEFAULT "-1" at the end

Problem solved.

[AndriiSherman]

Please include some repro example, because I can't reproduce it locally

[SpamixOfficial]

Having this issue at the moment, can drop my schema.ts + errors and commands if needed!

I am using Drizzle Mysql + Mariadb, works fine but this is literally a pain

[G4tes]

@SpamixOfficial, you should drop all tables, including __drizzle_migrations, and delete the migrate folder. After that, you can run:

npx drizzle-kit generate npx drizzle-kit migrate Remember, after making changes to your schema, always run npx drizzle-kit generate first, followed by npx drizzle-kit migrate.

[LeulAria]

im getting this

already exists

after rolling back one migration file.

[dencs08]

The bug was not resolved yet.

• Best workaround is to use push command. So for first migration use generate and migrate after that always use push - it preserves data in the tables while running all the new migrations correctly without any conflicts.

• Current workaround for migrate script for me is to drop everything and run generate and migrate again but this is not ideal when we want to preserve current data in the database.

• Also we can comment out table creations in each migration file which creates tables to not run it or replace:

CREATE TABLE table_name to: CREATE TABLE IF NOT EXISTS table_name ``

[krishna-santosh]

I've noticed similar issues with Drizzle migrations when it tries to recreate foreign key relationships or types in PostgreSQL. To work around it, I had to manually modify the migration file, changing CREATE TABLE to CREATE TABLE IF NOT EXISTS and wrapping certain statements in an anonymous code block (DO $$ ... $$;). I'm not sure if this is the standard approach, but after making these changes, the migrations applied successfully.

[zero8d]

The same issue with postgresql.

[⡿] applying migrations...error: relation "students" already exists .... length: 102, severity: 'ERROR', code: '42P07', detail: undefined, hint: undefined, position: undefined, internalPosition: undefined, internalQuery: undefined, where: undefined, schema: undefined, table: undefined, column: undefined, dataType: undefined, constraint: undefined, file: 'heap.c', line: '1149', routine: 'heap_create_with_catalog'

[medeirosjoaquim]

Migration should not fail as a whole if a column, table or any entity already exists. Drizzle should at least properly generate migrations with "if not exists" or similar. If for some reason part of the migration can't be applied because entity was already created, migration should continue.

[m-primo]

Any update on this??

[gokaygurcan]

So what's the proper way of using migration today? Or are we just yolo-ing with push --force instead?

I love how check returns "Everything's fine" when there's a clear error message one line above :)

[TheDogHusky]

Personally I run migrate() on each startup as it is an electron app using drizzle with sqlite. I get migration errors at startups because the table already exists. No ways for drizzle-kit to generate migrations with CREATE TABLE IF NOT EXISTS instead of CREATE TABLE?

[AndriiSherman]

Hey everyone!

I've created this message to send in a batch to all opened issues we have, just because there are a lot of them and I want to update all of you with our current work, why issues are not responded to, and the amount of work that has been done by our team over ~8 months.

I saw a lot of issues with suggestions on how to fix something while we were not responding – so thanks everyone. Also, thanks to everyone patiently waiting for a response from us and continuing to use Drizzle!

We currently have 4 major branches with a lot of work done. Each branch was handled by different devs and teams to make sure we could make all the changes in parallel.

---

First branch is drizzle-kit rewrite

All of the work can be found on the alternation-engine branch. Here is a PR with the work done: https://github.com/drizzle-team/drizzle-orm/pull/4439

As you can see, it has 167k added lines of code and 67k removed, which means we've completely rewritten the drizzle-kit alternation engine, the way we handle diffs for each dialect, together with expanding our test suite from 600 tests to ~9k test units for all different types of actions you can do with kit. More importantly, we changed the migration folder structure and made commutative migrations, so you won't face complex conflicts on migrations when working in a team.

What's left here:

• We are finishing handling defaults for Postgres, the last being geometry (yes, we fixed the srid issue here as well).
• We are finishing commutative migrations for all dialects.
• We are finishing up the command, so the migration flow will be as simple as drizzle-kit up for you.

Where it brings us:

• We are getting drizzle-kit into a new good shape where we can call it [email protected]!

Timeline:

• We need ~2 weeks to finish all of the above and send this branch to beta for testing.

Second big branch is a complex one with several HUGE updates

• Bringing Relational Queries v2 finally live. We've done a lot of work here to actually make it faster than RQBv1 and much better from a DX point of view. But in implementing it, we had to make another big rewrite, so we completely rewrote the drizzle-orm type system, which made it much simpler and improved type performance by ~21.4x:

(types instantiations for 3300 lines production drizzle schema + 990 lines relations)

TS v5.8.3: 728.8k -> 34.1k
TS v5.9.2: 553.7k -> 25.4k

You can read more about it here.

What's left here:

• We have 1 issue with TS that is already in progress of being fixed. The issue and Post about fixing.

Where it brings us:

• We are getting drizzle-orm into a new good shape where we can call it [email protected]!

Breaking changes:

• We will have them, but we will have open channels for everyone building on top of drizzle types, so we can guide you through all the changes.

Third branch is adding support for CockroachDB and MSSQL dialects

Support for them is already in the alternation-engine branch and will be available together with the drizzle-kit rewrite.

Summary

All of the work we are doing is crucial and should be done sooner rather than later. We've received a lot of feedback and worked really hard to find the best strategies and decisions for API, DX, architecture, etc., so we can confidently mark it as v1 and be sure we can improve it and remain flexible for all the features you are asking for, while becoming even better for everyone building on top of the drizzle API as well.

We didn't want to stay with some legacy decisions and solutions we had, and instead wanted to shape Drizzle in a way that will be best looking ahead to 2025–2026 trends (v1 will get proper effect support, etc.).

We believe that all of the effort we've put in will boost Drizzle and benefit everyone using it.

Thanks everyone, as we said, we are here to stay for a long time to build a great tool together!

Timelines

We are hoping to get v1 for drizzle in beta this fall and same timeline for latest. Right after that we can go through all of the issues and PRs and resond everyone. v1 for drizzle should close ~70% of all the bug tickets we have, so on beta release we will start marking them as closed!

[varuninnavoto]

Hey everyone!

I've created this message to send in a batch to all opened issues we have, just because there are a lot of them and I want to update all of you with our current work, why issues are not responded to, and the amount of work that has been done by our team over ~8 months.

I saw a lot of issues with suggestions on how to fix something while we were not responding – so thanks everyone. Also, thanks to everyone patiently waiting for a response from us and continuing to use Drizzle!

We currently have 4 major branches with a lot of work done. Each branch was handled by different devs and teams to make sure we could make all the changes in parallel.

First branch is drizzle-kit rewrite

All of the work can be found on the alternation-engine branch. Here is a PR with the work done: #4439

As you can see, it has 167k added lines of code and 67k removed, which means we've completely rewritten the drizzle-kit alternation engine, the way we handle diffs for each dialect, together with expanding our test suite from 600 tests to ~9k test units for all different types of actions you can do with kit. More importantly, we changed the migration folder structure and made commutative migrations, so you won't face complex conflicts on migrations when working in a team.

What's left here:

* We are finishing handling defaults for Postgres, the last being geometry (yes, we fixed the `srid` issue here as well).

* We are finishing commutative migrations for all dialects.

* We are finishing up the command, so the migration flow will be as simple as `drizzle-kit up` for you.

Where it brings us:

* We are getting drizzle-kit into a new good shape where we can call it `[email protected]`!

Timeline:

* We need ~2 weeks to finish all of the above and send this branch to beta for testing.

Second big branch is a complex one with several HUGE updates

* Bringing Relational Queries v2 finally live. We've done a lot of work here to actually make it faster than RQBv1 and much better from a DX point of view. But in implementing it, we had to make another big rewrite, so we completely rewrote the drizzle-orm type system, which made it much simpler and improved type performance by ~21.4x:

(types instantiations for 3300 lines production drizzle schema + 990 lines relations)

TS v5.8.3: 728.8k -> 34.1k
TS v5.9.2: 553.7k -> 25.4k

You can read more about it here.

What's left here:

* We have 1 issue with TS that is already in progress of being fixed. [The issue](https://x.com/DrizzleORM/status/1961524446638084533) and [Post about fixing](https://x.com/AndaristRake/status/1961722106846040396).

Where it brings us:

* We are getting drizzle-orm into a new good shape where we can call it `[email protected]`!

Breaking changes:

* We will have them, but we will have open channels for everyone building on top of drizzle types, so we can guide you through all the changes.

Third branch is adding support for CockroachDB and MSSQL dialects

Support for them is already in the alternation-engine branch and will be available together with the drizzle-kit rewrite.

Summary

All of the work we are doing is crucial and should be done sooner rather than later. We've received a lot of feedback and worked really hard to find the best strategies and decisions for API, DX, architecture, etc., so we can confidently mark it as v1 and be sure we can improve it and remain flexible for all the features you are asking for, while becoming even better for everyone building on top of the drizzle API as well.

We didn't want to stay with some legacy decisions and solutions we had, and instead wanted to shape Drizzle in a way that will be best looking ahead to 2025–2026 trends (v1 will get proper effect support, etc.).

We believe that all of the effort we've put in will boost Drizzle and benefit everyone using it.

Thanks everyone, as we said, we are here to stay for a long time to build a great tool together!

Timelines

We are hoping to get v1 for drizzle in beta this fall and same timeline for latest. Right after that we can go through all of the issues and PRs and resond everyone. v1 for drizzle should close ~70% of all the bug tickets we have, so on beta release we will start marking them as closed!

Is there any updates on this? I am facing issues with a production application and unable to fix it myself.

[manueljenni]

This is so frustrating, I can't apply any migrations because of this issue and I also can't delete all migrations and start over fresh using introspect because of this https://github.com/drizzle-team/drizzle-orm/issues/3793 ...

[haexyh]

Problem does also exists with pgSql driver

[loeiks]

Still waiting for alternation-engine update for that "relation already exists" error. (I use postgres)