[FEATURE]: add easy-to-find full examples for web sqlite and pglite

[AntonOfTheWoods]

Describe what you want

There are great examples and docs for the workflow for cases where there are external databases but I couldn't find anything for web sqlite or pglite, particularly regarding how to manage migrations and initial data loading.

[dgstewart]

I don't believe the migrate function from the migrator class is compatible in the browser since the following code throws a runtime error in the browser.

import * as schema from "~/drizzle/schema";
import { drizzle } from 'drizzle-orm/pglite';

import { PGlite } from '@electric-sql/pglite';
import { migrate } from 'drizzle-orm/pglite/migrator';

const client = new PGlite("idb://my-table");
const db = drizzle(client, { schema });
await migrate(db, { migrationsFolder: './drizzle' });

This won't work for obvious fs dependency reasons but if we could achieve something similar with PGlite in the future that would be awesome!

My current process just involves running pnpm drizzle-kit generate:pg to have drizzle generate the migrations for me and then copying the contents of the most recent migration file into a db.execute() roughly following the sql“ template section in the documentation. I doubt/hope this isn't the intended workflow, but I'm excited to see where this can go. I already enjoy using this workaround more than having to interact with IndexedDB with tools like dexie.

Hope this helps!

[AntonOfTheWoods]

Thanks! I have always seen JS as a browser tech first and so assume people would at least keep that in the back of their mind. I keep forgetting that many people don't share this affliction... And so can happily do projects that might look like they work in the browser but actually don't, or require nasty undocumented hacks. I guess it's faster than python/ruby et al, so why not?

[urbanisierung]

Today I was playing around with Drizzle and PGlite with indexeddb and also noticed that migrations don't work. Would be great if this will be supported somewhen in the future!

Until then I implemented a very basic migration logic that does the same. @AntonOfTheWoods @dgstewart here's a gist if you're interested: https://gist.github.com/urbanisierung/075af207082d08504083e5bf36093fe7

[JoschuaSchneider]

I'm facing the same Problem trying to run drizzle with PGlite in the browser.

My workaround for initial schema migration is to import the generated migrations using vite ?raw params to get the SQL statements as a string and then running them directly against the PGLite client:

import migrations from './migrations/0000_smiling_russian.sql?raw';

const client = new PGlite('idb://offline2');

client
  .exec(migrations)
  .then(() => console.log('Migrated'))
  .catch((err) => console.error(err));

const db = drizzle(...)

This lacks a lot of things though and I would love to be able to generate migrations code-first using the given schema. This already happens within drizzle-kit, but the drizzle-kit source does not seem to be open (other than packaged into the npm package).

What I am trying to achieve is a fresh setup of the database first, without migrations. Something like this (pseudo code):

// PSEUDO CODE
import { push } from "<somewhere>";
import * as schema from "./schema.ts";

const client = new PGlite('idb://offline2');
const db = drizzle(db, { schema });

await push(db, { schema }); // would drop all tables, and apply the new (or initial) schema

Does anybody know if there is a way to programatically generate the initial migration into sql without using drizzle-kit?

[Anulo2]

Hey everyone i've been working on an example to do such a thing but can't get it to work properly, maybe you have some ideas and we can create a workaround until there's official support: https://github.com/Anulo2/Drizzle-PGLite-IndexedDB

[AndriiSherman]

Hey everyone!

I've created this message to send in a batch to all opened issues we have, just because there are a lot of them and I want to update all of you with our current work, why issues are not responded to, and the amount of work that has been done by our team over ~8 months.

I saw a lot of issues with suggestions on how to fix something while we were not responding – so thanks everyone. Also, thanks to everyone patiently waiting for a response from us and continuing to use Drizzle!

We currently have 4 major branches with a lot of work done. Each branch was handled by different devs and teams to make sure we could make all the changes in parallel.

---

First branch is drizzle-kit rewrite

All of the work can be found on the alternation-engine branch. Here is a PR with the work done: https://github.com/drizzle-team/drizzle-orm/pull/4439

As you can see, it has 167k added lines of code and 67k removed, which means we've completely rewritten the drizzle-kit alternation engine, the way we handle diffs for each dialect, together with expanding our test suite from 600 tests to ~9k test units for all different types of actions you can do with kit. More importantly, we changed the migration folder structure and made commutative migrations, so you won't face complex conflicts on migrations when working in a team.

What's left here:

• We are finishing handling defaults for Postgres, the last being geometry (yes, we fixed the srid issue here as well).
• We are finishing commutative migrations for all dialects.
• We are finishing up the command, so the migration flow will be as simple as drizzle-kit up for you.

Where it brings us:

• We are getting drizzle-kit into a new good shape where we can call it [email protected]!

Timeline:

• We need ~2 weeks to finish all of the above and send this branch to beta for testing.

Second big branch is a complex one with several HUGE updates

• Bringing Relational Queries v2 finally live. We've done a lot of work here to actually make it faster than RQBv1 and much better from a DX point of view. But in implementing it, we had to make another big rewrite, so we completely rewrote the drizzle-orm type system, which made it much simpler and improved type performance by ~21.4x:

(types instantiations for 3300 lines production drizzle schema + 990 lines relations)

TS v5.8.3: 728.8k -> 34.1k
TS v5.9.2: 553.7k -> 25.4k

You can read more about it here.

What's left here:

• We have 1 issue with TS that is already in progress of being fixed. The issue and Post about fixing.

Where it brings us:

• We are getting drizzle-orm into a new good shape where we can call it [email protected]!

Breaking changes:

• We will have them, but we will have open channels for everyone building on top of drizzle types, so we can guide you through all the changes.

Third branch is adding support for CockroachDB and MSSQL dialects

Support for them is already in the alternation-engine branch and will be available together with the drizzle-kit rewrite.

Summary

All of the work we are doing is crucial and should be done sooner rather than later. We've received a lot of feedback and worked really hard to find the best strategies and decisions for API, DX, architecture, etc., so we can confidently mark it as v1 and be sure we can improve it and remain flexible for all the features you are asking for, while becoming even better for everyone building on top of the drizzle API as well.

We didn't want to stay with some legacy decisions and solutions we had, and instead wanted to shape Drizzle in a way that will be best looking ahead to 2025–2026 trends (v1 will get proper effect support, etc.).

We believe that all of the effort we've put in will boost Drizzle and benefit everyone using it.

Thanks everyone, as we said, we are here to stay for a long time to build a great tool together!

Timelines

We are hoping to get v1 for drizzle in beta this fall and same timeline for latest. Right after that we can go through all of the issues and PRs and resond everyone. v1 for drizzle should close ~70% of all the bug tickets we have, so on beta release we will start marking them as closed!

[maxwhipw]

Bumping this issue. The documentation doesn't seem to distinguish different implementations with pglite and from what I have seen so far migrations don't seem to be supported when persisting to IndexedDB. Don't know enough about the guts of each project to know why that would be the case, but embedded in-browser databases are so so useful I hope we can get clear documentation and support for projects like pglite in browser.