Next major drift version: Todos and whishlist

[simolus3]

We have a bunch of issues that require breaking changes to address. Drift would be better with them fixed, but I also want to a new major version until there's a really good reason. After all, breaking changes are annoying for users which then need to adapt their setups. Drift is aiming to be a stable project, and that's a good thing. In the last year, I've assumed that we would just do a good breaking release once macros are stable (since that adopting them requires breaking codegen changes either way) and held up breaking changes until then. Given that macros aren't happening, it's time to revisit that position and prepare a batch of improvements to be delivered in the next major version.

My basic plan is this:

1. In an upcoming major version I'm already working on, aggressively fix some of the design decisions that haven't worked out and require breaking changes. The main focus is to make drift much more efficient while keeping most of the API surface intact.
2. Then, watch the development around augmentations and language features for data classes and serialization. Once that nearing a state where it's usable for us, we can come up with better interfaces around our code generation. But I don't have any radical changes planed for code generation or the way tables are defined.

This issue mostly serves as a todo-list with worthwhile changes that require breaking changes in drift. But of course, if something annoys you with drift today, I'd love to hear about that because this is a really good time to fix it! Also, if you disagree with something on this list, it would be helpful to know that. For the next major version, I want to focus on these areas:

Better interfaces to talk to databases

At the moment, drift uses the QueryExecutor interface to access databases, with an API that hasn't really changed much since the very early days of the project. There are some problems with that, namely:

• [x] Some implementation may want to see more than the raw SQL being emitted. For instance, we have some implementations that send writes to one isolate and reads to another. The introduction of RETURNING writes (implemented as selects in that interface) breaks that. We should give the implementation full access to a structural representation of the statement drift intents to run.
• [x] We're currently representing rows as a Map<String, Object?>, something inherited from the early sqflite-only days. Most implementations can access columns in rows by their index too though, and that is much more efficient. We should use a row representation that allows both index and name lookups, allowing database implementations to pick whatever they support.
• [ ] Allow RETURNING in batches: https://github.com/simolus3/drift/issues/3320.
• [ ] Remove package:drift/remote.dart as an independent library. It's a generalization of the drift isolate APIs over arbitrary communication channels. I'm not aware of any uses outside of drift.
• [ ] Remove package:drift/isolate.dart or make it SQLite-specific. This library should have been an implementation detail for NativeDatabase (and it is now). We need to continue supporting computeWithDatabase and establishing connections through SendPorts stored in IsolateNameServer though.
• [ ] Use a unified exception type

Better support for other databases

Historically, drift was mainly built with SQLite in mind. This has served us well for mobile apps, but with server-side Dart usage growing, we also want to support server-side databases like Postgres better.

• [x] We currently expose dialects as an enum. This makes drift harder to port / experiment with when using other dialects. It's also not ideal for code size (there's no reason for a Flutter web app to contain Postgres-specific drift code). I've experimented with an API inspired from SQLAlchemy that essentially makes the query builder a visitor implemented by different dialects. This makes it easy to share common logic while allowing tree-shaking to remove unused code.
• [x] Make the APIs dialect-specific where necessary. DriftAny and the current datetime APIs should not be exported under a common core import but rather something like package:drift/sqlite3.dart.
• [x] On the other hand, every DBMS we care about has good JSON support nowadays. We should make JSON a builtin type and expose JSON operators for all databases by default.

Also related to this:

• [ ] Support null variables in Postgres (which requires "typed nulls")

Efficiency

The main goal is to reduce the size of generated code by delegating stuff drift shouldn't do to other packages. This includes:

• [ ] Avoiding JSON serialization: If you need JSON serialization, there are good packages for that. Drift has never been one of them and its (very limited) internal JSON serialization capability is no match for what the other packages have to offer.
• [ ] The default representation for database rows should just be a Record. We should investigate making @UseRowClass easier to use and update guides on combining drift with freezed, json serializable or built value.
• [ ] @UseRowClass should no longer allow mapping values asynchronously. Supporting that feature slows some internal logic when mapping join results to Dart down.
• [ ] Find a way to deserialize rows on a background isolate where necessary.
• [ ] Remove client-side data validation when inserting. Databases do this for us.

Better web support

• [ ] package:drift/wasm.dart and package:sqlite3_web are mostly the same thing and I have to maintain both of them. Drift should just use sqlite3_web internally.
• [ ] Find ways to make OPFS easier to use. The best OPFS implementation we offer only works in Firefox. On all browsers that support shared workers, we can "share" dedicated workers through a shared worker proxy and thus support OPFS too. This means that fewer users would have to use IndexedDB (which has durability issues).

Misc

• [ ] Uniform interface to access watched tables from statements (https://github.com/simolus3/drift/issues/3475), this also relates to the query builder changes.
• [ ] Independent DAO classes: https://github.com/simolus3/drift/discussions/3477
• [ ] Expose schema statically, without having to creating a database: https://github.com/simolus3/drift/issues/3500
• [ ] Allow destructuring row classes: https://github.com/simolus3/drift/issues/3461#issuecomment-2835184260

Making upgrading easy

Given that major version updates are always more work for users, we should make upgrading easy:

• [ ] Introduce a builder option that bundles compatibility options. E.g users could specify drift_compatibility: v2 and get the current data class format instead of records by default. Since we have plenty of tests for the current format, still supporting the current generation mode is not that costly.
• [ ] For most APIs that change, we can provide typdefs or deprecated legacy libraries to keep supporting them.
• [ ] Write a drift_dev command that migrates projects from v2 to v3 automatically (where possible, some changes that hopefully won't affect many users aren't worth automating).
• [ ] Given that major version updates are always painful for users, we should avoid them where possible. We expect new generation changes to make use of augmentations in the future, this is something a drift_compatibility option could also enable as an opt-in.

[RoarGronmo]

I hope for this one to be implemented in next major release, or maybe before !

https://github.com/simolus3/drift/issues/3295

[dickermoshe]

Current Problem

Drift's tight coupling of database objects to the main database class presents users with a difficult choice when writing code that interacts with the database:

1. Encapsulation within AppDatabase:

   class AppDatabase extends _$AppDatabase {
     Future<List<User>> getUsers(){
       return  select(users).get();
     }
   }
   

   While seemingly straightforward, this approach forces all database-related code into a single, often unwieldy class. This leads to organizational challenges, even with the use of .part files.

   Furthermore, it necessitates creating redundant functions, even for one-time uses, instead of directly calling the underlying methods. Imagine a scenario where you have a complex query you only need to execute in one specific part of your application. With the current Drift structure, if you want to avoid the verbose db. prefix and take advantage of the implicit scope within the AppDatabase class, you're essentially forced to define a dedicated function within that class, even if that function is only ever called once.

   class AppDatabase extends _$AppDatabase {
   Future<List<User>> getUsers(){
     return  select(users).get();
     }
   Future<List<User>> deleteUser(int id){
     return (delete(users)..where(...)).go();
     }
   }
   ...
   

2. External Implementation:

   class AppDatabase extends _$AppDatabase {}
   
   final db = AppDatabase();
   
   Future<List<User>> getUsers(){
     return  db.select(db.users).get();
   }
   

   This provides greater flexibility in code organization, allowing users to structure their database interactions as needed. However, it comes at the cost of increased verbosity. Every database reference must be prefixed with db., leading to repetitive and less readable code.

Solution

For future releases, I propose decoupling tables from the concrete database instance. Imagine if each table was a standalone variable, independent of the AppDatabase object:

// BEFORE:
class _$AppDatabase extends GeneratedDatabase {
  UserTable get users => UserTable(this);
}
db.select(db.users)


// AFTER:
final usersTable = UserTable();
class _$AppDatabase extends GeneratedDatabase {
  // Backwards Compatibility
  UserTable get users => usersTable;
}

db.select(usersTable);

This change would dramatically improve code clarity and maintainability, reducing the need for the db. prefix.

Looking Towards The Future

Ideally, I envision a future where select is also decoupled, allowing for a more flexible and reusable query API:

select(usersTable).get(connection)

But this would be very breaking.

Bonus

final getUserQuery = select(usersTable)
connection.inIsolate((conn)=> getUserQuery.get(conn));

By loosening these dependencies, Drift could become more modular, easier to use, and better suited for complex applications, especially those involving isolates.

[simolus3]

This is an interesting idea, it also points towards some of the things mentioned in https://github.com/simolus3/drift/discussions/3477.

In the current prototype, tables are entirely independent from the actual database connection already. Statements aren't, but I'm starting to wonder if they should be. I think a sensible way forward might be to (at first) have two variants of all statements:

1. One that isn't bound to any database, but can be executed by supplying a database instance.
2. A wrapper around the the first one that is bound to a database and thus can be executed as-is.

Methods from the database could then return instances of the second kind, while we'd still allow users to write statements with 1. That way it's not quite as breaking and we have a sensible path forward.

[dickermoshe]

@simolus3 I think we should also rename some classes so we have less clashes with default Flutter classes. The big one that comes to mind is Column. This has made me go mad.

We should make drift easier to use throughout the entire codebase. The above proposal coupled with this would be huge.

[dickermoshe]

@simolus3 As massive benefit to using records is that they won't be "copied" when passed between isolates. Even if we support some sort of custom classes, maybe we should throw a warning if the class is not "const"

Although maybe we can take advantage of this:

An exception to this rule is when an isolate exits when it sends a message using the Isolate.exit method. Because the sending isolate won't exist after sending the message, it can pass ownership of the message from one isolate to the other, ensuring that only one isolate can access the message.

[jackd]

One issue I'm having surprising difficulty with is extracting a dart field value from a dataclass instance based on a GeneratedColumn. I can call toColumns then unwrap the Variable - or go via toJson and then deserialize the specific field - but both of these get tricky when dealing with converters, and they seem potentially expensive for something that I'd like to embed in e.g. a dart sort comparator function. I feel like I'm being forced to decide between the flexibility of the map interfaces and the type safety that's a large part of the benefit of drift, all for the sake of an efficient sort.

How difficult would it be to:

• add a type parameter to a shared GeneratedColumn / GeneratedColumnWithConverter superclass that denoted the dart type after possible conversion, (so GeneratedColumn<C extends Object> extends CommonGeneratedColumn<C, C> and GeneratedColumnWithTypeConverter<C, S extends Object> extends CommonGeneratedColumn<C, S> - though I'd argue there are better choices for names if things are breaking anyway :).
• add a D type parameter that denotes the associated dataclass to GeneratedColumns
• add a generated S getFieldValue(D d) method to GeneratedColumns.

Similar S? getFieldValueFromCompanion could also work, or a more general S? getFromInsertable.

[simolus3]

I think adding a getFieldValue on the column might not be too helpful because the column would then have to know how to destructure the class.

I think we could add a similar method on the table / view class though, so you could do myTable.readFromDart<T>(Row row, GeneratedColumn<T> column) which returns a T?.

[jackd]

I think adding a getFieldValue on the column might not be too helpful because the column would then have to know how to destructure the class.

I think we could add a similar method on the table / view class though, so you could do myTable.readFromDart<T>(Row row, GeneratedColumn<T> column) which returns a T?.

I don't mind whether it's on the table or the column.

Is Row here QueryRow? If so that doesn't really solve the issue - we can already do this with column.$nullable? row.readNullable<C>(column.name) : row.read<C>(column.name). It's taking the "work in deserialized space" approach mentioned above. It's workable, but it forces me to abandon most of the good things I like about using drift.

Now if Row here is the table's D then that would be much nicer. I think an S? MyTable.readFromDartConverted<S, C>(Row row, CommonGeneratedColumn<S, C> column) would be nicer/also nice to have (i.e. access to the converted field). Generally I want to define my converters in my Table class then completely forget about them whenever I'm working in dart.

[simolus3]

Sorry, D is called Row in my pending work for drift 3. So yes, the idea is that the row classes would be destructed to extract column values there.

[melWiss]

Hello there, I'm currently using drift in one of my personal projects, I just noticed two non friendly issues.

• I've been creating the schema and I was enjoying it until I found it after three days that it's not postgres compatible, the role of an orm is making these kind of things straightforward and abstract.
• database migration isn't straightforward, I think that generating some piece of code for each version then return back to your ide then complete the migration isn't very helpful, I suggest that the orm should verify these things when the app or cli is launched and you set the strategies of migration from one version to another, like table/column renaming or column type migration.

I believe that drift could be a really kickass orm, it's just still not perfect for server side yet

[melWiss]

And thank you sir and community for the package 😁

[dickermoshe]

I've been creating the schema and I was enjoying it until I found it after three days that it's not postgres compatible

It is, it just doesn't support the full feature set that Postgre has to offer.

See strawberry or serverpod if you really need Postgre's full feature set. Or better yet, contribute😜