Generating table part files separate from database part?

[atcaines]

Since all tables are included in the database .part file, I need to import my database file any time I want to use any of my database's DataClasses.

Is there an existing way that I could generate my files but import model1.dart, model2.dart while restricting database operations to a select few files (repository) by importing the database.dart file there?

Alternatively, can I define my Table separate from the DataClass associated with the Tables so that I could import the DataClass without getting the companion, table, database etc?

I have it working in the meantime by manually splitting out the generated file into database.g.dart and model.g.dart after each generation but that is somewhat annoying.

[simolus3]

I see what you mean, but at the moment this is not possible. I think it's a valid request, but it won't be easy to implement. Moor also generates additional (private) classes in the part file to represent the table structure. These private classes have to be in a part file of the database file of course. This means that we'd have to split code generation over multiple files. I haven't found a good approach for this yet.

[atcaines]

Thanks for the info

In the meantime I wrote a python script that uses RegEx to extract each (Model, ModelCompanion) class pair to another file called model.g.dart starting with "part of '{model}.dart';". Then all that is left are the Table and Database classes in the original file.

Not ideal obviously since I'm relying on layout to extract them with RegEx but it does the job for now.

[simolus3]

We could have another set of builders generating files like this:

• each moor file and every Dart file declaring tables gets
  • a (build_to: cache) moor.json file containing entities (tables, triggers, indices) in a serialized form
  • a (visible) .moor.dart file containing all the generated classes
• instead of generating part files for databases, we'd generate regular files importing other files as necessary.

This scales much better to large projects, but a downside is that we'd generate a large amount of source files cluttering up the project. It'll also require another package to maintain. I have other priorities at the moment, but I think I can come up with a solution for this.

[ziakhan110]

any new on this?

[cervonwong]

This will be a great improvement for dependency inversion and managing dependencies between classes! :)

[CodeDoctorDE]

Seems like there is no active development on this feature?

[srix55]

Would love this. It's much easier to divide and manage with sub projects.

[CodeDoctorDE]

Anay updates? This will help my workflow a lot.

[simolus3]

I've been thinking about this this feature for a long time now, and I think it won't be easy to implement without changing some parts of the user-visible API substantially. That's not necessary a bad thing (after all, this is likely a feature mostly relevant for larger apps with database setups that are okay to be a little more complicated), but I'd love to have some feedback about possible directions.

Basically, "just generate multiple files" gets very unclear in some cases. Say we generate a .drift.dart file for each .drift file with the defined tables. Then we'd generate another .drift.dart file for the main database file importing all the necessary imports so that the tables are visible there. But then:

• where do generated queries go? In the file generated from the .drift file where they are defined? Then they are no longer members on the database class. In the database part? Then we'd still have potential to end up with huge generated files.
• how do we efficiently deal with views and circular imports between .drift files? This is not too bad when everything is resolved in the context from a single entrypoint (the database class), but it gets tricky when we have to analyze each drift file in isolation.
• we need to add new imports in generated code, so we can't generate part files anymore. Probably not a big deal, but that also means that we can't define private members that are only visible in the database part file (like the private superclass for the database class) anymore.

Essentially, my goals for this feature are:

• More fine-grained build-invalidation: Changing a query in some .drift file should not cause the build for the entire database and included files to re-run. It should only update the code in the .drift.dart file generated for that drift file.
• Automatically manage imports in generated code: It's pretty annoying that users have to manually add imports for type converters imported through .drift files because they are referenced in generated code. If we stop generating .part files, we can add those imports automatically.
• For easy maintenance, I want this generation mode to share as much code as possible with the current single-part-file generation approach.

---

My idea on how this feature could behave is:

• For each .drift file, a .drift.dart file is generated. It contains classes for all entities defined in that drift file. Queries are bundled into a class.
  • I can probably come up with some build hacks to avoid analyzing drift files more than once if there are no cyclic imports. For a cycle, that's the way it is :shrug:
• The main file for the database is changed to be a library instead of a part files. The query bundles are exposed as fields.

So e.g. if we had a drift file like:

-- users.drift
CREATE TABLE users (
  id INTEGER PRIMARY KEY,
  -- ...
);

allUsers: SELECT * FROM users;

Then we'd generate a Dart file like

// users.drift.dart

import 'package:drift/drift.dart';

class Users {}
class User {}
class UsersCompanion {}

class UsersQueries {
  final Users _usersTable;

  UsersQueries(this._usersTable);

  Selectable<User> allUsers() {}
}

An in the main database file, we'd generate something like

abstract class $MyDatabase {
  late final users = Users();

  late final UsersQueries usersDrift = UsersQueries(users);
}

So the generation for drift files is closer to what it would be if they were DAOs in Dart, changing how generated query methods are accessed.

If you're interested in this feature (or if you're running into large files generated by drift), I'd love to hear your feedback on this. Do you think the added indirection is worth the benefit of having more a more modular generation approach? If you have other ideas on how this could be implemented, I'm happy to hear about them!

There's also a question of whether the same approach should work for Dart files (if you want to split your Dart-defined tables across different files).

[adummy832]

Once completed, this feature will be a huge help for simple/complex db setups. Having a modern modular approach as well as ease of maintenance is always a good to go!

But one thing is for sure, having CRUD functions for these table definitions is a must have. I don't know how to properly implement this one but I think drift should have a "DriftModelBase" or "CrudInterface" class/interface atleast.

-- users.drift
CREATE TABLE users (
  id INTEGER PRIMARY KEY,
  -- ...
);

-- Optional: custom/complex raw sql query
getHistoryRecords: SELECT ...;

Generated file

/// users.drift.dart
import 'package:drift/drift.dart';

class Users {}
class User {}
class UsersCompanion {}

// Generated crud functions inside parent class and etc.
// index(), view(), create(), update(), delete()
class UsersQueries extends "DriftModelBase" {
  final Users _usersTable;

  UsersQueries(this._usersTable);

  /// Raw sql query gets transformed into drift query?
  getHistoryRecords() {}

  /// Additional user-defined drift query
  ...(){}
}

These are some of my suggestions and hopefully this feature gets on the main branch :)

[simolus3]

Apologies for the long wait, I finally have an update to share on this! As I suggested in my previous comment, preparing drift_dev to support modular code generation is a massive undertaking. I've finally managed to complete the important steps now :)

Previously, the internal analyzer for SQL and other drift elements in drift_dev always did its analysis from the perspective of a central entrypoint (a database or a DAO). This design decision pretty much prevented efficient independent code generation into different files. In the last months, I have rewritten the analyzer to analyze different elements like tables and views in isolation. This unblocks modular code generation, but it also makes the analyzer more efficient and easier to test.

Then, I have started work on generating modular code. This is an opt-in mechanism that doesn't change any of the existing APIs. This work is nowhere near being complete, but you can check a simple example in the modular example in this repository. Basically, the idea is that we generate a .drift.dart file for each .drift file. It contains classes for tables and views as well as top-level fields for indices and triggers. To access queries in drift files, we generate an implicit DAO making only the relevant classes available (simple example: drift file - generated DAO). This DAO is also added to the database class by default, allowing you to access it easily.

So far, I'm fairly happy with this approach - drift manages imports for you, it emits smaller and modular files, incremental rebuilds work. It only supports tables and queries at the moment, but getting the other elements to work should be manageable now. So hopefully there's something to try out soon-ish.

[kuhnroyal]

Should existing @DriftAccessor (DAOs) not just extends from ModularAccessor as well when using modular generation? Or should there be some kind of @DriftModularAccessor?

I can't get the @DriftAccessor.include to work currently (at least for queries).

I am thinking:

1. Generate an additional mixin FooDriftMixin on ModularAccessor for each .drift file
2. Change the the existing class FooDrift extends ModularAccessor to class FooDrift extends ModularAccess with FooDriftMixin
3. Generate the existing DriftAccessor mixin as superclass and mixin each FooDriftMixin that is referenced in the includes

// These can be used basically anywhere
mixin FooDriftMixin on i2.ModularAccessor {
  i3.BarDrift get BarDrift => this.accessor(i3.BarDrift.new);
}

// These work like currently, no change
class FooDrift extends i2.ModularAccessor
    with FooDriftMixin {
  FooDrift(i0.GeneratedDatabase db) : super(db);
}

@DriftAccessor(
  // . queries
  include: {
	'src/foo.drift',
	'src/foo_queries.drift',
	'src/baz_queries.drift',
  },
)
class FooDao extends $FooDao {
	FooDao(super.db);

	/// ....
}

class $FooDao extends ModularAccessor with FooDriftMixin, FooDriftQueriesMixin, BazDriftQueriesMixin {
	$FooDao(super.db);

	/// queries from @DriftAccessor()
}

This changes how the DAOs are declared but I think this would be fine for modular generation.

[simolus3]

Should existing @DriftAccessor (DAOs) not just extends from ModularAccessor as well when using modular generation?

Well, at least they should actually include the DAOs generated for the drift files they include - fixed in a6377085919c8c2270134fddfee90b35dd60c112, but not yet on develop.

Also, interesting idea with the modified mixins! I guess the main advantage from that is to make queries in included files available in the accessor directly instead of having to go over the generated getters. However, an interesting property of modular generation is that two different drift files can define a query with the same name. So you can kind of normalize your drift file layout and have queries with consistent names for common operations. This would stop working if we applied included files as mixins.

Philosophically speaking, I kind of see .drift files as independent DAOs in the modular generation mode. When writing Dart DAOs, you can include them and access them via a getter. But the main way to access the generated query should be by calling the method on that generated class.

[kuhnroyal]

Well, at least they should actually include the DAOs generated for the drift files they include - fixed in https://github.com/simolus3/drift/commit/a6377085919c8c2270134fddfee90b35dd60c112, but not yet on develop.

That works now, but the DAO has wrong table accessors when it extends DatabaseAccessor<GeneratedDatabase> or ModularAccessor.

Philosophically speaking, I kind of see .drift files as independent DAOs in the modular generation mode. When writing Dart DAOs, you can include them and access them via a getter. But the main way to access the generated query should be by calling the method on that generated class.

I think that makes sense. I managed to migrate most of our code. But traditionally DAOs are often used for a single entity class. So far we mixed in basic CRUD queries into each DAO for a main entity class. Now there is not much left in the DAO except these CRUD queries and the concern is that, if we move them to drift files, we will have copy & paste errors.

I wonder if Drift can help here with some TBD magic comment/query syntax that leads to Drift generating CRUD queries. I think that could also benefit a lot of people to get the initial jump start when testing or adopting the library.

-- drift:generate-crud(foo)

OR 

crud(foo):

leads to

findAllFoo:
SELECT * FROM foo;

findFooById:
SELECT * FROM foo WHERE PK = :PK;

deleteFooById:
DELETE * FROM foo WHERE PK = :PK;

existsFooById:
SELECT EXISTS (SELECT 1 FROM foo WHERE PK = :PK);

....

[simolus3]

the DAO has wrong table accessors when it extends DatabaseAccessor<GeneratedDatabase> or ModularAccessor.

It seems to work for this one, can you share a broken example?

I wonder if Drift can help here with some TBD magic comment/query syntax that leads to Drift generating CRUD queries.

Interesting idea, what about CREATE TABLE foo (...) WITH CRUD;? I think it doesn't really fit the existing one-DAO-per-drift-file model, but I can see us creating a new DAO for selected tables.

We could also allow adding explicit DAOs in drift files:

CREATE TABLE foo (...);

DAO FooDao WITH crud(foo) BEGIN
  /* additional queries that should go into this DAO */
  search: SELECT ... FROM foo_fts5 WHERE foo_fts5 MATCH ?; 
END;

[kuhnroyal]

It seems to work for this one, can you share a broken example?

This one has a DatabaseAccessor<Database> not GeneratedDatabase .

We could also allow adding explicit DAOs in drift files

I don't know, one DAO per file is actually nice, this would also break this. But I really think that the declaration for the CRUD queries, whatever it may look like, should not be tied to the table declaration. For my projects all queries are in separate files, not in the table files :)

[simolus3]

That works now, but the DAO has wrong table accessors when it extends DatabaseAccessor<GeneratedDatabase> or ModularAccessor.

Oh right, I've misread that part. Should be fixed in 16e6aaf4fec54ce06d3a6d4498dc13c7624d10a5. I've moved the idea of generating CRUD accessors for given tables into another issue.

If you run into any more problems with modular generation, please comment here or open new issues as well. Thanks for all your help testing this feature!