Customizing migration detection and generation

[dosolkowski-work]

trafficstars

Is there a correct/supported way in EF Core 8 (or planned for 9+) to customize the detection of model changes that should result in a migration, and customize the migration generation itself? We would like to track table permissions as part of our model so they can be specified declaratively and let code decide when and how to grant/revoke, without having to do it all as manual migration operations. For example, when we recently changed up our database service roles a bit, I had to go back through our entire migration history to reconstruct what the permissions should be to apply them to a new role.

My current best guess is that, although I could use annotations to store this custom data in the model, it requires deep diving into MigrationsModelDiffer--which is considered internal and unstable--in order to extend EF Core to understand something has actually changed that requires a migration. Is that the only option?

[roji]

You can indeed add annotations to the model; this shouldn't involve any changes to the differ, which works in a general way over any annotations. You'd need to make sure these annotations are flown to the migrations, and then pick them up in a customized MigrationsSqlGenerator. For example, any annotation change on an entity type would ultimately result in an AlterTableOperation reaching the MigrationsSqlGenerator; your customized implementation of it would be aware of your custom annotations and generate the correct DDL to account for the changes.

[dosolkowski-work]

@roji Thanks for the quick response! One question, what do you mean by "make sure these annotations are flown to the migrations"? I think that's the key part I'm missing right now. If I just manually add an annotation to a table, and then run dotnet ef migrations add SomeTest, there isn't anything in the generated file. If I write a custom CSharpMigrationOperationGenerator to change how the file is generated, there are no migration operations passed to it. Something upstream of that in EF Core has already decided that the annotation is not something that requires migration--I have a bad feeling that's the MigrationsModelDiffer that can't be easily and safely customized, so I think that's where I'm stuck?

[dosolkowski-work]

We configure our tables using classes implementing IEntityTypeConfiguration<Table> and ApplyConfigurationsFromAssembly(), which seems to work fine. In the IEntityTypeConfiguration<Table>.Configure(EntityTypeBuilder<Table> builder) method, I call builder.HasAnnotation("property", "value") for test purposes, and I see it show up in the context snapshot. However, if I try to customize MigrationsModelDiffer.Diff(ITable source, ITable target, DiffContext diffContext) (I think that's about here in 8.0) just to experiment, that annotation doesn't show up in either source.GetAnnotations() or target.GetAnnotations(), so there won't be an AlterTableOperation generated. Is there a different way I need to add the annotation?

[roji]

what do you mean by "make sure these annotations are flown to the migrations"?

Check out e.g. SqlServerAnnotationProvider - if you want model annotations to flow to the migrations SQL generator, they have to be added there. No need to touch the differ.

[dosolkowski-work]

That would require us to generate SQL code to handle any prior state though, because by that point we don't know what the prior model is, right? I would like to build something that runs when I execute dotnet ef migrations add <Something> that will let me inject code into the <Timestamp>_<Something>.cs file that is generated which can then say "grant permissions A and B, revoke permissions X and Y", because it knows what the permissions were before and what they are now. I'm not sure this can be done at any point afterwards.

Additionally, I tried making a custom annotation provider just to experiment and I could not get it to run during dotnet ef migrations add; if it is tied to the particular database being used, then I assume it doesn't even run until the migrations are actually being turned into SQL?

[dosolkowski-work]

I was able to create a minimal reproduction: https://github.com/dosolkowski-work/dotnet-ef-migrations-issue

[ajcvickers]

I investigated this--thanks for the code @dosolkowski-work. As I suspected, the reason this never works for anyone is that the annotation provider has to be replaced in the runtime services. For example:

        var options = new DbContextOptionsBuilder<MyContext>()
            .UseNpgsql(builder.ConnectionString)
            .ReplaceService<IRelationalAnnotationProvider, MyAnnotationProvider>()
            .Options;

Just replacing it in the design-time services, which is the obvious thing, does not work. I created a docs issue for this a long time ago, but it never bubbled up high enough to do it.

Marking for discussion by the team into whether or not we should make the design-time registration work here, and other places where this is a problem.

This is probably a dupe of #10258

[AndriySvyryd]

Due to the way we set up the service provider ConfigureDesignTimeServices can only replace the design-time services and their direct dependencies:

• IDatabaseProvider
• IMigrationsIdGenerator
• IRelationalTypeMappingSource
• IModelRuntimeInitializer
• LoggingDefinitions
• ICurrentDbContext
• IDbContextOptions
• IHistoryRepository
• IMigrationsAssembly
• IMigrationsModelDiffer
• IMigrator
• IDesignTimeModel

[dosolkowski-work]

It's still confusing how all of this is connected together--it feels like EF Core has two different sides that aren't talking to each other.

If I create an IEntityTypeAddedConvention or IModelFinalizingConvention, it definitely runs during an operation such as dotnet ef migration add; in fact, it seems to run twice, first a "not design time" pass and then a "design time" pass, I think? Then I can call entityTypeBuilder.Metadata.AddAnnotation() or entityTypeBuilder.HasAnnotation(), but the annotation I'm adding seems to have no effect--where does it go?

If I instead do the scary overriding of "pub-ternal" IRelationalAnnotationProvider and IMigrationsModelDiffer, the annotations from the convention are just nowhere to be found (even though the convention runs first), but the annotations injected by IRelationalAnnotationProvider do appear in IMigrationsModelDiffer and in theory I would be able to turn those into migration operations.

Is there a reason the changes made by conventions seem to have no effect, even though they're definitely running?

[dosolkowski-work]

Okay, so my best guess at the moment is there's a generic "base" model, and then a completely separate Relational model. The new conventions system operates only on the base model, not the Relational model. When the Relational model is created and derived from the base model, the default behavior is to throw out all annotations that may have been created on the base model. I don't know if there's anything else that Relational throws out as well; maybe other data is also lost.

By the time we get to the point of dealing with creating or applying migrations, the only model left is the Relational one, so anything our conventions have done to the base model that doesn't get copied over just disappears with no explanation.

[AndriySvyryd]

@dosolkowski-work Most of your understanding is correct.

The conventions add the annotations to the core Model while it's mutable. After model building is finished, EF creates the relational model from the core model as a view that will be used for Migrations and other relational code. You need to also implement IRelationalAnnotationProvider to find the annotations on the mapped entity types and output an annotation for the table. Could you update the repro with the suggestion from @ajcvickers to show the issue where the annotations don't appear?

It's not recommended to replace IMigrationsModelDiffer. If any annotations change the default implementation should already generate an operation.

[dosolkowski-work]

@AndriySvyryd Thanks, this is making a bit more sense now, if I can find some time I will see if I can make some minimal reproductions.

If we are not intended to replace IMigrationsModelDiffer, is there a different/better way to transform the annotation changes into custom migration operations? Should we customize/replace CSharpMigrationOperationGenerator instead? That requires different/extra package references than what is normally used, although I think the package references end up being simpler in the end.

[AndriySvyryd]

If we are not intended to replace IMigrationsModelDiffer, is there a different/better way to transform the annotation changes into custom migration operations? Should we customize/replace CSharpMigrationOperationGenerator instead? That requires different/extra package references than what is normally used, although I think the package references end up being simpler in the end.

@dosolkowski-work As long as the values of the annotations added by IRelationalAnnotationProvider are strings or numeric you shouldn't need to create a custom operation, only implement IMigrationsSqlGenerator. And this would be improved by #6546

Also see #10258 and #17740

[paillave]

what do you mean by "make sure these annotations are flown to the migrations"?

Check out e.g. SqlServerAnnotationProvider - if you want model annotations to flow to the migrations SQL generator, they have to be added there. No need to touch the differ.

You realise that google returns only 6 entries on what you ask to check out, and that none of them is the start of the begining of the shadow of an actual documentation?

Actually, this whole thing related to how to do custom thing related to migration generation has not a single start of official documentation. I am compelled to see source of EFCore of git hub to maybe have a change to understand what I should do.

What did you mean by checkout SqlServerAnnotationProvider? Where can I get more information about this?

[AndriySvyryd]

I am compelled to see source of EFCore of git hub to maybe have a change to understand what I should do.

That's the correct approach for this advanced scenario

What did you mean by checkout SqlServerAnnotationProvider? Where can I get more information about this?

Look at https://github.com/dotnet/efcore/blob/main/src/EFCore.SqlServer/Metadata/Internal/SqlServerAnnotationProvider.cs for an example of how this is done.

Basically, it would be something like

public override IEnumerable<IAnnotation> For(ITableIndex index, bool designTime)
{
    var baseAnnotations = base.For(index, designTime);

    if (!designTime)
    {
        return baseAnnotations;
    }

    var customAnnotations = index.MappedIndexes.SelectMany(i => i.GetAnnotations())
        .Where(a => a.Name == "SqlServer:IndexCreateIfNotExists")
        .Take(1);

    return baseAnnotations.Concat(customAnnotations);
}