BulkInsert: How to not re-insert previously inserted entities?

[will-heger]

First, thanks for this excellent product. I contract for the University of Chicago and they recently acquired a license.

Problem: I have a LARGE and complex graph I wish to Bulk Load.

If I BulkInsert beginning with any BulkInsert<Entity> and include the graph, BulkInsert will smartly only attempt to insert a unique entity once (even if it's reference several times). The problem is that I would like to carry that behavior forward and continue to load other "root" Entities which may themselves reference previously loaded items in their graphs. But if I launch additional BulkInsert operations, I will get duplicate key exceptions.

In effect, I would like to bulk insert graphs, but avoid reinserting any previously loaded items. Or otherwise, BulkInsert multiple types under the same operation, such that an already loaded entity isn't reloaded. Under EF, I would mark a previously loaded item as unchanged, but under EntityFrameworkExtensions, I seem to need to block them out by type via "bulk.IsReadOnly = true;" I tried to use "InsertIfNotExists" but it didn't seem to apply to items in the graph.

EDIT: Oh sugar, I just tried this:

options.IncludeGraphOperationBuilder = operation => {
     operation.InsertIfNotExists = true;
}

And it looks like it works! If this is the right approach, please feel free to close. Thank you!

[JonathanMagnan]

Hello @will-heger ,

Just to make sure, do your entity already have an ID when inserting (so no identity column is used)?

It was already planned to put some time this week to add chaining operation and review some kind of issue like this one reported to use with the IncludeGraph options.

Best Regards,

Jonathan

[will-heger]

That's exactly right. All the entities in the ReadModel have IDs already assigned and they are their respective primary keys, so no identity column was specified in the options.

Also yes, a sort of "chain" of operations to work across types would be great if it meant that subsequent loaded entities were skipped. I need to do some more testing, but "operation.InsertIfNotExists = true;" applied within the "IncludeGraphOperationBuilder" seems to sidestep the problem, though probably introduces some more round trips.

Thanks @JonathanMagnan