[NEW FEATURE REQUEST] Generate index for old data

[heavenwing]

After add index into a Dictionary with old data, this index can generate indexed data for old data. Or can rebuild indexed data for whole data include old data and new data.

[jfalameda]

What is the correct approach to add an index on old data when migrating to service-fabric-indexing? The old data on my collections are not indexed.

[jessebenson]

Yeah ... this was always a gap. In my mind, there's two ways to do this:

[1] Add an API to IReliableIndexedDictionary like RebuildIndexAsync(), which will enumerate the values in the dictionary and add/update to the indexes. [2] In a single transaction: remove the value from the IReliableDictionary, add the value to the IReliableIndexedDictionary, commit. Do this for all values in the dictionary.

Service Fabric has limitations on the size of changes (in bytes) within a single transaction. So both approaches above will likely need to be done across multiple transactions. That has consistency considerations.

Alternatively, I would encourage you to raise the issue with Service Fabric directly to add index support to IReliableDictionary directly :)

[jfalameda]

Thanks a lot. This is very useful.

[jfalameda]

Just to make sure, you are proposing that I create a separate dictionary? Or that I just wrap the IReliableDictionary into the IReliableIndexedDictionary and then perform delete/add?

Kind regards, José.

[jessebenson]

I was suggesting using the same dictionary, something like below:

var dict = await stateManager.GetAsync<...>("dict-name");
var index = await stateManager.GetIndexedAsync<...>("dict-name");
for ((key, value) in dict) {
   using (var tx = ...) {
      await dict.RemoveAsync(key);
      await index.AddAsync(key, value);
      await tx.CommitAsync();
   }
}