Migration crashes when sorting the old migration by a field deleted in the new migration.

[artyom-ivanov]

I changed the Realm schema and want to create new entities based on the previous ones. If for a new entity I use the same name that was in the previous scheme, then the migration crashes with this error:

 exception in notifier thread: N5realm10LogicErrorE: Column does not exist 

If I rename the entity in the new migration to something other than "Routine", then everything goes correctly.

I use Realm 10.8.0.

Old schema:

const OLD_SCHEMA = [
  {
    name: 'Routine',
    primaryKey: 'id',
    properties: {
      id: 'string',
      title: 'string',
      triggerDays: 'int[]',
      triggeredOn: 'string[]',
      tiles: 'RoutineTile[]',
      categories: 'string[]',
      builtInBehavior: { type: 'string', optional: true },
      createdAt: 'date',
    },
  },
  {
    name: 'RoutineTile',
    embedded: true,
    properties: {
      id: 'string',
      categories: 'string[]',
      title: 'string',
      duration: 'int',
      planningType: 'string',
      plannedTime: 'int[]',
      plannedEndTime: 'int[]',
    },
  },
];

New schema:

const NEW_SCHEMA = [
  {
    name: 'Routine',
    primaryKey: 'id',
    properties: {
      id: 'string',
      active: { type: 'bool', default: true },
      title: 'string',
      triggerDays: 'int[]',
      triggeredOn: 'string[]',
      categories: 'string[]',
      activityId: { type: 'string', optional: true },
      duration: 'int',
      builtInBehavior: { type: 'string', optional: true },
      position: 'int',
      plannedTime: 'int[]',
      plannedEndTime: 'int[]',
      planningType: 'string',
    },
  },
];

Migration:

function migration(oldRealm, newRealm) {
  let oldRoutines = oldRealm.objects('Routine').sorted('createdAt');
  let oldRoutineTiles = [];
  let newRoutines = [];

  // 1. Generate hashmap to get routine box from routine tile
  let routinesMapping = {};
  oldRoutines.forEach((oldRoutine) => {
    oldRoutineTiles = [...oldRoutineTiles, ...oldRoutine.tiles];
    oldRoutine.tiles.forEach((oldRoutineTile) => {
      routinesMapping[oldRoutineTile.id] = oldRoutine;
    });
  });

  // 2. Generate new routines
  let positionIndex = 0;
  for (let oldRoutineTileId in oldRoutineTiles) {
    let oldRoutineTile = oldRoutineTiles[oldRoutineTileId];
    let oldRoutine = routinesMapping[oldRoutineTile.id];

    if (!oldRoutine) {
      return;
    }

    newRoutines.push({
      activityId: null,
      id: oldRoutineTile.id,
      active: true,
      title: oldRoutineTile.title,
      position: positionIndex,
      triggerDays: oldRoutine.triggerDays,
      triggeredOn: oldRoutine.triggeredOn,
      categories: oldRoutineTile.categories,
      plannedTime: oldRoutineTile.plannedTime,
      plannedEndTime: oldRoutineTile.plannedEndTime,
      duration: oldRoutineTile.duration,
      builtInBehavior: oldRoutine.builtInBehavior,
      planningType: oldRoutineTile.planningType,
    });

    positionIndex++;
  }

  // Delete all old entities
  newRealm.delete(copyRoutines);

  // Generate new entities
  newRoutines.forEach((newRoutine) => {
    newRealm.create('Routine', newRoutine); // ⛔️ Crashes here
  });
}

I know that it would be hard to debug that just from that data, so could send the whole realm file if necessary.

[takameyer]

@artyom-ivanov Thanks for reporting. I believe it would also be easier for us to attempt to look into this with your realm file. Please send it to [email protected] and we can investigate further.

[artyom-ivanov]

@takameyer I sent an email

[takameyer]

Hello @artyom-ivanov, sorry about the delay. I finally around to setting up an environment to replicate the migration using the data you sent us. I did not see any issues. However I set this up using node. In which platform are you seeing this error? React Native?

[artyom-ivanov]

Hi, @takameyer! Yep, we use React Native.

[takameyer]

@artyom-ivanov I was able to reproduce the issue you are having. The bug stems from querying oldMigration and sorting on a value that is deleted createdAt. It may take some time for us to come up with a fix for this, but in the meantime, you could just copy unsorted collection into a new Array and write a sort function. For example:

  let oldRoutines = [...oldRealm
    .objects<OLD_SCHEMA['routine']>('Routine')
    ].slice().sort((a,b) => {
      if (a.createdAt === b.createdAt) {
        return 0;
      }
  
      return ((a?.createdAt ?? 0) > (b?.createdAt ?? 0))
        ? 1
        : -1
    })

Let us know if that helps.

[somebody32]

Hi @takameyer,

I'm helping @artyom-ivanov with this issue. Your solution works, thanks.

But just curious, we've tried kinda the same thing by calling toJSON on a resulting collection, but it didn't help, so do you have any idea why? I thought it would convert results into JS primitives and then we can just work with them directly

[takameyer]

@somebody32 Can you perhaps provide more context into what you want to do with toJSON? What exactly isn't working for you?

[somebody32]

@takameyer following your example I was hoping that this code would do the same: getting the list of objects already sorted and then converting them to POJOs, but for some reason, creation is still failing with Column does not exist in this case.

So I'm curious what is wrong with my mental model of that code :)

  let oldRoutines = oldRealm
    .objects<OLD_SCHEMA['routine']>('Routine')
    .sorted('createdAt')
    .toJSON();

[takameyer]

@somebody32 Your mental model is correct and this should work, but we have a bug here. Just invoking sorted on a deleted field, during a migration will cause a crash (you don't even have to create anything). My assumption from reading the stack trace, is that a notifier is being associated with that sorted realm data that has landed in memory. When the migration code is finished, it tries to fire the notifier on the deleted field.
The workaround just avoids calling sorted altogether by using native javascript methods.

[somebody32]

@takameyer, thanks, you're right! Even accessing properties on the old collection causes the crash. But as soon as the migration passes on the second run, it looks like we reproduced it poorly and pinned the problem on creation.

[takameyer]

@somebody32 I've written a failing test for this. https://github.com/realm/realm-js/pull/4030 We will fix this at some point, but I can't offer a timeline. At least there is a way around it, and an app should only need to run the migration function once per version.

[somebody32]

@takameyer no worries, thanks! we've managed in our case without a lot of hassle, that issue was just for you to know about the problem