Poor insertion performance 10k+ records

[cmcnicholas]

Goals

I am trying to understand why the performance of inserts is so poor from a react native environment vs xamarin (using the Realm .Net library).

My use case is initial syncing a mobile application with a backend of items (things like street data, assets, jobs etc.) that can range from 10-500k records for use offline over the period of a week to later be synced back on completion.

The .Net library appears to be orders of magnitude faster and has a more linear timing curve for large inserts, realm-js inserts degrade over time given my tests.

Expected Results

I am expecting similar performance on inserting many records in the realm-js library as it is in the .Net library.

Actual Results

Test	1k inserts	10k inserts	100k inserts
realm-js	0.45s	5.8s	200s
Realm.Net	0.5s	1.78s	15.4s
realm-js is...	10% faster	225.8% slower	1198.7% slower
realm-js (no indices)	0.33s	2.78s	26.9s

Steps to Reproduce

Attempt to insert a large number of records into a realm.

Code Sample

The following should be copy-pasteable into a brand new react-native app with typescript as the App.ts file. Alternatively the onPress function and models at the end of the code snippet should be all that's necessary.

The .Net code is almost like for like.

import React, {Component, Fragment} from 'react';
import {Button, Alert} from 'react-native';
import Realm from 'realm';

// test app component
export default class App extends Component {
  render() {
    return (
      <Fragment>
        <Button title="My Button" onPress={() => this.onPress()} />
      </Fragment>
    );
  }

  // run a fake sync
  onPress() {
    requestAnimationFrame(async () => {
      const config: Realm.Configuration = {
        path: 'myDb',
        schema: [ItemSchema],
      };

      Realm.deleteFile(config);

      // make some items to eventually insert
      const itemsToInsert: Item[] = [];
      for (let i = 0; i < 10000; i++) {
        itemsToInsert.push({
          collection: 'Live',
          colour: '#800855',
          dodiCode: 'designs_myDesign',
          icon: 'icon-teste',
          itemId: '5e4baedd22475918f03acf4a' + i,
          subtitle: 'alright partner',
          title: 'keep on rollin baby!',
        });
      }

      const realm = await Realm.open(config);

      // start timing
      const start = new Date().getTime();

      realm.write(() => {
        for (let i = 0, total = itemsToInsert.length; i < total; i++) {
          realm.create(ItemSchema.name, itemsToInsert[i], Realm.UpdateMode.All);
        }
      });

      // end timing
      const end = new Date().getTime();

      realm.close();

      Alert.alert('finished: ' + (end - start) + 'ms');
    });
  }
}

// schema for item
const ItemSchema: Realm.ObjectSchema = {
  name: 'item',
  primaryKey: 'itemId',
  properties: {
    itemId: {
      type: 'string',
      optional: false,
    },
    dodiCode: {
      type: 'string',
      optional: false,
      indexed: true,
    },
    collection: {
      type: 'string',
      optional: false,
    },
    title: {
      type: 'string',
      optional: true,
      indexed: true,
    },
    subtitle: {
      type: 'string',
      optional: true,
    },
    icon: {
      type: 'string',
      optional: true,
    },
    colour: {
      type: 'string',
      optional: true,
    },
  },
};

// model for item
interface Item {
  itemId: string;
  dodiCode: string;
  collection: string;
  title: string | null;
  subtitle: string | null;
  icon: string | null;
  colour: string | null;
}

Version of Realm and Tooling

• Realm JS SDK Version: ^4.0.0-beta.0
• Node or React Native: React Native
• Client OS & Version: Android Emulator and Physical Device
• Which debugger for React Native: None

[cmcnicholas]

Updated table with results when no indices are specified, why are indices in the .Net library so much less expensive than realm-js?

[kraenhansen]

Thanks for reporting this @cmcnicholas - we need to investigate this further. There has recently been reported other performance related issues, you're probably experiencing the same regression. Have you tried running the benchmark with other versions of Realm JS? For example version 3?

[cmcnicholas]

Thanks for reporting this @cmcnicholas - we need to investigate this further. There has recently been reported other performance related issues, you're probably experiencing the same regression. Have you tried running the benchmark with other versions of Realm JS? For example version 3?

unfortunately the window I had for testing Realm JS and .Net for our use case is over and due to the large number of records we have meant we ended up going with a Xamarin based solution.

[noldidrita]

@kraenhansen

I've been having the same issues with very poor performance on a large number of insertions (~5K). It takes around 30s-60s for the insertions to complete, even in Release mode, sometimes even more. While with Realm v3 it takes ~2s.

[oojikoo]

is it true that Realm v3 is faster?!

[noldidrita]

Yes, totally! I've tried v6 for a couple of releases, then v10, but ultimately switched to v3.6 as it's the only version that's consistent enough (had a few crashes on both v6 and v10 that never happen on v3).

[bmunkholm]

@noldidrita Which was the last version you tried with? We did fix and shipped a performance regression. So if you notice any other major regressions, we would love to hear about them!

[oojikoo]

@noldidrita thank you! but I decided to get realm out of the project. waste so much time with this.

[steffenagger]

These tests are a bit outdated, seeing as they were done with [email protected]. I re-ran the supplied sample-code with [email protected] & [email protected] on iOS Simulator & an Android emulator.

(simulator/emulator benchmarks are not ideal, but perhaps ok for comparison in this case)

[email protected]:

(similar times found with [email protected])

Test	1k inserts	10k inserts	100k inserts
realm-js iOS	144ms	777ms	7.36s
realm-js Android	201ms	1.65s	18.03s

[email protected]:

(for comparison)

Test	1k inserts	10k inserts	100k inserts
realm-js iOS	240ms	884ms	6.04s
realm-js Android	148ms	881ms	7.52s

We'll leave the issue open, primarily with focus on Android, which seems to be lacking a bit behind.

[FSPinho]

Yes, totally! I've tried v6 for a couple of releases, then v10, but ultimately switched to v3.6 as it's the only version that's consistent enough (had a few crashes on both v6 and v10 that never happen on v3).

I tried to downgrade to 3.6 but I'm unable to install Realm with node 15. What node version you guys had to use?

[steffenagger]

I tried to downgrade to 3.6 but I'm unable to install Realm with node 15. What node version you guys had to use?

@FSPinho node@10

[vikas-singh-fareye]

any update on this? facing same issue, finally reverted it back to v3.6.0. AsyncStorage seems faster than realm v6.1.5 & v10.0.1 😅😅

[FSPinho]

I fixed it without downgrade by doing the following:

• I was using UUID strings as primary keys. Once I changed to integers, the speed increased a lot! For sure, I forgot the professor's tips on my DB classes.
• Also, if you are checking any other property during inserts besides the primary key, make sure to enable indexing on it.
• Besides that, I was trying to use a loading during the insertion that was causing re-renders. That may sound obvious but didn't realize.

Now I'm getting good times for realm@10!

[kneth]

@FSPinho Thank you for your real-world tips!!

@vikas-singh-fareye If possible, it would be very interesting if you can share some code which we can profile to find the bottlenecks.

[vikas-singh-fareye]

Version of Realm and Tooling

• Realm JS SDK Version: 10.0.1
• Node or React Native: RN
• Client OS & Version: Android 10
• Which debugger for React Native: NA

const schemaVersion = 110;
const schema = [Table1.schema, Table2.schema,.....TableX.schema];
var encryptionKey = new Int8Array(64);

let realm = new Realm({
    schemaVersion,
    schema,
    encryptionKey,
    shouldCompactOnLaunch: (totalBytes,usedBytes)=>{
		return true
	}
});

// Generic method to save multiple realm records
export function saveList(tableName, array, update = true) {
    return realm.write(() => {
        array.forEach(data => realm.create(tableName, data, update));
    });
}

I just updated realm from v3.6.0 to v6.1.5, nothing else has been changed and noticed this slowness issue, then upgraded it to v10.0.1 though there wasn't anything related to it in the v10 change log and issue still persists in both debug and release mode. So finally reverted it back to 3.6.0.

[sync-by-unito[bot]]

➤ Finn Andersen commented:

Confirmed: performance regression for string based primary keys.

[jaltin]

Hi @kneth,

Did you make any progress on this? We were forced to find an alternative solution because of this performance issue back in 2020 and I was wondering if it has been resolved so we can try it out again.

[kneth]

@jaltin Only for string based primary keys (fix released in v10.5.0).

[jaltin]

@jaltin Only for string based primary keys (fix released in v10.5.0).

Thanks for quick reply. I will try to test it out.

Also, do you have any idea if it will be addressed further (and if so when) for other types of keys?