Optimize copyToRealm / copyToRealmOrUpdate

[asarazan]

It doesn't happen on genymotion, only on a physical Android device.

From my logs:

Benchmark realm clear: 73ms
Benchmark realm create 1183 objects: 398ms
Benchmark realm remote: 642ms
Benchmark realm local clear: 0ms
Benchmark realm local create: 11ms
Benchmark realm local: 26ms
Benchmark card listener: 20025ms

As you can see, the actual calls to clear existing objects, and then even construct and copy the new objects, take around 1s. Then it churns for anywhere from 20-45(!!!!!) seconds on a lot of ART gc calls.

We were ready to ship to production and now we may have to scrap a week's worth of work, please advise.

Including the top level RealmObject description below. Note that there are mutiple possible sub-objects that are declared inside the top level. It should only ever actually contain one of them.

@RealmClass
open class DisplayCard() : RealmObject() {

    @PrimaryKey
    open var id: String = ""

    open var sort: Long = 0L
    open var typeOrdinal: Int = 0
    open var dismissible: Boolean = false
    open var local: Boolean = false
    open var protoBytes: ByteArray? = null
    open var data: ByteArray? = null // Currently only used for CardOptions class/proto
    open var relevantTids: RealmList<RealmString> = RealmList()
    open var relevantBids: RealmList<RealmString> = RealmList()

    // Sub Cards
    open var normie: DisplayCardNormie? = null
    open var recur: DisplayCardRecur? = null
    open var track: DisplayCardTracker? = null
    open var transfer : DisplayCardTransfer? = null
    open var suggestedBucket: DisplayCardSuggest? = null
    open var suggestedExisting: DisplayCardSuggestExisting? = null
    open var text: DisplayCardText? = null
    open var adjust: DisplayCardAdjust? = null
    open var newMonth: DisplayCardNewMonth? = null
    open var link: DisplayCardLink? = null
    open var explainer: DisplayCardExplainer? = null
}

[cmelchior]

Hi @asarazan Sorry to hear that. That amount of GC sounds really strange though. Can you share a sample project that reproduces this? and can you share the details of exactly what device you where testing on?

[asarazan]

I'm testing on a Nexus 5X running stock Marshmallow. If you give me an email address I'm happy to send you a full list of class definitions and my allocation capture file.

[cmelchior]

Thank you. You can send it to [email protected]

[asarazan]

Sent.

[asarazan]

It's possible that this is specific to Marshmallow's implementation of ART. It doesn't appear to happen on our lollipop samsung devices. About to test on a Lollipop Sony.

Edit: Eh still 20 seconds on a Lollipop Z3 Compact

[cmelchior]

Did some testing with the following unit test (using classes from out unit test suite)

        List<OwnerPrimaryKey> owners = new ArrayList<>();

        for (int i = 0; i < 10000; i++) {
            OwnerPrimaryKey owner = new OwnerPrimaryKey(i + 1, "Owner:" + i);
            owner.setDogs(new RealmList<Dog>());
            DogPrimaryKey dogPk = new DogPrimaryKey(i + 10000, "Dog");
            owner.setDog(dogPk);

            for (int j = 0; j < 10; j++) {
                Dog dog = new Dog("Dog" + i + " - " + j);
                owner.getDogs().add(dog);
            }

            owners.add(owner);

        realm.beginTransaction();
        long start = System.currentTimeMillis();
        realm.copyToRealmOrUpdate(owners);
        long end = System.currentTimeMillis();
        Log.e("Bench", "copyToRealmOrUpdate_manyObjects: " + (end - start));
        realm.commitTransaction();

It takes about 27 seconds to insert 100.000 objects on my Nexus 7 (6.0.1) and about 1.7 seconds on a Genymotion x86 emulator. The change seems to be linear though as 10.000 objects take about 2.7 seconds.

@asarazan Do you know how many objects in total you are importing, also including referenced objects. You main model reference at least to RealmLists so I'm guessing it is a potentially quite large set?

[cmelchior]

Digging further into this, it doesn't look like it's a bug. The amount of GC is actually very small, so this is a function of a relatively slow copyOrUpdate method that does a lot of things that could be optimised. In no prioritised order:

• Avoid allocating RealmObjects that are not returned to the user (beyond depth 0)
• Adding elements to RealmLists adds a big overhead in terms of additional checks.
• Disable thread check while inserting objects.
• Consider a copyOrUpdate method that doesn't return any objects. That way we can remove allocations for depth 0 as well.
• See also https://github.com/realm/realm-java/issues/1684

This also means that there currently is no easy work-around. Some suggestions could be:

1. Prepare realm files that can be downloaded instead of doing it on the device.
2. Use createOrUpdateUsingJson* methods to skip intermediate steps.

[asarazan]

Hey Christian, thanks for looking so deeply into this. Regarding your previous question, I did an object count a couple days ago and I think it was around 4,300 total. Over the weekend, I've been attempting to find workarounds-- non-blocking small-batch processing, null instead of empty realmlists, etc. I came away with roughly unchanged results.

Since it seems like my use case isn't well supported at the moment, and we don't use json in our Android app, I'll probably just roll back to our old cursor structure from before the refactor.

Thanks for all your help!

[cmelchior]

No problem, and sorry I didn't have a better answer to this.

[nthuat]

public <E extends RealmModel> List<E> copyToRealmOrUpdate(Iterable<E> objects) {
        if (objects == null) {
            return new ArrayList<E>(0);
        }
        ArrayList<E> realmObjects = new ArrayList<E>();
        for (E object : objects) {
            realmObjects.add(copyToRealmOrUpdate(object));
        }
        return realmObjects;
    }

@cmelchior I can't find any method that returns void instead of a collection. I just want to add all objects into realm but I don't want return any objects. I think it's better and faster for my case. Do you have any idea?

[cmelchior]

@thuat26 No such method exists currently. This will most likely come as part of of #1684 . For now I would just ignore the returned collection.

[Zhuinden]

This was released in v1.1.0

[cmelchior]

Both yes and no. insertOrUpdate() could be optimized to a much larger degree than copyToRealm ever could be, at the expense of returning void. We still have room to optimize copyToRealm(), but it probably has a lot lower priority now.

[sipersso]

I used copyToRealmOrUpdate, but changed it to insertOrUpdate as per this thread. However, the Items I want to insert have a two way releation. In the case of copyToRealmOrUpdate it works, but the insertOrUpdate results in a stackoverflow error. Do I have to use copyToRealmOrUpdate if I have bidirectional relations in the object I want to insert?

[Zhuinden]

You would need to insert them with copyToRealmOrUpdate in order to receive a managed proxy that you can use to set each other as each other's links

(but it's worth noting that you shouldn't get stack overflow error anywhere, so that's a bug)

[cmelchior]

@sipersso It could be a bug. Could you create an separate issue with code to reproduce it?

[sipersso]

@Zhuinden I don't understand. The objects I try to insert already have their relations set, although they were set in a separate realm file. copyOrUpdate works just fine and sets the relations correct. It's just a little slow.

@cmelchior Sure, I'll see if I can set up a simple example to reproduce it.

[sipersso]

I created a separate issue for the StackOverflowError at https://github.com/realm/realm-java/issues/3732