Support Projections (incl. View Models and AutoValue)

[cmelchior]

When it comes to displaying data on the UI thread there are roughly 3 approaches in the current database eco-system on Android:

1. Copy database entities directly into memory, with some rules on how to handle relationships (Most ORM's).

2. Allow you map entities and queries to projection classes (Room),

3. Expose entities, but lazy-load everything (Realm).

All of the above has advantages and disadvantages, some more than others, but I'll focus specifically on the differences between 2) and 3).

Room Projection Classes

• Each class has a single responsibility.
• Only loading the properties needed makes it memory-efficient.
• You need to modify the projection class every time the UI requirements change.
• More classes in the system.
• All work is guaranteed to be run on background threads.
• Decoupling from Cursor makes it hard to do "infinite scrolling"

Realm Lazy-loaded entities

• Fewer classes overall in the system.
• Lazy-loading all properties makes it memory-efficient (with a caveat around deeply nested object hierarchies).
• Exposing Realm data directly makes it very easy to do "infinite scrolling".
• Tighter coupling between model and UI layer.
• Rely on the operating system to do memory-mapping efficiently (This works in practice but corner cases might exists).
• Realm behavior leak into other parts of the architecture (like threading constraints).

As seen above there are both advantages and disadvantages to both approaches, but one key complaint we do hear from people not wanting to adopt Realm is how it tends to creep into all layers of the architecture. This proposal is thus an attempt to add the concept of Projections to Realm.

Use cases supported

• Decouple Realm classes from the UI layer.

• Add support for AutoValue. See https://github.com/realm/realm-java/issues/2538

• Add support for Kotlin data classes.

• Prevent unnecessary allocations when fetching objects from deep object hierarchies.

• Make it possible to bulk-load data, like e.g. ObjectBox does it.

• Move all work to a background thread.

Proposed solution

Add two annotations: @RealmProjection and @RealmField annotation that makes it possible to map fields from a RealmObject to a standalone in-memory class, only copying the fields defined. This makes it possible for Realm to efficiently move all of these through JNI using only one method call as well as decoupling the UI from the Database.

Projections are not lazy-loaded, they are full in-memory copies.

Examples

public class Person extends RealmObject {
    @PrimaryKey public String id;
	public String name;
	public int age;
	public String socialSecurityNumber;
	public RealmList<Person> children;
	public RealmList<Pet> pets;
}


public class Pet extends RealmObject {
        @PrimaryKey public String id;
	public String name;
}


// Simple Projection Class

@RealmProjection(Person.class)
public class MyViewModel {
	public String name;
	public int age;
}

List<MyViewModel> persons = realm.where(Person.class).findAll().asProjection(MyViewModel.class);


// Only return a single primitive type

List<String> names = realm.where(Person.class).findAll().asProjection(String.class, "name");


// Simple magic syntax for selecting child objects.

@RealmProjection(Person.class)
public class MyAdvancedViewModel {

	@RealmField("name") // use different variable name than stored in Realm
	public String fullName;

	@RealmField("children[0].name") // Magic syntax for selecting objects in the object graph
	public String firstChildName;
}


// We quickly end up where full querying support are needed in the annotation
// Question: Could this be described using @LinkingObjects and other concepts?

@RealmProjection(Person.class)
public class MyOtherAdvancedViewModel {

	@RealmField("name") // use different variable name than stored in Realm
	public String fullName;

	@RealmField("children.name = 'John' AND max(children.age)")
	public int oldestChildNamedJohn;
}

API concerens that needs to be addressed

• More advanced use cases needs to be described, especially queries with dynamic input cannot be expressed in an annotation.

• How should the API look for doing the projection class conversion on a background thread?

Disadvantages

• Unknown syntax for specifying properties in the object hierachy, especially advanced cases might be difficult

• The tradeofs using a ViewModel approach is different from using a normal Realm, especially if they are copied into memory.

• Realm annotations still creep into the UI layer.

• No changelisteners on projections

• It becomes more difficult to update the Realm data.

• Maintaining the Projection class might be difficult as the entity changes.

[Zhuinden]

Decoupling from Cursor makes it hard to do "infinite scrolling"

Not true though with its integration with Pagination library using LivePagedListProvider

[cmelchior]

Depends on the definition of "hard" and "easy" :smile:, but yes, "hard" might be too strong a word because the paging library do massively improve the situation. Example is here: https://developer.android.com/reference/android/arch/paging/PagedListAdapter.html

[MrNovado]

I don't think this approach is the right way to go because:

• You don't necessarily want to create a VM on top of your existing data-model
• You wont be able to make dynamic projections this way
• You will drown in magic syntax which is static, not debuggable, and must be defined in the annotation descriptor which wouldn't scale/work for big queries
• You will not solve anything (?) for the recursively nested objects

This is not an easy case of course, yet I think the better approach would be to go the graphQL way of doing queries.

Suppose you have a model:

open class Module : RealmObject() {
    var id: String
    var data: Data
    var children: RealmList<Module>? = null
}

Where's data is also a huge object with its own dependencies (say, with the depth of 6). Your overall hierarchy, starting from a root-module to the very bottom level modules, consists of (N), say, 200 levels.

• Now how would you make a projection (a copy) of this data that only holds a 3rd level module and its immediate (4th lvl subset) children?

Looking at the examples you provided I have no idea how to do that... Without having a some kind of way to define local depth restrictions on fields it seems impossible.

Rather than force queries' job on annotations, how about improving on your actual queries and cursor? What if we would be able to do a query like this:

realm.where(Module::class.java)
    .equalTo('id', moduleId)
    .toProjection('
        data: {
             id
             type
             ... // other data-object fields
        }
        children: [
             // realm does not support (polymorphism)
             // heterogeneous lists yet, but maybe it will at some point
             ... on Module {
                  data: {
                     id
                     type
                     ... 
                  } 
                  // notice we are not requesting childrens here
             }
        ]
     ')

So with this query (toProjection) we are requesting a Module with an id moduleId and we are telling the cursor to copy only those fields that are present in a toProjection query and only as deep as the query defined.

The fields that are not defined in a query should be returned with a null value.

To ease the pain of merging (updating) these objects back to the DB you could generate a marker (is<Field>Requested) for every field that will tell the cursor that if the fields are not requested, then they will simply not be updated.

To ease this even more for yourselves, rather than entierly generating a marker you could force a convention for users to define a map:

open class Module : RealmObject() {
    var id: String
    var data: Data
    var children: RealmList<Module>? = null
    
    // a map that will hold markers { <field>: true/false, ... }
    // that will state whether or not a field was requested
    // with toProjection query
    @RealmProjectionFields
    var isRequestedForProjection: RealmMap<String, Boolean>? = null
}

With the solution like this you will be able to generate any projection you like on the same RealmObjects (data-models). And whats even better - you will be able to potentially merge them back in the DB without breaking anything.

[MrNovado]

Well, if you actually want to map to a VM, maybe you should go like this:

@RealmProjection
open class VM {
    @RealmField
    var data: Data
    @RealmField
    var children: RealmList<Data>? = null
    @RealmField
    var someVmValue: String? = null
    // do you really need these annotations ?
}

realm.where(Module::class.java)
    .equalTo('id', moduleId)
    .toProjection(VM::class.java, '
        data@<vm-field-name-for-data>: {
             id
             type
             value@someVmValue
        }
        children: [
             {
                  data@<vm-field-name-for-children-data>: {
                     id
                     type
                     value
                  } 
             }
        ]
     ')

A little magic'ky but will scale better. Hard to tell what to do with these @RealmField("children.name = 'John' AND max(children.age)") kind of restrictions though. A hybrid solution?

And you wont be able to merge these back probably. Or it will be very tricky and will require of you to save PKs to your VM projection (with the RealmObject's projection PK obviously should not be affected by the query):

realm.copyToRealmFromProjection(Module::class.java, VM::class.java, VM, '
        id@<vm-field-for-id>
        data@<vm-field-name-for-data>: {
             id
             type
             value
        }
        children: [
             {
                  data@<vm-field-name-for-children-data>: {
                     id
                     value
                  } 
             }
        ]
     ')

[cmelchior]

@MrNovado Interesting ideas, and thank you for the feedback. The one major problem I have with re-using the model classes for projections is that a bunch of the fields in the class is then unused and it is up to consumers to know how the class was constructed. This really breaks any kind of layered architecture.

So having an individual class for each projection is still a lot better for a type-safe perspective, but there is definitely a non-trivial mapping problem that needs to be solved.

[MrNovado]

@cmelchior

The one major problem I have with re-using the model classes for projections is that a bunch of the fields in the class is then unused and it is up to consumers to know how the class was constructed. This really breaks any kind of layered architecture.

Well, I kind of agree with the first part, but I disagree with the last one. The thing is, the consumer always has a strong opinion on how to use the data its requesting, right? The consumer will always use only whatever parts of the data-model that they need, and will not use the other parts of the data at all.

The VM' job for the most part is to solidify these needs in a particular structure - most of the time it's a quality of life thing and not something you must! provide.

So if the consumer already knows on how will he use and what parts of the data, - why wouldn't you allow them to specify their needs into a query?

Mind you, I'am not against the idea of mapping projections to VM at all, I'm just pointing out that projecting on the data-class itself could be just fine (and probably much easier as well).

And from the type-safe perspective all of the data-class fields in realm are inherently nullable anyway (or at least they can be), so... why not?

[Zhuinden]

Hmmm...

Realm is how it tend to creep into all layers of the architecture.

That's because of the thread-local Realm instance that you need to pass to methods because it is thread-local reference counted.

People often don't end up creating their own thread-local to access a single open Realm without incrementing the ref count, so any query/write will need a Realm method argument, from the outside, so that it belongs to the right thread.

Then you must keep it alive to close it at the right time, when it is no longer needed. So scope management is up to the user.

---

I managed to bind Realm and Pagination together on a background looper thread, so technically I could get notifications for projections for paged lists using arch comps.

See https://github.com/realm/realm-java/issues/5486

[MkazemAkhgary]

I used an in-memory realm instance to store projections in RealmObject like this.

public interface RealmObjectProjection extends BaseModel64 {
    RealmList<RealmObject> getObjects();
}

How ever in order to get managed RealmList from a custom array projection, you need a simple container.

public class ItemProjection extends RealmObject implements RealmObjectProjection {
    public ItemProjection() {
    }

    public ItemProjection(RealmList<Item> items) {
        this.items = items;
    }

    @PrimaryKey
    private long id;

    private RealmList<Item> items;

    // id getter and setter

    public RealmList<Item> getItems() {
        return items;
    }

    @Override
    public RealmList<RealmObject> getObjects() {
        return (RealmList) items;
    }
}

after copying this simple object into the in-memory realm (*with unique id), you will get a managed realm list. then you can attach/detach listeners to it. you can easily chain listeners from source to the projected view.

source => listeners => projection => listeners

Here if source (RealmList/RealmResults) is changed, it will notify it's listeners. those listeners will then modify the projection list which will then cause it to notify the projection listeners.

you have to be aware of memory leaks tho, you must detach listeners as soon as objects are no longer needed. also you must make sure that in-memory realm is valid while the projection is being used.

I used this in a RecyclerView Adapter and its working so far. onAttachedToRecyclerView and onDetachedFromRecyclerView must be used to attach/detach listeners. just like in RealmRecyclerViewAdapter.

[Zhuinden]

Always test against process death with in-memory things