Consider integration with Paging library once it is stable

[Zhuinden]

See https://medium.com/@Zhuinden/live-paged-lists-architecture-components-and-room-or-realm-268c9299a661

and

https://github.com/Zhuinden/realm-helpers/blob/4b83911b20304355e95870e51a1743bc141a36fd/realm-pagination-example/src/main/java/com/zhuinden/realmpaginationexample/data/dao/TaskDao.java#L25-L28

---

Although I'm sure the API can be nicer (not having to explicitly open/close, and while keep the notification could be moved to a background handler thread, but otherwise use refresh() in the do { while loop on the io() scheduler of the computable live data.

[Zhuinden]

I've talked to @ericmaxwell2003 and he said he'll look into it later.

[aperfilyev]

Paging is 1.0 as of may 8th

[Zhuinden]

Yeah I have to do it here too https://github.com/Zhuinden/realm-monarchy/issues/3

[mhdtouban]

Any plans to support paging?!

[Zhuinden]

Slightly experimental, but Monarchy supports it now.

 DataSource.Factory<Integer, RealmDog> realmDataSourceFactory = monarchy.createDataSourceFactory(realm -> realm.where(RealmDog.class)); 
 dataSourceFactory = realmDataSourceFactory.map(input -> Dog.create(input.getName())); 
 dogs = monarchy.findAllPagedWithChanges(realmDataSourceFactory, 
                                         new LivePagedListBuilder<>(dataSourceFactory, 20)); 

(not having to explicitly open/close, and while keep the notification could be moved to a background handler thread

Well would you look at that, that's what Monarchy does 😄 I'm surprised I eventually sat down and made a library for it.

[cmelchior]

@mhdtouban Out of curiosity, what is the use case for wanting to implement the Paging Library? With Realms lazy-loading architecture you don't actually need it since Realm doesn't have a cursor limit?

[Zhuinden]

It's for people who want to read pages of data from Realm on a background thread and passed to UI thread, instead of having lazy-accessed cursor where the actual read happens on UI thread.

The cool thing is that this mechanism supports projection via DataSource.Factory's map() function.

[mhdtouban]

@cmelchior I want to use paging for many reasons, first we now officially have a way for pagination in Android instead of relying on recycling view scroll listener which has downsides, second paging offers cool features like a placeholder and continuous loading mechanism etc.

[cmelchior]

@Zhuinden I guess that is a valid point. I still haven't seen it being demonstrated as a problem in practice though, so IMO the problem is largely theoretical. But I guess the paging library also provides an abstraction for the underlying data source if you want to swap them later.

@mhdtouban My point was that you don't need those loading mechanisms and placeholders with Realm. Our lazy-loading works very differently than how SQLite does it.

[cmelchior]

Note, It isn't that we don't want to support paging, but I strongly suspect that the majority of cases don't actually need it and people are just using patterns from elsewhere that doesn't really apply to Realm.

[mhdtouban]

Am not quite sure I get you with "don't need those loading mechanisms with realm", let's say I want to fetch data through an API with pagination. How would realm help in lazy loading? I mean I understand that it will help if am hitting the database, but what if am getting the data from API?

[Zhuinden]

Personally I started investigating the background thread read because of this. realm.refresh() is a bit less free than it used to be.

"don't need those loading mechanisms with realm", let's say I want to fetch data through an API with pagination. How would realm help in lazy loading? I mean I understand that it will help if am hitting the database, but what if am getting the data from API?

Technically you get the benefits of BoundaryCallback being called for zero items loaded and for item at end loaded. Previously you had to intercept zero-loaded in the Results change listener, and you had to know you reached the end if RecyclerView could no longer scroll down.

BoundaryCallback does wrap these cases.

[cmelchior]

@mhdtouban Sorry, I was mainly thinking about the use case where you had all data stored locally. I do agree that the paging library nicely provides a lot of utility where you need to fetch the data from the server first.

[ozzioma]

Guys great work!

Can I suggest examples explaining how exactly pagination is not needed with Realm? Sorry but this is not sufficient

Since queries in Realm are lazy, performing this sort of paginating behavior isn’t necessary at all, as Realm will only load objects from the results of the query once they are explicitly accessed.

If for UI-related or other implementation reasons you require a specific subset of objects from a query, it’s as simple as taking the IQueryable object, and reading out only the objects you need.

I mean this problem of pagination with Realm gets most new users stumped, and there's no example! Honestly it's a hard concept telling users to replace well known pagination routines with a loop...

Most of the UI controls and grid/list display routines out there have callbacks that include query, sort parameters...and pagination parameters. A simple example would have clarified this feature of not needing "those loading mechanisms with realm". Would it be too much work to add a few lines of examples to your documentation showing exactly how it's done?

It's clear the use case being referenced in your documentation is not sufficient to cater to the majority of pagination use cases out there.

You do need a way to execute SKIP, TAKE, TOP, LIMIT routines. If Realm already does that for local data, can we see how?

Thanks.

[Zhuinden]

Technically what they mean by not needing pagination is that data is lazy-loaded so that means you can access any item at any index, and that item will only be loaded on access. This lazy loading means that you don't need to "load more" because you can just throw everything into the list regardless of result size, and it'll be fine.

But Realm doesn't actually support "SKIP" at this time.

[ozzioma]

@Zhuinden Thanks for the response!

However, I'm afraid my concerns were not addressed. I do understand lazy loading and proxying virtual objects. That use case does not address the majority of the scenarios out there where pagination parameters come into play. I am talking about scenarios where pagination parameters ARE NEEDED.

For example, say you have this API or databound control that allows you implement a call back to retrieve a window of rows. You are passed these set of parameters int Page =234, int PageSize=30

That is after you have applied very specific sorting and filtering parameters to the data. This is an example from a Xamarin Forms app:

OnLoadMore = async () =>
                {
                    IsBusy = true;
                    if (TotalRows > 0 || DataList.Count > 0)
                    {
                        if (!(DataList.Count < TotalRows))
                        {
                            return null;
                        }
                    }

                    int page = DataList.Count / PageSize;

                    PageSize = PageSize;

                    CurrentPage = PageSize * page;

                    //This is what I should do... LINQ style...
                    var tableData = DbContext.LocationUpdates.OrderBy(r => r.DateUpdatedUtc).Skip(CurrentPage).Take(PageSize);

                    PageCount = tableData.Count;
                    TotalRows = tableData.LongCount();
                    return new InfiniteScrollCollection<LocationUpdate>(tableData);


                //This is what I should not have to do. Is this the Realm way of pagination data?
                  var tableData = DbContext.LocationUpdates.OrderBy(r => r.DateUpdatedUtc).ToList();
                  List<LocationUpdate> rows = new List<LocationUpdate>();

                    //faulty pagination logic I know
                    for (int count = CurrentPage; count < tableData.Count(); count++)
                    {
                        rows.Add(tableData[count]);
                    }

                    PageCount = rows.Count;
                    TotalRows = tableData.LongCount();
                    return new InfiniteScrollCollection<LocationUpdate>(rows);

For emphasis, you cannot pass an List, Map, IQueryable to a control or API that is expecting a specific number of rows. If its an API call, say a REST service response where the app has to sync a subset of rows to the backend server in the background, I'm hoping you're not suggesting just lazy loading a List or IQueryable??

If it's local data and you have to deal with pagination call backs, then Realm simply does not support pagination or so it seems.

In the case of IQueryable, the documentation makes it clear that Skip, Take are not supported. So the option of doing a ToList() just increases the confusion, since the databound control still has to call Take and Skip on something. How do you track just how many rows have been loaded as the user scrolls down or up? It doesn't matter??

You need a way to reason abount the logic of batching rows, it is the most popular use case.

An example would have sufficed I think, you're assuming data pagination is not a valid use case. Even if Realm returns a lazy loaded List, and I absolutely need to paginate it for whatever reason, how do I get it done? Say the app specs requires an auto pull to refresh after the first 2,000 rows? Or in response to a DataGrid call back with filter and page parameters? Looking forward to your response.

Thanks.

[BapNesS]

Hi everyone,

To be consistent and use support libraries, I'm wanting to implement Paging support library because I need to fetch data from the API when needed. LivePagedListBuilder require DataSource and PagedList.BoundaryCallback implementations.

Because previous comments are 5 months old, I just wanted to know if an official RealmDataSource is on a way or not at all ?

I think it would be nice to have one, like ObjectBox do for example.

Thanks.

[deepakkumardk]

Any update on this issue as it's very crucial. I am fetching items from realm asynchronously using livedata (as per given sample), there's no lag in fetching the data, but when I update the recyclerview using notifiydatasetchanged with the fetched data of size 400 items the UI freezes with black screen. Paging would solve this. Is there any workaround of this.

[Zhuinden]

@deepakkumardk if your UI freezes on notifyDataSetChanged, your RecyclerView probably is inside a NestedScrollView or has nested scrolling disabled, and therefore doesn't actually recycle.

Also, Paging 3.x will be so vastly different from Paging 2.x, that there is no point in creating an integration.

[deepakkumardk]

@Zhuinden Yes, I am using the NestedScrollView and to support the pagination and remove the autoscroll to top or in between them in recyclerview (I had to disable nestedscrolling) on updating the recyclerview, same issue#144 as of RealmAndroidAdapters but I am not using this instead I am getting this issue in standard recyclerview. I am just observing the db and loading the data from db only. Is there any workaround to prevent both of this issues?

[Zhuinden]

You need to remove the NestedScrollView and whatever it's currently solving you have to most likely solve with item view types.

[arunkumar9t2]

I just published Compass which provides set of APIs to integrate Realm with Jetpack Paging 3.

The API looks like this:

val pagedPersons = RealmQuery { where<Person>() }.asPagingItems()

Here asPagingItems() will internally maintain an active realm instance, lazily load items in response to paging lib callbacks in a dedicated worker thread and automatically cleans up once Flow collection is stopped.

It also provides a way to read only subset of data into memory with the overload:

val pagedPersonNames = RealmQuery { where<Person>() }.asPagingItems { it.name }

Example with Android ViewModel:

class MyViewModel: ViewModel() {

    val results = RealmQuery { where<Task>() }.asPagingItems().cachedIn(viewModelScope)
}

It can support transforms, caching and seperators from Paging lib as described here. Any feedback appreciated.