App re-downloads all articles on refresh

[deanishe]

Q	A
Device?	iPhone
Device Os version	12.2
App Version	4.0.1 (211)
Wallabag server version	2.3.8

Steps to reproduce/test case

Pull down article list to refresh. The app downloads all the articles from Wallabag. In my case, 200+ pages and 100+ MB.

This uses a crazy amount of data (as well as taking a long time).

In addition, the app appears to immediately start overwriting its local database on sync, so if sync fails (Internet connection drops or you kill the app to stop it downloading 100MB), you can end up with a partial or empty list of articles, depending on when sync was interrupted.

Improvement

The app should only download updated or new articles on refresh, not everything. It shouldn't delete locally-cached items until it has verified they've been removed from Wallabag.

Thanks!

[bourvill]

You are right for downloading all the database at each refresh.

I saw recently in the API the way to recover only the changes since a certain date. This could be a solution.

I will look shortly. But it will not be included in the 4.1.0 maj

[deanishe]

Looking at the API, you can use the since parameter to only fetch updated/new entries, but identifying deleted entries appear trickier.

It looks like you'd have to check all the entry URLs against /api/entries/exists.

[bourvill]

Yes, that exactly the endpoint i want to use.

And /api/entries/exists can be a good way to find deleted entries and can be more smaller for data usage and battery consumption.

I need to finish and polish tag. but this point can be really interesting...

I will try and think if it is possible to integrate it in version 4.1.0, but I do not want to delay tags

[deanishe]

No problem. For the time being, I'll just be careful to only update when I'm on WiFi.

[deanishe]

Any progress on this?

I've noticed a couple of updates to the app, but it's still downloading the entire Wallabag DB on sync and consuming an extraordinary amount of data as a result.

This isn't really a nice-to-have feature so much as a major issue with the app. Downloading every single article on every sync is basically the worst possible way to handle loading the data from Wallabag.

[bourvill]

not yet, I know it's a problem that can be embarrassing for some users.

I have it in mind for rewriting the current application

[thibaultamartin]

@bourvill I think this behavior is fixed in beta. Can you confirm?

@deanishe The Wallabag app is being rewritten. I think the beta handles it correctly but am waiting for @bourvill's confirmation. This means your issue will be fixed when the v5 app is released.

If you want to join the beta, you can follow the Testflight link. Keep in mind the beta still is unstable, and any update can completely break the app (and potentially do some nasty things on your Wallabag instance).

[deanishe]

Thanks for the beta invite, @thibaultamartin. Unfortunately, it's not accepting new users at the moment.

[thibaultamartin]

Hej @deanishe, here’s the actual beta link: https://apple.co/2HRfIcY

[bourvill]

Currently the synchronization system has not changed, it still downloads all the articles. I think we can include work in the next version. Version 5.0.0 will remain in current synchronization mode

[thibaultamartin]

A hint for the fix in the 5.1.x version: the GET /api/entries.{_format} API has two interesting parameters:

• since, which allows to get only changes since a given time
• detail which allows to get only the metadata of the articles, instead of download the articles content along with the metadata (default behaviour is to download metadata and content)

Using those two parameters combined could significantly reduce battery and data usage

[deanishe]

Thanks, @thibaultamartin. I've installed the beta, but it doesn't show any articles.

I can see in the server logs that it fetches the first few pages, but that's only a small fraction of the articles, and the app shows nothing in the GUI.

[thibaultamartin]

Interesting test case, do you have a large collection of articles in your Wallabag @deanishe?

@bourvill any clue regarding that issue?

[deanishe]

What qualifies as "large"? I have ~6500 items.

[thibaultamartin]

The huge article list support cannot be achieved while Wallabag doesn't actually support /api/entries&detail=metadata. This is not supported in current version (2.3.8) but will be supported for the next release.

Here is wallabag's code to find entries in the db as of the release of 2.3.8: https://github.com/wallabag/wallabag/blob/9bbafdaad4c11666c0188f570d167dff5629314d/src/Wallabag/CoreBundle/Repository/EntryRepository.php#L145

And here's wallabag's code to find entries in the db in master https://github.com/wallabag/wallabag/blob/master/src/Wallabag/CoreBundle/Repository/EntryRepository.php#L164

So today even when you add the detail parameter, you get all the articles.

This raises a problem: since current Wallabag release simply ignores detail=metadata, the iOS client won't be able to determine whether the end-user's instance supports it just by polling /api/entries endpoint.

The deprecated /api/version API returns the instance version. This is a dirty client-side hack, but I can't see any other way to do it, since APIs are not versioned.