Switching contacts backend to QtContacts-SQLite

[mardy]

The rationale behind this plan was explained a few months ago in a message sent to the -dev mailing list. Please read that first, but in short:

• Our current QtContacts backend is addressbook-service, which in turns uses EDS.
• We don't have a CardDav plugin
• Jolla has a CardDav plugin in Sailfish OS
• Sailfish OS uses QtContacts, but wrote their own backend (qtcontacts-sqlite) which uses SQLite
• That backend has also a plugin for Google contacts, VK and maybe others
• Switching to the qtcontacts-sqlite backend will also give us:
  • automatic contact merging (when the same contacts is fetched from different sources)
  • a well maintained component, we'd share the burden with Jolla
  • some more free RAM (no addressbook-service, no EDS contacts service)

I've been occasionally working on this task; so far I managed to reduce our delta with the upstream QtContacts (I've submitted most of our patches into QtPim, and they were accepted), updated Ubports to use the latest QtPim, and built the qtcontacts-sqlite on Xenial.

However, the component cannot be used as-is, because it does not support the QCollection API which was added to the latest QtPim and which is necessary to get the backend to work. So, my focus now is adding QCollection support to the backend, after which we can try to switch the addressbook to use it.

The work will happen on branches named xenial_-_mercontacts, so it can be tested by adding this repository (but don't do it now, there's nothing really to see in there yet).

List of pull requests and new repos which are part of this feature:

Prerequisites (these do not actually cause the backend to switch):

• https://gitlab.com/ubports/core/packaging/qtbase/-/merge_requests/1 (:heavy_check_mark: merged)
• https://gitlab.com/ubports/core/packaging/qtpim/-/merge_requests/1 (:heavy_check_mark: merged)
• https://gitlab.com/ubports/core/apparmor-easyprof-ubuntu/-/merge_requests/5 (:heavy_check_mark: merged)
• https://gitlab.com/ubports/core/address-book-app/-/merge_requests/201 (:heavy_check_mark: merged)
• https://github.com/ubports/messaging-app/pull/320 (:heavy_check_mark: merged)
• https://github.com/ubports/dialer-app/pull/179 (:heavy_check_mark: merged)
• ABI break fixups (issue 1928, see merge requests linked from there) (:heavy_check_mark: merged)

The new backend (merging these just adds the backend into the repo, but not into the image):

• https://github.com/ubports/qtcontacts-sqlite/tree/xenial_-_mercontacts (new package)
• https://github.com/ubports/buteo-sync-plugins-social (new package)

The real switch (these cause the old Galera aka AddressBookService backend to be replaced by qtcontacts-sqlite):

• https://gitlab.com/ubports/development/core/telephony-service/-/merge_requests/48
• https://github.com/ubports/bluez-packaging/pull/12 (:heavy_check_mark: merged)
• https://gitlab.com/ubports/development/core/packaging/buteo-syncfw/-/merge_requests/4
• https://gitlab.com/ubports/core/address-book-app/-/merge_requests/200

Cleanups

• https://gitlab.com/ubports/development/core/apparmor-easyprof-ubuntu/-/issues/2

[cyberpunkedu]

@mardy How are you and how is this going? Would anyone/anything be able to help you with this?

[Thatoo]

Hello @mardy any news on this subject?

More globally, is there any news on a cardDAV plugin for UT addressbook app?

[mardy]

Hi @Thatoo! I resumed working on this about a week ago. I'm making good progress in the backend, to the point where it's properly loading and storing contacts, but there are still a few issues to fix here and there.

[obacht8]

I´m very glad to read that cardDAV -implementation is making progress!

[mardy]

I wrote up a blog post with some more details about the progress: http://www.mardy.it/blog/2019/06/wip-changing-the-backend-for-contacts-in-ubports.html

[Thatoo]

Thank you for this report and these great news. Thank you so much for your work.

[Thatoo]

Hello @mardy , any news on this issue? How can someone fork and keep on your work? I'm not saying I will as I'm afraid it's too heavy task for my coding ability but I keep hoping this will be available soon. Hopefully before pinephone and pinetab reach to us, users... Thank you,

[Flohack74]

I guess we move this to OTA-23?

[lduboeuf]

Let it land on OTA-23 and early if possible. But i know this is a huge one I've been testing it without major issues. Fallback to RC works too. There is 2 issues i've noticed though:

• You can't search contacts having accents.
• Some duplicate contacts after db migration ( i don't see any diff in db between at first sight,, haven't dig more )

Note that only tested with a non daily driver phone ( no Sim card )

[luksus42]

Does this affect the way, how I sync my contacts using syncevolution?

[mardy]

Does this affect the way, how I sync my contacts using syncevolution?

So far (in OTA-22) we are landing only some preparatory changes, which should not affect the functionality. In the next cycle, though, we are changing the backend, so syncevolution will stop working. With what services are you using syncevolution with?

[luksus42]

Ok, thanks for the info.

With what services are you using syncevolution with?

With a carddav server on my synology diskstation.

[mardy]

With what services are you using syncevolution with?

With a carddav server on my synology diskstation.

With the new backed we will also have a carddav sync plugin; I'm just not sure whether we should enabled it right at the beginning of the next OTA cycle, or if we'll need to do a few other changes to the Addressbook app first.

[luksus42]

No Problem, for me at least. Better to get things working right. But I wonder, how we could communicate that change in a way, that the users who are using the syncevolution script will be well informed about that change.

[Flohack74]

Just FYI of All, this has been merged today ;)

[lduboeuf]

Some side effects: ~~https://github.com/ubports/messaging-app/issues/347 https://github.com/ubports/dialer-app/issues/203~~ already known: https://github.com/ubports/dialer-app/issues/177

Other than that everything seems fine, congrats :).

[luksus42]

So for me, all contacts vanished after this change got merged... I was using a custom address-book (not default), which was synced by syncevolution. I only did the sync manually, so it is not the sync itself, which make the contacts vanish, but maybe the alternative address-book, which is not getting migrated, after the change?!

[Flohack74]

Yeah we see some real issues with that. If there is no proposal to fix them we gonna revert this.

[lduboeuf]

@mardy Note that the last qtcontacts-sqlite build does not include the last commit: ~~https://gitlab.com/ubports/development/core/qtcontacts-sqlite/-/issues/1~~

So i end up with duplicates, and that may explain long lasting lomiri boot animation for users with lots of contacts

[luksus42]

I have the impression that no migration process was started in my case... I got my contacts back, by exporting them with syncevolution into a vcf and importing it with the gui of the contacts app. I will just document it here:

phablet@ubuntu-phablet:~$ syncevolution --print-databases backend=evolution-contacts
evolution-contacts:
   Persönlich (system-address-book)
   lukas_contacts (0ea1a86d-0318-4b7b-9c89-1ed58f913f95)
   lukas_contacts (54bcbe53-e9c8-4b33-ac0d-33aabd59ffe4) <default>

Export each of it into vcf file (maybe have look before, if there are Items in it):

phablet@ubuntu-phablet:~$ syncevolution --export ./mycontacts1.vcf backend=evolution-contacts database=system-address-book
phablet@ubuntu-phablet:~$ syncevolution --export ./mycontacts2.vcf backend=evolution-contacts database=0ea1a86d-0318-4b7b-9c89-1ed58f913f95
phablet@ubuntu-phablet:~$ syncevolution --export ./mycontacts3.vcf backend=evolution-contacts database=54bcbe53-e9c8-4b33-ac0d-33aabd59ffe4

Or just print it, for example:

syncevolution --export - backend=evolution-contacts database=0ea1a86d-0318-4b7b-9c89-1ed58f913f95

Maybe we could make some script out of it, identifying the addressbooks, exporting and importing it, of course not as vcf, but by printing the items somewhere and importing it via commandline?

[lduboeuf]

@luksus42 Just to know, do you have that file: $HOME/.local/share/evolution/addressbook/system/contacts.db and if so if you query your db ( sqlite3 ...contacts.db ) with "SELECT vcard FROM 'folder_id'" do you have something ? And also do you have that one: $HOME/.local/share/evolution/addressbook/system/migrated_to_qtcontacts ?

[luksus42]

do you have that file: $HOME/.local/share/evolution/addressbook/system/contacts.db?

Yes

if you query your db ( sqlite3 ...contacts.db ) with "SELECT vcard FROM 'folder_id'" do you have something ?

No, it is empty, since my contacts are in another addressbook (.local/share/evolution/addressbook/0ea1a86d-0318-4b7b-9c89-1ed58f913f95/ -> my synced "lukas_contacts" from my previous post)

And also do you have that one: $HOME/.local/share/evolution/addressbook/system/migrated_to_qtcontacts?

No.

[lduboeuf]

No, it is empty, since my contacts are in another addressbook (.local/share/evolution/addressbook/0ea1a86d-0318-4b7b-9c89-1ed58f913f95/ -> my synced "lukas_contacts" from my previous post)

ok, so we have the cause issue @mardy ;-)

[stanwood77]

Just to comment on Lionel advice that even when returning back from devel to stable channel, Google sync does not work anymore. (still ok with the the calendar)

I tried to erase the Google Account, recreate it again, but no luck, all contacts are deleted and address book does not sync them.

No error message. Contacts are well enabled in "account", but nothing's happening.

In address book settings => address book by default: Only "personal" is displayed.

[lduboeuf]

@mardy some side effects : ( devel update 2022/09/19): ~ ~~https://gitlab.com/ubports/development/core/address-book-app/-/issues/203~~ ~~https://github.com/ubports/dialer-app/issues/206~~ ~~https://gitlab.com/ubports/development/core/address-book-app/-/issues/204~~

[lduboeuf]

https://gitlab.com/ubports/development/core/address-book-app/-/issues/206 https://gitlab.com/ubports/development/core/address-book-app/-/issues/205

[lduboeuf]

~~https://github.com/ubports/dialer-app/issues/208~~ https://gitlab.com/ubports/development/core/address-book-app/-/issues/207

[josevidalrt]

Hi. I received an update notification today: Channel: rc Device: Krillin Release after today's update: 2022-w39

I have been checking on github for changes, and I have arrived here, if this was not the right place to expose this problem, I will heed your guidance. I see that they are working on changing the way contacts are handled, I have found that the "favorites" work better and the search in contacts is very fast. A great work behind these important advances for the day to day. I have noticed that not all the contacts I had in the agenda have been migrated, so I only have one agenda, but when reviewing the file system, in the evolution folder I have seen the following, which makes me think that the mobile managed two sql files but as an agenda, in the options I only had one and it was the one synchronized with a google account. Now I do not have the contacts of the main agenda.

[Flohack74]

@mardy plz can you give a short answer ;)

[ivoxavier]

The change, in my daily device, went well. No contact's lost. Didn't noticed if favorite contacts was preserved, because I don't use this feature. I was using Google Account.

Device: OnePlus 6T Channel: RC Version:(2022-W39)