Import is very slow

[tom93]

Checklist

• [X] I can reproduce the bug with the latest version given here.
• [X] I made sure that there are no existing issues - open or closed - to which I could contribute my information.
• [X] I made sure that there are no existing discussions - open or closed - to which I could contribute my information.
• [X] I have read the FAQs inside the app (Menu -> About -> FAQs) and my problem isn't listed.
• [X] I have taken the time to fill in all the required details. I understand that the bug report will be dismissed otherwise.
• [X] This issue contains only one bug.
• [X] I have read and understood the contribution guidelines.

Affected app version

1.0.1

Affected Android/Custom ROM version

Android 8.1, Android 14

Affected device model

Moto G5 Plus, emulator

How did you install the app?

Built from source

Steps to reproduce the bug

Import a .json backup with thousands of messages (example with 3000 messages, generated using the code at the end).

Expected behavior

The import should either complete quickly or display proper progress indication (for example as a notification). Related: #120 (request for progress dialogs across all apps).

Actual behavior

There is a toast saying "Importing...", and then it disappears and it seems like nothing is happening. A long time later (possibly several minutes) there is a toast saying "Importing successful". This is unacceptable, because the user has no idea what's going on (whether the import is still running or whether it crashed).

Screenshots/Screen recordings

No response

Attempt to make import faster using bulkInsert()

I tried making the import faster by switching from ContentProvider.insert() to ContentProvider.bulkInsert() as well as some other improvements (branch "import-bulk", commit), however that doesn't help that much. It reduces the time by 36% on Android 8.1 (66 sec => 42 sec) and 60% on Android 14 (72 sec => 26 sec) in an emulator with the attached 3000-message example (tested using this commit, which adds debug toasts and logcat messages to allow calculating the duration from the timestamps). The patched version is super fast on messages that already exist, but that doesn't help during the initial import. I'm not sure if it's worth merging, because it's a little complicated for not-very-big gains and the error handling isn't great (if one message in a chunk fails, then the rest of the chunk is skipped).

I don't think we can make the import any faster than that, because almost all of the time is spent in the bulkInsert() call waiting for the Telephony provider, which is outside of our control. We'd have to file a bug report against Android and hope that Google makes the code faster.

So I guess we have to show a proper progress indicator (e.g. using a notification; or at the bare minimum show multiple toasts).

Prior art

It might be worth comparing to the SMS Import / Export app to check if they have any tricks to speed up the import (based on a quick skim it doesn't appear so, but I think it's still worth doing a quick benchmark).

Testing code

For testing, the following Python code can be used to generate a large backup:

print("[")
for i in range(3000):
    print('%s{"subscriptionId":1,"address":"6505550","body":"Message %d","date":%d,"dateSent":0,"locked":0,"protocol":null,"read":1,"status":-1,"type":%d,"serviceCenter":null,"backupType":"sms"}' %
          ("," if i != 0 else "", i + 1, 1700000000000 + i * 60*1000, i % 2 + 1))
print("]")

Credit

Originally reported by @Abcd1234-dot in https://github.com/FossifyOrg/Messages/issues/88#issuecomment-2002326219.

[4fosaefp]

I just did an import and it took a solid 5 minutes to complete. Some kind of progress indication would be really helpful.

[naveensingh]

There is a progress bar in the latest versions but it's not very useful for tracking progress:

[4fosaefp]

Yeah I did see the progress bar but it just runs indefinitely, and I can't tell if the import is still going or if it hangs, especially after a couple of minute.