Messages icon indicating copy to clipboard operation
Messages copied to clipboard

Use streams to encode and decode the JSON backups

Open tom93 opened this issue 1 year ago • 1 comments

What is it?

  • [x] Bugfix
  • [ ] Feature
  • [ ] Codebase improvement

Description of the changes in your PR

  • Use streams to encode and decode the JSON backups instead of storing the file contents in memory. This significantly reduces memory usage when importing/exporting backups.

  • Report OutOfMemoryError when importing and exporting messages. (Less likely to happen after this PR, but still possible.)

  • Delete exported file on error. (Fixes an existing issue that is somewhat related this PR.) Implementation notes: I initially tried to delete using contentResolver.delete(uri, null, null), but that failed with "UnsupportedOperationException: Delete not supported" so I switched to DocumentsContract.deleteDocument(). But when using the Fossify File Manager (FFM) as the file picker, ~~deleting doesn't seem to do anything in both cases.~~ Correction: With FFM, contentResolver.delete() works but DocumentsContract.deleteDocument() doesn't do anything. I think it's okay to just use DocumentsContract.deleteDocument(), so that it works well with the native file picker, and accept that if using FFM then the files won't be cleaned up on error. I don't think it's a big deal because I can't even choose FFM as the file picker on Android 14, and because FFM currently allows the user to pick an existing filename so it's probably not a good idea to delete when FFM is used. Also, it's possible that DocumentsContract.deleteDocument() not doing anything is a bug in FFM; if that's the case then that bug should be fixed instead of adding a workaround to Messages.

Before/After Screenshots/Screen Record

N/A

Fixes the following issue(s)

  • Fixes #6
  • Fixes #169

Other apps

There are several other apps that have a similar JSON import/export feature. Should I port this fix to them?

Future work

(Copied from https://github.com/FossifyOrg/Messages/issues/6#issuecomment-2225743345)

The version in this PR still loads all the messages into RAM, so it can still run out of memory if the backup is too large.

It's possible to reduce the memory usage to virtually nothing, but it would take more work. The idea is to read the messages one at a time or in chunks, and then encode them and write them to the backup individually. It's not that bad -- we just need to manually write a "[" at the start, then write the messages (encoded individually and manually separated by ","), then write a "]" at the end.

Doing something equivalent for import should also be possible, using decodeToSequence.

@gusr has pointed out that the SMS Import / Export app (which is also GPL-3 and Kotlin) works better and suggested using their code. Personally I'm not sure if that's practical, but they have some good ideas that we could learn from. The core import/export code is here (snapshot).

Acknowledgement

tom93 avatar Mar 18 '24 17:03 tom93

I can confirm this fixes the issue. Tested exporting on a Pixel 5 and importing on a Pixel 4 and all MMS and SMS were available. Generated archive was ~120MB. It's slow but it works. Please merge.

knuxyl avatar Apr 19 '24 10:04 knuxyl

There are several other apps that have a similar JSON import/export feature. Should I port this fix to them?

If you are still around, sure :)

Thanks!

naveensingh avatar Dec 25 '24 13:12 naveensingh