QField icon indicating copy to clipboard operation
QField copied to clipboard

Published 20 hours ago • verified publisher icon opengisch

Performance boost when downloading packages

Open suricactus opened this issue 1 year ago • 9 comments

Use md5 checksum instead of sha256 checksum to gain extra performance when downloading packaged files in QField. The sha256 causes N+1 requests to the S3 storage, therefore makes the QFieldCloud response on the packages/<projec-id>/latest endpoint super slow.

This change will cause an extra initial download of the project file for all users when they sync, since we renamed lastProjectFileSha256 to lastProjectFileMd5. IMO it is acceptable, otherwise we have to handle extra code that I would prefer not to maintain.

See https://github.com/opengisch/qfieldcloud/pull/885

suricactus avatar Feb 14 '24 11:02 suricactus

Task linked: QF-3835 Use md5 checksum instead of sha256 checksum to gain extra performance when downloading packaged files in QField

duke-nyuki avatar Feb 14 '24 11:02 duke-nyuki

🎉 Ta-daaa, freshly created APKs are available for d8120cf63257e2a146bb2d9d3897562d56977680: arm64-android

Other architectures

qfield-fairy avatar Feb 14 '24 11:02 qfield-fairy

@suricactus , can we merge this today so we can have it in the dev channel for a couple of days prior to QField 3.2.0 release? Thanks

nirvn avatar Feb 20 '24 02:02 nirvn

Looks good from my side :+1:

Did the following tests:

  • Linux
    • Syncing of a project with more than 2000 attachments
      • QField can now get the large file list which failed before
        • see https://github.com/opengisch/qfieldcloud/pull/885
      • But failed to actually sync on my machine because of:
"Downloading project `cfafd6f6-1a33-4533-97af-16e316769fd6` finished with an error: Project `sha-test`, file `files/dummy_5BE4F7F7-613D-4F0C-8823-B0D8A1B7FA61.jpg`: File system error. Failed to open file for writing on temporary `/tmp/QField.Wklgqi`. System message: Too many open files"`
  • Android
    • Syncing of a project with more than 2000 attachments
      • Works as expected

boardend avatar Feb 21 '24 11:02 boardend

Ready to merge and as soon as we have the 0.25.0 QFC live.

suricactus avatar Feb 21 '24 20:02 suricactus

Ready to merge and as soon as we have the 0.25.0 QFC live.

suricactus avatar Feb 21 '24 20:02 suricactus

@suricactus @nirvn what is the expected timeline for this?

m-kuhn avatar Mar 02 '24 21:03 m-kuhn

I'd love to see this in for 3.2, very much fits with what's already in the changelog :)

nirvn avatar Mar 03 '24 07:03 nirvn

This can be published by Wednesday after a bit more testing. Shall we somewhere note this version is incompatible with older QFC versions? Well, not incompatible, but will make downloading projects/syncing much slower?

suricactus avatar Mar 04 '24 13:03 suricactus