capacitor-updater bug: Allow resumable download

Currently, if a network issue happen or the user kill the app, the download is lost and restarted from 0, so implementing resumable download will improve a lot the bandwidth usage and for user with bad connexion

Aug 29 '23 15:08 riderx

/bounty $200

Aug 29 '23 15:08 riderx

~~## 💎 $200 bounty • Capgo~~

~~### Steps to solve:~~ ~~1. Start working: Comment /attempt #255 with your implementation plan~~ ~~2. Submit work: Create a pull request including /claim #255 in the PR body to claim the bounty~~ ~~3. Receive payment: 100% of the bounty is received 2-5 days post-reward. Make sure you are eligible for payouts~~

~~### Additional notes:~~ ~~ℹ️ If something is not clear ask before working on it, otherwise your chance to rework it is high~~ ~~🎥 To claim you need to provide in your PR a demo video of the change~~ ~~👨‍👩‍👧‍👦 Join the Discord to get help~~ ~~📏 Check all Bounty rules~~

~~Thank you for contributing to Cap-go/capacitor-updater!~~

~~Add a bounty • Share on socials~~

Attempt	Started (GMT+0)	Solution
🔴 @neo773	Aug 29, 2023, 3:58:42 PM	WIP
🔴 @ayewo	Sep 20, 2023, 9:32:06 AM	WIP
🔴 @yashk7oo	Oct 19, 2023, 5:04:34 AM	WIP
🔴 @Sambit003	Dec 20, 2023, 1:36:13 PM	WIP
🔴 @Samankhalid01	Feb 6, 2024, 7:23:59 PM	WIP
🔴 @githubarj	Jun 5, 2024, 3:48:20 PM	WIP
🟢 @YanisOUALAN	Jul 2, 2024, 9:12:47 PM	#400
🔴 @yncrahmet	Jul 29, 2024, 10:48:32 AM	WIP

Aug 29 '23 15:08 algora-pbc[bot]

/attempt #255 continuing this from our DM in discord

Options

Cancel my attempt

Aug 29 '23 15:08 neo773

Isn't the partial download (individual file download) already a partial fix for this?

Aug 29 '23 19:08 WcaleNieWolny

@WcaleNieWolny it seems resumable is easier to do than partial.

So I had conversation with @neo773 that easier to do, so I want to add this now and partial when ready

Aug 29 '23 22:08 riderx

@neo773: Reminder that in 7 days the bounty will become up for grabs, so please submit a pull request before then 🙏

Sep 05 '23 15:09 algora-pbc[bot]

The bounty is up for grabs! Everyone is welcome to /attempt #255 🙌

Sep 12 '23 15:09 algora-pbc[bot]

/attempt #255

Options

Cancel my attempt

Sep 20 '23 09:09 ayewo

@ayewo as far as I know @neo773 has already wrote some code for this and he is nearly completed this. Although he said that it is 90% complete 5 days ago

Sep 20 '23 09:09 WcaleNieWolny

I'm sure @neo773 has no trouble speaking for him/herself so there's absolutely no need for you @WcaleNieWolny to speak on their behalf.

Sep 20 '23 09:09 ayewo

And I am sure he will. I am not saying you should not attempt this. Go ahead, just letting you know what I know about the status of this issue

Sep 20 '23 09:09 WcaleNieWolny

@ayewo Feel free to work on it, my attempt wasn't successful @WcaleNieWolny that "90% completion" was actually partial downloads

Sep 21 '23 11:09 neo773

@neo773 gracias 👍!

@riderx I did some preliminary research and I have a few quick questions:

the response headers for requests to the https://api.capgo.app/updates endpoint includes this header: CF-Cache-Status: DYNAMIC which suggests you have Cloudflare caching enabled, correct?

HTTP/1.1 200 OK
Content-Type: application/json
Transfer-Encoding: chunked
Connection: keep-alive
CF-Ray: 7fac66fd7dd30b85-GPS
CF-Cache-Status: DYNAMIC
...
...
Server: cloudflare
Content-Encoding: br

if you have caching enabled, I want to assume you also have caching enabled for updates that are downloaded from the capgo bucket hosted on R2, correct? In other words, downloads from a presigned URL like https://9ee3d7479a3c359681e3fab2c8cb22c0.r2.cloudflarestorage.com/capgo/apps/d94748c9-e609-449e-84d1-30cbfbd4dde5/ee.forgr.demoapp/versions/1.0.0.zip are cached, correct?
if yes to 1.0 and 2.0, then Cloudflare's caching is going to interfere with the ability to resume downloads from the iOS/Android plugin unless caching is turned off for the capgo R2 bucket through cache rules.

For instance, with caching disabled, if a 100MB download is interrupted after 70% has been downloaded, when the download is resumed, the server will send the remaining 30MB. But with caching enabled, the server will always send the full 100MB any time an end user attempts to continue an interrupted download.

Do you still need this feature now that the partial update feature is ready for review?

@WcaleNieWolny it seems resumable is easier to do than partial.

So I had conversation with @neo773 that easier to do, so I want to add this now and partial when ready

I can assure it is not easier 😀. It is closer in complexity to the partial update feature as it will require opening PRs in this repo as well as the Cap-go/capgo repo.

For this repo, the background download logic will have to be refactored so that background downloads can survive network interruptions, end user interruption or termination by the operating system.

For the Cap-go/capgo repo, an additional endpoint will be added to add support for resuming a download from your R2 bucket since you currently use presigned URLs to authorise access.

The payment on this issue completely underestimates the effort: it is currently too low.

Sep 22 '23 08:09 ayewo

@ayewo

For the Cap-go/capgo repo, an additional endpoint will be added to add support for resuming a download from your R2 bucket since you currently use presigned URLs to authorise access.

Pre-signed URLs have content-length header so not sure why you need to an additional API endpoint I did some initial testing with MZDownloadManager and it did work, but couldn't manage to integrate the library since I'm new to Swift.

Sep 22 '23 09:09 neo773

Can't you just add "Range: bytes=0-NN" header from iOS or android and then just get the r2 file?

I think you can download part of a S3 file https://stackoverflow.com/questions/36436057/s3-how-to-do-a-partial-read-seek-without-downloading-the-complete-file

Sep 22 '23 09:09 WcaleNieWolny

To support resumable downloads you need 2 headers. The Content-Length: header is one piece. The other piece is the Range: header.

There are 2 reasons why a new endpoint is required:

Capgo currently creates a presigned URL with a validity of 120s (2 min). If a download interruption lasts for more than 2mins, the presigned URL would have expired, so the plugin will need the Capgo server to generate another presigned URL. There is currently no endpoint that handles the renewal of presigned URLs and this is needed for downloads to be resumable.
In order to support arbitrary range seeking using the Range: header, the presigned URL must first encode the actual Range: for a valid URL to be generated. So, if a client wants to fetch Range: bytes=0-19 the Capgo server must encode that extra bit of information before returning a presigned URL. Here's an example using the AWS S3 library in JS: https://github.com/aws/aws-sdk-js-v3/issues/4823#issuecomment-1587749816.

As for why caching will need to be disabled for downloads from R2, it because we don't want Cloudflare to ignore the Range: header when it is included by the plugin. This Cloudflare forum discusion makes the case for why caching will need to be turned off.

Sep 22 '23 10:09 ayewo

Capgo currently creates a presigned URL with a validity of 120s

That is not true, capgo has a way longer validity due to redis caching the url

This used to be true, but is no longer the case

Sep 22 '23 11:09 WcaleNieWolny

That is not true, capgo has a way longer validity due to redis caching the url

lol 😂🤣!

This was true just 2 weeks ago so not sure what point you are trying to make. In any case your rebuttal doesn't invalidate what I wrote as to why another endpoint is needed.

You could have corrected me by saying Martin has since changed it from 120 to 604800 ... instead you chose to come out guns blazing like someone trying to win an argument (that I didn't start).

Anyway just take it easy as I can't possibly be on top of all the changes you and Martin are making✌️.

Sep 25 '23 10:09 ayewo

I was just trying to inform you about the change to the R2 validity time and I was trying to argue against creating a new endpoint

I am sorry if I did not made it clear enough

Sep 25 '23 10:09 WcaleNieWolny

@ayewo: Reminder that in 7 days the bounty will become up for grabs, so please submit a pull request before then 🙏

Sep 27 '23 09:09 algora-pbc[bot]

@algora-pbc yes, I know but without @riderx's input, I can't really forge ahead. Martin is working on fixing issues with billing (which is of higher priority).

Sep 27 '23 10:09 ayewo

@ayewo Bruh, you’re replying to a bot 💀

Sep 27 '23 10:09 neo773

@neo773 I know, right 😁!

(I also know that the founders of Algora keep an eye on the bot's email notifications too 😉).

Sep 27 '23 10:09 ayewo

Roger that 🫡