RFC: Add support for resumable uploads

[cwalo]

What kind of change does this PR introduce?

This PR introduces a ResumableUploadApi, integrating TUSKit, for resumable upload support. So far this adds basic functionality for uploading individual files and data blobs, canceling/resuming uploads, observing upload status, and configuring the client with a background session configuration.

At the moment I'm looking for feedback on the api and any considerations I might be missing.

Design Details:

• SupabaseStorageClient is initialized with a ResumableClientStore, creating an instance of ResumableUploadApi
• ResumableUploadApi is the public interface for interacting with the client and uploading files.
  • Supports uploadFile, uploadData, pause, resume, retry, cancel, pauseAll, cancelAll, resumeAll, getUploadStatus, getUpload
• ResumableClientStore is an actor responsible for lazily creating and storing instances of ResumableUploadClient, keyed by bucketId - one instance per bucket.
• ResumableUploadClient wraps TUSClient and tracks active/finished uploads
• ResumableUpload is a reference type that can be used to observe the status of an upload, pause, and resume
  • The status can be observed at any time via upload.status(), creating an AsyncStream<ResumableUpload.Status>. The stream will replay the last emitted status and a stream can be created an any time, as long as you have a handle to the upload. Multiple streams can be created.
  • The upload maintains a weak reference to the client, allowing consumers to pause and resume the upload directly.

New dependencies:

• TUSKit

Considerations:

• By having one client per instance, each bucket maintains has its own storage directory. This adds some complexity, but is ultimately more flexible. For instance, if we wish to cancel and remove all stored uploads for a particular bucket, we can simply use TUSClient.reset without affecting other buckets.
• We might be able to make the ResumableUploadClient initializer non-throwing by initializing the TUSClient lazily, but I don't have a strong opinion yet.
• TUSKit errors are not very helpful, particularly for 400s (e.g. diagnosing RLS issues). I have an open PR to address this.

TODOs:

• Wrap TUSClientErrors(?)
• Add ability to observe uploads via a single stream
• Handle duplicate uploads - remove or resume existing
• Support multiple file uploads in one call
• Method to get all stored upload ids
• Current tests are written against a local supabase instance. Should be updated to mock requests.
• Review sendability and concurrency use
• Add sample app and test
• Can we automatically configure the session configuration by inspecting the background mode entitlement?
• Handle token expiration between retries

Basic upload:

let api = supabase.storage.from("bucket").resumable
// optionally pass FileOptions
let upload = api.upload(file: someLargeFileURL, path: "large-file.mp3", options: FileOptions(upsert: true))
for await status in upload.status() {
    print(status)
}

Resume stored (not failed) uploads on launch:

let api = supabase.storage.from("bucket").resumable
try await api.resumeAllUploads()

Pause and resume an upload:

let api = supabase.storage.from("bucket").resumable
let upload = api.upload(file: someLargeFileURL, path: "large-file.mp3")
...
upload.pause()
...
upload.resume()

More examples in Tests/StorageTests/ResumableTests

What is the current behavior?

#171

[grdsdev]

Hey @cwalo thanks for this proposal, I'll check it soon and post my thoughts on this here.

Thanks again.

[grdsdev]

Hey @cwalo, thanks a lot for your excellent work on this proposal.

Here are a few things to consider:

If we decide to pursue this path, which I’m still uncertain about, I’d prefer not to rely directly on the TUSKit library. Instead, Supabase should expose a protocol that the TUSKit library implements in a separate library. Then, when someone wants resumable upload support, they would need to add this third dependency (which acts as a bridge between Supabase and TUSKit).

Another option is that iOS 17 already supports the TUS protocol by default. We could implement TUS using the native Apple implementation and make it available only for iOS 17+. This is the approach I’m more inclined to take.

Let me know your thoughts regarding these points.

[cwalo]

Thanks for taking a look @grdsdev!

Instead, Supabase should expose a protocol that the TUSKit library implements in a separate library. Then, when someone wants resumable upload support, they would need to add this third dependency (which acts as a bridge between Supabase and TUSKit).

I don't think that would be too hard - just need to think through where we'd hook in. Are there any examples of this approach for other "add-ons?"

Another option is that iOS 17 already supports the TUS protocol by default. We could implement TUS using the native Apple implementation and make it available only for iOS 17+. This is the approach I’m more inclined to take.

Based on my research, the IETF standard is similar to, but distinct from, the TUS v1 protocol. Unless the supabase backend supported both, I don't believe the Foundation implementation would work.

[grdsdev]

Based on my research, the IETF standard is similar to, but distinct from, the TUS v1 protocol. Unless the supabase backend supported both, I don't believe the Foundation implementation would work.

I didn’t know that. From a quick glance, I thought they were the same. I’ll check internally with the storage team to see if Apple’s implementation can be easily supported.

I don't think that would be too hard - just need to think through where we'd hook in. Are there any examples of this approach for other "add-ons?"

No examples are provided, so this would be the first “add-on.” We could integrate it into the configuration for Supabase client. Additionally, we could introduce a new option for a ResumableUploadProtocol, which a third-party library would import TUSKit and provide an implementation of that protocol.

Another option worth considering is to implement something similar to the Kotlin library. It implements the resumable protocol without relying on any external dependencies. You can check it out at https://github.com/supabase-community/supabase-kt/tree/master/Storage/src/commonMain/kotlin/io/github/jan/supabase/storage/resumable.

We could use this implementation to offer a first-party resumable upload implementation as well.

[cwalo]

Additionally, we could introduce a new option for a ResumableUploadProtocol, which a third-party library would import TUSKit and provide an implementation of that protocol.

I was just poking at something like that. Let me play with creating a separate package and hooking into the StorageClientConfiguration.

Another option worth considering is to implement something similar to the Kotlin library. It implements the resumable protocol without relying on any external dependencies.

I looked at that first for inspo and actually tried to mirror the user-facing api - storage.from(bucket).resumable... :) I was also curious to know how difficult it would be to add the kotlin implementation as a dependency, but I haven't used kotlin-multiplatform and not sure how much of the SDK it would need to pull in to work. Lastly, I couldn't tell if it cached uploads to disk to more easily resume them between launches.

[cwalo]

I think this might do the trick

Define a protocol with associated types:

public protocol ResumableUploadProtocol: Sendable {
  associatedtype UploadType
  associatedtype UploadStatus

  init(bucketId: String, url: URL, headers: [String: String])

  func upload(file: URL, to path: String, options: FileOptions) async throws -> UploadType
  func upload(data: Data, to path: String, pathExtension: String?, options: FileOptions) async throws -> UploadType
  func pauseUpload(id: UUID) async throws
  func pauseAllUploads() async throws
  func resumeUpload(id: UUID) async throws -> Bool
  func retryUpload(id: UUID) async throws -> Bool
  func resumeAllUploads() async throws
  func cancelUpload(id: UUID) async throws
  func cancelAllUploads() async throws
  func getUploadStatus(id: UUID) async throws -> UploadStatus?
  func getUpload(id: UUID) async throws -> UploadType?
}

Provide the configuration with a closure that returns an instance:

public struct StorageClientConfiguration Sendable {
  public var url: URL
  public var headers: [String: String]
  public let resumable:  (@Sendable (_ bucketId: String, _ url: URL, _ headers: [String: String]) -> any ResumableUploadProtocol)?,
}

To avoid littering the storage apis with generics, we can extend the StorageFileApi with a function getter that casts the return type:

extension StorageFileApi {
  // Calls the provided getter, casting to the requested concrete type
  public func resumable<R: ResumableUploadProtocol>() -> R? {
    configuration.resumable?(bucketId, configuration.url.appendingPathComponent("upload/resumable/"), configuration.headers) as? R
  }
}

let api: MyResumableUploadAPI? = supabase.storage.from("bucket").resumable()

Then consumers can create and store clients however they please.

[cwalo]

With the addition of the protocol, I'll probably still want to package my implementation separately just because it provides a nice concurrency api over TUSKit :)

[grdsdev]

@cwalo I asked Claude to port the Kotlin implementation over to Swift and this was the result https://github.com/supabase/supabase-swift/pull/799

[cwalo]

@grdsdev Nice! I'll check it out.

[grdsdev]

We initiated an internal discussion about providing first-party support for TUS in the libraries. Once we have a minimal design, I’ll share it with the community to gather feedback.

I’ll keep this PR open for reference until then.