http-extensions
http-extensions copied to clipboard
httpwg
Resumable upload: client-generated tokens vs server-generated tokens
We have 2 reasons for using client-generated tokens in the initial draft: (1) we want the minimum number of round trips, and (2) we don't want to depend on 1xx status code for the core functionalities, since it's not available in all situations.
It was brought up that server-generated tokens can provide the ability for the server to encode extra information, e.g. how to route the request. Let's discuss pros and cons of each and what we can do to minimize the drawbacks.
If we switch to server-generated tokens, we might need to standardize 2 ways to start an upload, both through 1xx status code for automatic upgrades, and an empty request to retrieve the token in the case where 1xx cannot be depended upon.
My impression is that there is little value in saving the round trip.
Using a resource identity is the best way to integrate with HTTP semantics, and a client should not be determining the identity of resources. A URI template might help manage those concerns, but it still denies the server a lot of flexibility.
A client isn't prevented from commencing the upload immediately, so the value of reducing latency for the client only applies to subsequent uploads. And this is for large resources, so this is likely a long-lived action anyway. One round trip to learn of a resource identity isn't going to affect the client that much.
Though it might seem like latency is improved if the client can set an identifier, because the client can start parallel uploads of other parts of the resource, I don't think that is really that useful.
The only conditions under which a client might benefit from reduced latency seem to be:
- ... where it has multiple connections open to the same resource, but when those share a different network path. There is no value in reusing the same connection for a parallel upload. The available capacity on that connection is better spent on the content of the request. The same applies to parallel connections to the server that might share a network path (if the concern is congestion control limits, then you are probably looking for a more aggressive congestion control algorithm). Of course, that narrow applicability erodes when you consider the option of using multipath TCP or QUIC.
- ... where the important parts of the resource that it is uploading are not contiguous, but they are highly latency sensitive.
For this latter case, I think that the challenges for the server in terms of managing the coordination of parallel ingest streams - challenges that a client-chosen design imposes on ALL implementations - are probably justification for a non-standard approach, either in terms of content-type design or server resource architecture. I would prefer to keep that sort of thing out of scope for this work.
We hope to get to a place where we can attempt resumable upload on all uploads. Having an extra header isn't much overhead so it's fine to try regardless of server support, but we really want to avoid Safari needing to send extra requests during uploads.
@royfielding suggested using 1xx response to carry a temporary URI to resume the upload. I think that will work and should satisfy all our concerns, but that also means we need to find an alternative solution for clients that cannot rely on 1xx.
We have 2 reasons for using client-generated tokens in the initial draft
There is actually a third reason for using client-generated tokens: It allows clients to resume uploads even if they didn't receive the response for the initial upload creation request.
This is actually a problem we have with the current iteration of our tus protocol (https://tus.io). If the response for the initial upload creation request gets lost, the client has no upload identifier (or upload URL) to contact the server and continue the upload. Basically the client has to options is such a case:
- Error out and abort the upload.
- Create a new upload. But this is problematic because upload creation is not idempotent and should not be retried in general without knowing that the server supports this without problems. From my experience, many upload servers identify an upload resource with other resources (e.g. users). Having multiple uploads identified with one resource can be problematic in these situations.
In my opinion, resumable uploads should be able to resume from all kinds of connection interruptions, including if any response to any request gets lost. As such, I see client-generated tokens not only as performance-improvement but actually as a useful tool to ensure resumability in all cases.
Or do you think the question of idempotent upload creation requests should be solved in different ways? For example, some other applications have a dedicated Idempotency-Key (or equivalent) header to make POST request idempotent: https://www.ietf.org/id/draft-ietf-httpapi-idempotency-key-header-01.html
How likely is it that you have a server that supports resumable upload, the 1xx doesn't make it back to the client, and a meaningful amount of data is successfully uploaded?
I would like to add one more case for client side generated tokens: observability. From operational experience (we’ve been running the spiritual ancestor of this protocol in production at scale for many years), if a request travels through multiple loadbalancers/proxies and one is acting up (due to load, connectivity issues, scaling, subtle misconfiguration), so that the final destination (let’s say, tusd, in our current setup) is not reached, it is harder to debug as there is no unique identifier handed out by tusd yet, and ops are in the dark with problems reported by the clientside. If instead the client generates it, it can log that or return it in an error, and ops have a unique identifier to search logs for, they can now trace the request going through all intermediate layer 7 components that are involved until tusd is reached (or not). And fix the underlying issue quicker. This ultimately amounts to more robust platforms and uploading experiences.
As to how likely is it that a response is lost, in our case a very low percentage but at scale that may still amount to thousands of cases a day, real users reporting real problems even if they only make up 0.005% of total traffic.
@martinthomson How likely is it that you have a server that supports resumable upload, the 1xx doesn't make it back to the client, and a meaningful amount of data is successfully uploaded?
For my argument is not relevant if any amount of data is transferred because simply the act of starting a new upload (the upload creation procedure) cannot always be safely retried in practice (unless we require it to be, but I have my doubts with this as well).
In our production environment using, we regularly see situation where the 2XX response of the initial upload creation request did not make it to the client. I don't have concrete numbers but I image that a 1XX response would also not be received in such cases.
if any amount of data is transferred because simply the act of starting a new upload [...] cannot always be safely retried
Attempting to solve for both this robustness goal and the latency goal at the same time is - at least in my opinion - a mistake. You might make some progress on those, but at somewhat greater expense on the server side.
For me, robustness in the face of disconnection is better provided by pre-flighting the non-idempotent aspects of the transaction, disconnecting them from the expensive and repeatable parts. Yes, this adds a round trip, but we're still talking about a transaction that is large enough to justify the additional infrastructure of all this retry/continuation machinery.
For me, robustness in the face of disconnection is better provided by pre-flighting the non-idempotent aspects of the transaction, disconnecting them from the expensive and repeatable parts
That is an understandable point.
we're still talking about a transaction that is large enough
I think that leads to another question: Do we want to design these resumable uploads only for large objects, or also smaller ones? In my experience, there is interest from the side of service providers to have one HTTP API for uploads of different kinds and sizes. Whether it be big videos or small images. Having one endpoint for both types makes development and maintenance a lot easier. So it would be helpful if we can make resumable uploads also efficient for smaller files. Or do you disagree with that thought?
The scale of impact for any upload interruption is related to, at least, how much exisiting transfer has occured for any file and how fast their upload speed is.
This raises a point I hadn't considered before. In an era of multiplexed HTTP, we can support concurrent uploads. If a connection problem is going happen, how much impact does a user feel if they are uploading 100 x 100 KB objects, or 10 x 1 MB objects, or 1 x 10 MB objects? Some of this will be down to whether an implementation choses to upload in parallel or serial. I think we need some WG input on that. Is it equivalent to lose 1% of 10x100 KB vs. 1% of 1x10 MB?
https://github.com/httpwg/http-extensions/pull/2292 contains a proposal for server-generated upload URLs, using which we might be able to achieve all of our goals.
This raises a point I hadn't considered before. In an era of multiplexed HTTP, we can support concurrent uploads. If a connection problem is going happen, how much impact does a user feel if they are uploading 100 x 100 KB objects, or 10 x 1 MB objects, or 1 x 10 MB objects? Some of this will be down to whether an implementation choses to upload in parallel or serial. I think we need some WG input on that. Is it equivalent to lose 1% of 10x100 KB vs. 1% of 1x10 MB?
Is it equivalent to lose 1% of 100x100 KB vs. 1% of 1x10 MB?
I think that depends a lot on the further usage of these files. If you have 100 x 100 KB images and loose 1 file of them, you can process the other 99 images while the 100th image is being uploaded again. This, of course, assumes that each image can be processed individually, which depends on the application.
On the other hand, if you loose 1% of a 10M upload, the entire file cannot be processed and you have to wait until the entire upload is resumed and completed.
From that point of view (there are others, of course), the second scenario seems worse.
From that point of view (there are others, of course), the second scenario seems worse.
It's also interesting to compare chunked vs. whole data, regardless of content type.
First scenario: lose 1% of 100x100KB Second scenario: lose 1% of 1x10MB I think we can see this from 4 point of views, based on resumable vs. non-resumable and serial vs. parallel uploading. This is also based on the assumption that our goal is to successfully upload ALL bytes.
No resumable protocol, serial upload: 99 of the 100 successfully upload, we keep track of the last one that failed, and retry that upload. If round trip time is negligible, this first scenario is definitely better than restarting the whole 10MB in the second. However, if RTT is non-negligible, then we've built up a lot of extra time ensuring that the uploads are serial (need to wait for 2xx before starting next upload). Now I'm not sure how well that would fair compared to the second scenario, might be pretty equal.
No resumable protocol, parallel upload: The first scenario is very likely better than the second (on average). There's the horrible worst case scenario that you perfectly lose the last bytes of all 100 objects, at which point the second is better (only have to retry 1 request instead of 100). I'm guessing the real expected number of objects that fail is significantly lower, so retrying those few uploads is better than entirely starting over the 10MB.
Resumable protocol, serial upload:
You have the extra round trip to receive the Upload-Offset in the second scenario, but again you have plenty of RTT add-ons to make the uploads serial in the first scenario. I think the second scenario is better.
Resumable protocol, parallel upload: Parallel uploads don't seem to benefit from a resumable upload protocol unless they're quite large (RTT and resume must be faster than restarting entirely). In any case, the second scenario is at least as fast as the first, requiring less requests to do so, so second is better.
There's probably optimizations to be made in the chunk sizes, but overall it seems to me that 1) parallel uploads only benefit in a non-resumable world and 2) uploading a single chunk is the better method for a resumable world.
There's probably optimizations to be made in the chunk sizes, but overall it seems to me that 1) parallel uploads only benefit in a non-resumable world and 2) uploading a single chunk is the better method for a resumable world.
@jrflat is this a thought experiment or do you have data to support those conclusions? I would like to see some data if you have it. I get the impression that this has not been modelled over HTTP/2 or HTTP/3 connections. To wit, take HTTP/2 over TCP. If am uploading 100 streams in parallel and the TCP session disconnects, then I can reconnect and query the state of all 100 resources in parallel, getting a response in 1-RTT (plus any processing time), rather than serializing and waiting at least 1-RTT before querying the next resource. With a large RTT, there should be little difference in time for querying the state of 100 resources upon reconnect compared to querying the state of 1 resource upon reconnect. However, when uploading 100 resources, and potentially also leveraging HTTP/2 PRIORITY_UPDATE to manage stream priority, some resources could finish uploading and be processed by the backend instead of waiting for the entire upload of all objects to complete.
Chunk size might be tuned by the client. Client could choose (and dynamically adjust) chunk size based on RTT and on disconnection frequency in order that progress continues to be made. Are there any models or data for that with resumable uploads?
No resumable protocol, serial upload: 99 of the 100 successfully upload, we keep track of the last one that failed, and retry that upload. If round trip time is negligible, this first scenario is definitely better than restarting the whole 10MB in the second. However, if RTT is non-negligible, then we've built up a lot of extra time ensuring that the uploads are serial (need to wait for 2xx before starting next upload). Now I'm not sure how well that would fair compared to the second scenario, might be pretty equal.
@jrflat I do not understand what you mean with "need to wait for 2xx before starting next upload". HTTP/1.1 pipelining has been around for over 20 years.
Sorry, I should have marked it as a late-night thought experiment, taking HTTP/1.1 without pipelining as a base (which is simple, but as you're suggesting, flawed for the real world). I agree that more thought should go into this for HTTP/2 and 3 connections, and supporting data is definitely needed. You're also right about the HTTP/2 example, which now I think shows that parallel uploads can benefit from the same (general) resumable upload protocol as serial uploads (voiding my first conclusion in that case).
With that, I'm starting to think that this resumable protocol in its current form approximately equalizes the impact of disconnecting a 100x100KB parallel upload vs a single 1x10MB upload. I might be missing some corner cases. Overall, though, I'm definitely in favor of having a more general protocol that works for either case (at least in the scope of the draft).
It might have been a mistake for me to bring up the idea of optimizing chunk sizes since that's likely out-of-scope for the general resumable upload protocol. The client should have the freedom to do that on top of the protocol. But it is interesting, and while I'm unaware of such data/modeling, I'd be interested in seeing it too if anyone has worked on it.
Based on feedback from the IETF 115 we have for now settled on server-generated upload URLs. This decision is not entirely settled in stone and we might add client-generated tokens as an optional alternative in the future. So please provide any feedback if you have any.