httpwg
RESUMABLE: feature request - chunk granularity
The authors received an off-list message from a server implementer. They have an existing resumable upload deployment based on a custom protocol and are working to integrate support for draft-ietf-httpbis-resumable-upload.
One of the features that the custom protocol provides is the notion of a chunk granularity. This particular protocol uses an HTTP response header field to advertise a value in bytes. Clients that upload chunks are expected to use an integer multiple of the chunk granularity value (modulo the last chunk, which can be a fragment of it). For example , if a value of 262144 bytes was advertised, a client could send chunks of 262,144, or 524,288, or 2,621,440 bytes.
Other custom reliable upload protocols depend on out-of-band advertisements of this concept. For example, see these requirements listed on a webpage [1]:
Chunk size must be divisible by 256 KiB (256x1024 bytes). Round your chunk size to the nearest multiple of 256 KiB. Note that the final chunk of an upload that fits within a single chunk is exempt from this requirement.
draft-ietf-httpbis-resumable-upload does not support this notion of chunk granularity. The protocol does support in-band advertisement of limits via the Upload-Limit field but it defines only the min-append-size and max-append-size properties. The chunk granularity falls somewhere between these two - the range of chunk sizes can be between a min and a max, but a set of sizes are allowed.
There's a few ways we might be able to support this draft-ietf-httpbis-resumable-upload. However, the key point is that this is a feature request that we need to first consider if we want to support.
[1] https://developers.cloudflare.com/stream/uploading-videos/resumable-uploads/#requirements
Thanks @LPardue - adding some more detail here based on some internal discussions:
-
Technically we can represent a hacky/limited form of chunk granularity via min-append-size=max-append-size=chunk_granularity*N. However this artifically constraints the client into sending chunks of a fixed size when really any value of N would have been OK.
-
Even if our server uses the workaround described above, some spec complaint implementations might run into trouble with this as min-append-size + max-append-size are SHOULD from a client point of view, not MUST.
-
Clients that include an unaligned chunk in Upload Creation (with Upload-Complete: ?0) are more troublesome. AFAICT the spec doesn't expect clients which include chunks in Upload Creation requests to respect min-append-size/max-append-size at all -- that makes sense given the name, but means these clients can hypothetically get stuck entirely, especially if they (or the server they're talking to) don't support interim HTTP responses. However it seems somewhat unlikely that a "generic" HTTP client would implement the resumable upload spec in this manner, since it would require the client to do feature detection first which is a bit awkward. Instead "Optimistic Upload Creation" seems to be the likely default path for generic HTTP clients (e.g. this appears to be the path NSURLSession has taken on iOS).
-
Despite these points we still think it would be valuable to include a granularity concept in the spec. Ideally it would apply to both Upload Creation (with Upload-Complete: ?0) and Upload Append requests. Based on my reading of the spec I expect it would land as a SHOULD rather than a MUST, and we'd be OK with that. It'd serve as a clear signal to client implementors that this is something that at least some servers care about, so if they are aiming to be widely interoperarable they will want to consider respecting it.
-
Regardless of whether this lands in the spec or not, our tentative plan (to be maximally interoperarable with as many clients as possible) is to accept as much data as possible for Upload Append requests (up to the nearest granularity boundary), and then fail the request with a 4xx error, including an Upload-Offset header. I think this should allow a spec complaint client (that is unaware of min-append-size / max-append-size and/or a hypothetical granularity constraint) to pick up where they left off, ideally via the provided Upload-Offset in the error response, but otherwise via the Offset Retrieval mechanism. 400 is probably the more "correct" code, but it's tempting to (ab)use 409 CONFLICT in this scenario since clients need to deal with this anyway. It might be helpful for the spec to say that servers MAY return Upload-Offset in other error responses so that clients are more likely to take advantage of this to avoid the extra RT for recovery.
Curious for others' thoughts on this!
With HTTP/1.1 pipelining, and HTTP/2 and later streams, requests can be pipelined. There is diminishing returns reducing the number of requests as large append sizes are increased to being even larger request sizes. Would specifying a list of acceptable and precise chunk sizes work for you? 256k, 512k, 1M, 2M How many more would you list?
Agreed there are diminishing returns. I think that would be an acceptable alternative (assuming you mean specifying via a server advertised list as opposed to a hard-coded list in the spec), though I think I personally prefer the ergonomics of leaving it up to the client to send in multiples of a server specified granularity. In practice we have different "dropzones" with granularity ranging from 256KiB to 8MiB (with several steps along the way). We'd likely produce a separate list per "dropzone" and could almost certainly make do with < 5 items in the list.
@danielresnick Are you able to explain or provide more detail about why granular chunk sizes are desirable? I'm aware it can/is done already for custom protocols but it's not clear what the motivation is. (We can speculate but I'd rather here something concrete from current implementers)
I'm not an expert in this area but from what I gather it allows multiple simplifications/optimizations throughout our stack. It helps align writes to our underlying filesystem block size (which supports replication & different encoding schemes). Data is divided into atomic blocks/units for replication, deduplication, caching, garbage collection, etc so having clear guarantees around intermediate write granularity helps simplify metadata management and can allow more efficent data packing.
Thanks!
Do you have any sense if the set of values tends to benefit from being an arithmetic progession (i.e a constant delta between values) or if you have a non-lonear range like 266K, 512K, 1M, 2M, 8M
I kinda like the idea of a list to be explicit and allow flexibility in the sequence. The downside might be that a server provides nonsense values in the set and there'd need to be some additional parsing/validation logic defined to be defensive to that. For example, an unsorted list, a list with duplicates, etc. However, any format for expressing a granularity would need some validation against other hints, so a list might not push that much further
I had a look but wasn't able to find any concrete data on this unfortunately. I tend to think a non-linear range would be more client friendly/flexible though
Thanks for the additional input, @danielresnick. It seems that there are a few cases where upload systems have to be restricted to a certain granularity. Since this isn't the first report on this, I think we should consider these situations better in the draft.
I'm in favor of adding a granularity value to Upload-Limit. On the question of whether the value should be a single integer indicating a number that the append size must be fully divisible by or a list of allowed append sizes, I would side with the single integer, as its handling on the client- and server-side appear simpler to me. However, if other's experience shows that this isn't sufficient to accommodate limits of existing infrastructure, I'm happy to change my mind.
In my interpretation, the server is currently already allowed to drop trailing bytes, thereby making the appended data fit the required granularity. However, this isn't very clear from the document and we could improve the text.
5. Regardless of whether this lands in the spec or not, our tentative plan (to be maximally interoperarable with as many clients as possible) is to accept as much data as possible for Upload Append requests (up to the nearest granularity boundary), and then fail the request with a 4xx error, including an Upload-Offset header. I think this should allow a spec complaint client (that is unaware of min-append-size / max-append-size and/or a hypothetical granularity constraint) to pick up where they left off, ideally via the provided Upload-Offset in the error response, but otherwise via the Offset Retrieval mechanism. 400 is probably the more "correct" code, but it's tempting to (ab)use 409 CONFLICT in this scenario since clients need to deal with this anyway. It might be helpful for the spec to say that servers MAY return Upload-Offset in other error responses so that clients are more likely to take advantage of this to avoid the extra RT for recovery.
A 4xx status code fits the semantics best. However, the draft currently recommends to not retry on generic 4xx response codes: https://httpwg.org/http-extensions/draft-ietf-httpbis-resumable-upload.html#section-4.4.1-6.4.1. Not every client will retry on a 409 Conflict response because the draft had been written assuming that a compliant client would never trigger a 409 response and therefore not need a retry. However, with required granularities, even a compliant client might run into these statuses. With the current text, a 5xx would likely be needed to make it work with compliant clients. That doesn't appear as correctly, so maybe we have to revisit handling of 409 on the client-side.
Great point, I'd missed that the spec suggests not to retry unexpected 4xx responses. So yes I agree that within the current spec a 5xx response is most appropriate to maximize interoperability. I didn't quite understand your last sentence though - mind clarifying? Are you saying you think that this is not ideal and are therefore thinking maybe the draft spec should be refined so as to allow a different / more specific recovery mechanism for clients that are spec complaint (in a minimal sense) and opt not to support an optional granularity constraint?
I didn't quite understand your last sentence though - mind clarifying? Are you saying you think that this is not ideal and are therefore thinking maybe the draft spec should be refined so as to allow a different / more specific recovery mechanism for clients that are spec complaint (in a minimal sense) and opt not to support an optional granularity constraint?
Not entirely. My thinking is that with the current spec, a 5xx status code is needed to trigger the client to fetch the correct offset (which isn't the offset the client expects since representation data was discarded). However, this doesn't seem like a server error which 5xx status codes indicate, in particular if we extend Upload-Limit to explicitly communicate the granularity. A 4xx would be more suitable, but require the spec to be updated so clients react properly. Does that help you?
Thanks for the clarification, @Acconut. That makes perfect sense.
I agree that relying on a 5xx response feels like a necessary but semantically incorrect workaround under the current spec. A 4xx response is definitely more appropriate, and updating the spec to guide clients on how to handle it would be the ideal long-term solution.
Your comment, "Not every client will retry on a 409 Conflict response because the draft had been written assuming that a compliant client would never trigger a 409 response", got me thinking about other scenarios though...
Even with a perfectly compliant client, a 409 Conflict might occur due to factors outside its control. For example, network-level retries (e.g., from a proxy or load balancer) or internal server-side race conditions could result in a chunk being processed more than once. While the HTTP spec (I think?) discourages uninformed retries of non-idempotent methods like PATCH, it's a condition that robust server infrastructure should probably handle, and consequently, a robust client should probably anticipate.
Given this, a client aiming for wide interoperability should likely be prepared to receive a 409 Conflict at any point and use the Offset Retrieval mechanism to recover. If that's a fair assumption, then perhaps recommending a 409 response for this granularity mismatch isn't a significant departure from what a robust client should already implement.
This leads to a practical question for a major client implementation. @guoye-zhang, out of curiosity, how does the current macOS/iOS client (using NSURLSession) handle this today? Will it attempt to recover from an "unexpected" 409 Conflict on a PATCH request by fetching the latest offset, or does it treat it as a fatal error? How does its behavior compare when receiving a generic 5xx response?
URLSession does not retry 4xx responses, only 5xx responses or dropped connections without any responses. 4xx responses will be delivered as the final response to the application.
Got it, thanks @guoye-zhang.
So if the client and server get out of sync due to a flaky connection, the client's retry would get a 409 Conflict and the upload would end up failed?
Seems like any robust client has to treat 409 as a special case to recover, and maybe the spec should be clearer on that?
We haven't received any complaints about 409 yet. I think it is useful to have a mechanism for the server to say "you messed up, don't talk to me again", and 4xx status code seems to be the best option for that. Middleware retrying non-idempotent requests would likely cause worse bugs like duplicated payments, but I'm not a server developer so I don't know how common it is. Do you think it's important for clients to recover from that?
Hey @guoye-zhang, thanks for the perspective.
Do you think it's important for clients to recover from that?
Yes, absolutely. To me, this is the entire point of the protocol. If a spec-compliant client gives up on a recoverable 409, the protocol fails to deliver on its core promise of reliable uploads over unreliable connections. The spec even anticipates this exact scenario in the Concurrency section, describing how clients and servers can get out of sync even when both are behaving correctly. These sorts of network interruptions (and therefore desyncs) are definitely quite common at broad scale on the internet.
For that reason, I think the spec should unambiguously state a client MUST treat a 409 Conflict w/ Upload-Offset as a recoverable signal and use the provided offset (or the offset retrieval mechanism) to resume. I could perhaps be persuaded that SHOULD is more appropriate (at the moment it's left kind of ambiguous), but given the trend of this spec being implemented in otherwise low-ish level generic HTTP libraries like NSURLSession I think it might be wise to err on the side of being prescriptive with MUST. While blindly retrying non-idempotent requests is rightly discouraged as it can cause issues like duplicated payments—that's not what this recovery mechanism is. A client resuming after a 409 isn't performing a blind retry; it's using the server-provided state (Upload-Offset) to make a corrective request. Mandating this (or at least strongly encouraging it) ensures critical recovery logic is baked into these foundational libraries, providing robust, out-of-the-box reliability for application developers who might not be aware of this specific failure case.
Since this is a bit tangential to the granularity discussion, I'm happy to spin it off into a separate issue if that makes more sense. Not sure to what extent this has been discussed previously
Yes, it's probably best as a separate issue.
The main point of the concurrency section is to avoid the server generating a 409 to a spec-compliant client, so it is the server's responsibility to discard the in-process upload at the point of offset retrieving in order to finalize the offset. A client should never get a 409 if it resumes an upload at the offset given by the preceding offset retrieving.
It was brought up that terminating an in-progress upload is too hard to implement in a distributed system, an alternative implementation strategy is for the server to overwrite the bytes it already received rather than sending a 409 which can also satisfy the spec.
Thanks I've spun off into #3275 where I've tried to add some more detail about why I think spec complaint servers will inevitably need to return 409 Conflict responses.
Another scenario to consider is that proxy body size limits can make the max-append-size required in practice, meaning the upload creation request will likely get a 413 response outside of the server's control. Depending on the proxy, the response might happen based on Content-Length without even reaching the server, or after the client actually uploads that amount.
Perhaps 413 could be used in the spec to indicate a limit-related issue. It could prompt the client to make an OPTIONS request to learn the upload limits before continuing, if it hasn't already learned them. The server could also optionally include the Upload-Limit header in the 413 response so the client can skip the OPTIONS request (optional because as noted, it may not always be possible to provide).
For more context, I'm currently working on implementing an RUFH server in my project that caters to self-hosted deployment. Administrators often use services like Cloudflare Tunnel, which require individual requests to not exceed a certain value (100MiB in most cases), even if the total transferred data is allowed to be much higher. It seems my only option to make RUFH work for these environments is to make the server respond with 5xx before the proxy decides to respond first with 413, which is quite brittle and unclear if it even works.
Edit: testing with URLSession, the described workaround unfortunately does not work either since a request over 100MiB never even reaches the server. I feel that expressing and respecting limits should be handled with more care. Features such as chunk granularity are useful to the extent the client will respect such limits and seek them out when needed. The Upload-Limit header is simply unreliable from a server perspective as of now, which makes it difficult to integrate with actual limits in other parts of the architecture.
Sent https://github.com/httpwg/http-extensions/pull/3320 to try to discuss a more concrete proposal.
One thing I noticed in writing this is there is some ambiguity at the moment about whether clients need to respect limits or not since the phrasing "A client that is aware of this limit MUST ..." is used. Any idea what the intent of this wording is? I can see this making sense for new limits that are introduced after the initial spec is finalised, but for now it seems a bit unclear.
The limits are advisory. URLSession does not currently parse the Upload-Limit header so it does not respect the limits.
Gotcha, I think we should try to rephrase a bit then even independent of the potential introduction of append-granularity since the language is a bit contradictory:
"The client SHOULD respect any limits (Section 4.1.4) announced in the Upload-Limit header field in interim or final responses" -> implies optional but strongly recommended
"max-append-size: Specifies a maximum size counted in bytes for the request content in a single upload append request (Section 4.4). The server might reject requests exceeding this limit. A client that is aware of this limit MUST NOT send larger upload append requests. The value is an Integer." -> implies clients are required to respect this limit (since why wouldn't they be "aware" of it? It's included in the spec, and clients are guaranteed to have received it in the Upload Creation response if the server sets it).
Overall I'd be keen to make as clear as possible that it's (at least) strongly recommended that clients which intend to be maximally interoperable with a variety of servers respect all known Upload-Limits. I'm happy to take a stab at rewording to try to remove this ambiguity if we have alignment here.
@danielresnick Thank you for feedback regarding the language around the Upload-Limit header. Would you find filing this as a new issue?