httpwg
RESUMABLE: What happens when a POST request is made for an already-created incomplete resource?
My use-case is that a given user can only make one resource with a given checksum. This checksum is provided in the Repr-Digest header in the initial request and validated on upload completion. The resource URI itself is not a checksum, but a UUID. Once the client has created the resource with the initial POST request, it should either complete it (possibly in multiple parts) or cancel it. Once cancelled, the user is allowed to create a new resource with a different UUID for the same checksum.
The spec is clear enough that a complete resource should not be modified and provides a response to use for this case. However, what should happen if the client creates this resource without completing it, then tries to create it again?
- Should it be rejected with a 400 code?
- Should it cause the incomplete resource to be invalidated?
- Should it include a Location header to direct it to the existing resource?
- Should it truncate the existing resource and start over?
Given the client needs to know the resource in order to resume or cancel it in this case, I think it makes sense to include the Location header with an error status code to allow them to take further action, possibly also including the Upload-Offset.
A good way to think of the resumable upload is that it's just an in-progress regular upload. What do you do if a user performs 2 concurrent POSTs with the same checksum without using resumable upload? Whatever you decide to do should apply here as well.
In general, we recommend that the new attempt overrides the old attempt since the client might have recovered from a crash and unaware of the old attempt. Yes, you could design something to try to recover the state, but I'm not certain that we need to add the complexity to the protocol. Essentially this would be adding a lookup mechanism of the temporary resource from the file checksum.
As I understand it, the spec as written doesn't seem to have a way for the client to resume an upload if it doesn't know or forgets the URI provided by the server, such as if it crashes or restarts, unless it persists this somewhere. This would be a way for the client to retrieve this URI from the server based on some criteria (in this case the user ID and checksum). It would also mean the upload could be resumed from a different client with the same file.
It's of course possible to have a custom design that does this when the client and server agree on this behavior, but it'd be nice to have a standard way to do this that all RUFH clients can understand.
It is achievable today with a 104 response followed immediately by a 5xx response. The client would perform offset retrieving and upload appending. We might want to change this sentence in "Offset Retrieval" but it's not a MUST:
The offset can be less than or equal to the number of bytes of representation data that the client has already sent.
That's clever! It's maybe semantically incorrect since it isn't really a server error, but it addresses the use-case in question.
Yeah we could in theory define a separate status code for "please retry", but the workaround above should already work with the implementation today on iOS/macOS
It is achievable today with a 104 response followed immediately by a 5xx response. The client would perform offset retrieving and upload appending. We might want to change this sentence in "Offset Retrieval" but it's not a MUST:
The offset can be less than or equal to the number of bytes of representation data that the client has already sent.
The spec doesn't prohibit the offset from jumping ahead (from the client's perspective), but it explicitly allows clients to error out in such cases:
The client MAY reject an offset which is greater than the number of bytes it has already sent during this upload.
Therefore, I would assume that this server behavior won't work with every client.
Should it allow the client to reject a greater offset? Even a streamed input can work with a greater offset.
Allowing clients to reject greater offsets can make their implementation simpler, especially if they are uploading streaming data. In addition, the offset should not jump ahead for usual uploads where one client uploads a representation independently of previous uploads. That was the reason behind allowing this client behavior so far.
I can say that as a server implementer, although supporting all RUFH clients is ideal, my priority is targeting URLSession and probably one or two more clients for other platforms. So long as it works for those target clients, I will consider the approach viable. If I did want to handle other clients as a server, I would need to go by user agent or something to detect what the client supports and behave differently for them. Allowing clients to reject this thus comes with the caveat of potentially fractured interoperability.
URLSession passes the responsibility to its users. If they are uploading a file, arbitrary offset is supported automatically. Otherwise they are asked to provide a request body stream from a certain offset which they might be able to do or decline.
I suppose that would still be fine as long as the client DELETE's the upload resource if declined. The only issue is if they both can't accept the offset and also take no action, since it would just get stuck at that point.
We haven't implemented upload cancellation in URLSession yet, so yeah it will be stuck
We haven't implemented upload cancellation in URLSession yet, so yeah it will be stuck
Hmm, perhaps it should be an opt-in behavior through a request header to be safe. If it isn't provided, the server can assume it won't support the offset and overwrite the earlier attempt instead.
Out of curiosity, is the RUFH handling open source? I might be able to contribute some spec functionality if so, mainly cancellation and limits in mind.
URLSession is not open source unfortunately. Please file feedbacks requesting this and send me the FB number, so it can be prioritized.
URLSession is not open source unfortunately. Please file feedbacks requesting this and send me the FB number, so it can be prioritized.
Thank you for prioritizing this! I made the feedbacks FB20751965 and FB20752104. I'm happy to provide more details if needed or participate in dev betas to give more feedback.
Can we close this issue or are there inquiries to adapt the resumable upload process? To me it seems that the questions have been answered and no further action is necessary for now.
The spec could probably use different wording to acknowledge that the server is allowed to send a greater offset and that the client should accept it if it's able. The 5xx status is also a bit awkward and might benefit from a more explicit response. These are the main things that come to mind for this issue.
https://github.com/httpwg/http-extensions/pull/3319 proposed text for handling 409 on the client-side.