nostr-protocol
NIP-96: List files / rewording
Adding an endpoint to list uploads
Also included a bit of rewording, this NIP is far too long.
read: https://github.com/nostr-protocol/nips/blob/nip96-sync/96.md
Much better than before for sure!
Regarding the listing route, the response is JSON which may eat too much memory when generating it if user has uploaded too many files.
One way to solve this is to let the server optionally add a "nextPage": "<url-to-list-remaining-files>" to the response if it wants to split response in pages.
I’ll review and provide comments in the next 1-2 days. Thanks for the tag.
Love the simplification and corrections, not a fan of the listing thing for many reasons that could expose the service to DDoS, bandwidth overconsumption, extra complexity and lack of consideration for large lists that are not always safe to share. Due to users' privacy concerns and potential harmful (CSAM, etc.) images that are not yet filtered/monitored, and are not safe to list.
This is for users to list their own files only
Love the simplification and corrections, not a fan of the listing thing for many reasons that could expose the service to DDoS, bandwidth overconsumption, extra complexity and lack of consideration for large lists that are not always safe to share. Due to users' privacy concerns and potential harmful (CSAM, etc.) images that are not yet filtered/monitored, and are not safe to list.
This is for users to list their own files only
Ah, I missed Returns a list of files linked to the authenticated users pubkey. line completely. That makes much more sense now.
@v0l For the listing route, please check my review comment above. You are exposing fields that are already part of the nip94_event object.
No one is counting much on my opinion but it's fine with me 😀😅🤓
nostrcheck-server will implement it.
Could we add an optional param to request "no transformation" from the server, so that GET requests would always serve the original file by default? And if it's not supported by the server it could reply with some 40x code. It seems like this single addition would make this nip supercede blossom.
Could we add an optional param to request "no transformation" from the server, so that GET requests would always serve the original file by default? And if it's not supported by the server it could reply with some 40x code. It seems like this single addition would make this nip supercede blossom.
I think i would rather drop the entire "original file" concept and just allow uploads with no compression, i think the ox tag was a big mistake.
When you upload a file to some server and it gets compressed then that file should be the original hash.
@v0l Perhaps the sever could reply with a hash for each of the variants it intends to store so that the client can verify that they have not changed.
Could we add an optional param to request "no transformation" from the server, so that GET requests would always serve the original file by default? And if it's not supported by the server it could reply with some 40x code. It seems like this single addition would make this nip supercede blossom.
I think i would rather drop the entire "original file" concept and just allow uploads with no compression, i think the
oxtag was a big mistake.When you upload a file to some server and it gets compressed then that file should be the original hash.
Then why are you running imgproxy on snort and not serving the original media? I think you are mixing file server and media server and nip-96 is mainly for media.
@v0l Perhaps the sever could reply with a hash for each of the variants it intends to store so that the client can verify that they have not changed.
What if variant computed on-demand as it should be? Did you take into account the time it will take to compute and the resources it will require to do it? Did you consider video?
Could we add an optional param to request "no transformation" from the server, so that GET requests would always serve the original file by default? And if it's not supported by the server it could reply with some 40x code. It seems like this single addition would make this nip supercede blossom.
I think i would rather drop the entire "original file" concept and just allow uploads with no compression, i think the
oxtag was a big mistake. When you upload a file to some server and it gets compressed then that file should be the original hash.Then why are you running imgproxy on snort and not serving the original media? I think you are mixing file server and media server and nip-96 is mainly for media.
My imgproxy does hashcheck on every image, which is why images were broken for a time from nostr.build because of CF Polish, nostr.build was not serving the original images
Could we add an optional param to request "no transformation" from the server, so that GET requests would always serve the original file by default? And if it's not supported by the server it could reply with some 40x code. It seems like this single addition would make this nip supercede blossom.
I think i would rather drop the entire "original file" concept and just allow uploads with no compression, i think the
oxtag was a big mistake. When you upload a file to some server and it gets compressed then that file should be the original hash.Then why are you running imgproxy on snort and not serving the original media? I think you are mixing file server and media server and nip-96 is mainly for media.
My imgproxy does hashcheck on every image, which is why images were broken for a time from nostr.build because of CF Polish
The point is that the user does not see the original, regardless where it is modified. And imgproxy does a fairly poor optimization to begin with. In the end, asking a media host to serve immutable file is very single-sided argument that benefits only few clients and not the user. Plus, users are very bad at sanitizing their media and leave all kind of identifying metadata in them, hence the need to strip, re-compress, and resize.
nip-96 is mainly for media.
The title of the NIP is "File Storage Integration"
Does it matter what the title is when the "spirit" of the NIP is towards media? Sure it can be expanded to other media, hence the request to add "immutability" flag from the @nostrband
@fishcakeday i dont think anybody was arguing about removing the resize/compress step, i think its vital, not sure where you are going with these comments.
Perhaps I misread "I think i would rather drop the entire "original file" concept and just allow uploads with no compression, i think the ox tag was a big mistake." part to think that we want to stop compressing and leaving the file as-is. Now reading it again, I see that the proposal is opposite, and to just drop the original hash. But, we do compute the original hash for the purposes of naive deduplication, so it is still there. Sorry if I misunderstood you.
@fishcakeday
What if variant computed on-demand as it should be? Did you take into account the time it will take to compute and the resources it will require to do it? Did you consider video?
Fair, you are right.
@v0l
I think i would rather drop the entire "original file" concept and just allow uploads with no compression, i think the ox tag was a big mistake.
Yes I think the server should just return one hash that has different meanings:
- If the uploader requests no transformation, then this is the hash of the original file and it is expected that the server will continue serving a file that matches this.
- Otherwise (default) this is not expected to match any files served since the server might apply any transformations it wants. This can be viewed more like an id, although in practice the server may use the hash of the initially compressed file or whatever, as long as it is unique.
Could we add an optional param to request "no transformation" from the server, so that GET requests would always serve the original file by default? And if it's not supported by the server it could reply with some 40x code. It seems like this single addition would make this nip supercede blossom.
I think i would rather drop the entire "original file" concept and just allow uploads with no compression, i think the
oxtag was a big mistake.When you upload a file to some server and it gets compressed then that file should be the original hash.
How does "allow uploads with no compression" differ from my proposal for "no transformation" option? I'm missing the point maybe.
I don't care what hash the server returns, it may lie, and that's what I need to protect against. A client knows the file hash, uploads the file, and then hash is published somewhere, and then everyone can download using that hash and verify file's integrity, and can reupload the same file everywhere and everyone can verify it. AFAIU that's the point of blossom. I thought a single optional upload param could work for this. Or not?
Could we add an optional param to request "no transformation" from the server, so that GET requests would always serve the original file by default? And if it's not supported by the server it could reply with some 40x code. It seems like this single addition would make this nip supercede blossom.
I think i would rather drop the entire "original file" concept and just allow uploads with no compression, i think the
oxtag was a big mistake. When you upload a file to some server and it gets compressed then that file should be the original hash.How does "allow uploads with no compression" differ from my proposal for "no transformation" option? I'm missing the point maybe.
I don't care what hash the server returns, it may lie, and that's what I need to protect against. A client knows the file hash, uploads the file, and then hash is published somewhere, and then everyone can download using that hash and verify file's integrity, and can reupload the same file everywhere and everyone can verify it. AFAIU that's the point of blossom. I thought a single optional upload param could work for this. Or not?
I think it will work, and I see no issues with it. We just need to add the spec (optional) to this NIP and that is all. I see no issues if the client (preferably only for some media or files) will request immutability of the content. Server will still have the right to reject based on the presence of private metadata or malicious content/exploits, etc.
How does "allow uploads with no compression" differ from my proposal for "no transformation" option? I'm missing the point maybe.
The reason i want this is to allow copying of files between NIP-96 servers without modification so the hash is the same, in the same way that blossom allows cloning of files, if 2 different implementations process the media in different ways then the resulting hash might be different if you re-upload the original.
If the server returns the file hash (after processing) then i take this as the "original" and then replicate it to other servers (keep in mind that i think ox is a mistake), i think this is better than uploading an un-compressed image to multiple servers, you ask your first preference server to compress the media then you copy it, other servers can always reject the upload if they dont want to allow replication of files (no-compress).
Probably each implementation needs to check if an image/video is "compressed enough" to allow replication.
At the end of the day, the user is still going to be paying for space (usually) so it shouldn't matter if its compressed or not.
@v0l
If the server returns the file hash (after processing) then i take this as the "original" and then replicate it to other servers (keep in mind that i think ox is a mistake), i think this is better than uploading an un-compressed image to multiple servers, you ask your first preference server to compress the media then you copy it, other servers can always reject the upload if they dont want to allow replication of files (no-compress).
This assumes that the first server will not change the file again or serve different versions so I'm not sure that your scheme will work. I would suggest compressing client side and then sending the result of that to whichever servers you want with the "no modification" flag set.
How does "allow uploads with no compression" differ from my proposal for "no transformation" option? I'm missing the point maybe.
The reason i want this is to allow copying of files between NIP-96 servers without modification so the hash is the same, in the same way that blossom allows cloning of files, if 2 different implementations process the media in different ways then the resulting hash might be different if you re-upload the original.
If the server returns the file hash (after processing) then i take this as the "original" and then replicate it to other servers (keep in mind that i think
oxis a mistake), i think this is better than uploading an un-compressed image to multiple servers, you ask your first preference server to compress the media then you copy it, other servers can always reject the upload if they dont want to allow replication of files (no-compress).Probably each implementation needs to check if an image/video is "compressed enough" to allow replication.
At the end of the day, the user is still going to be paying for space (usually) so it shouldn't matter if its compressed or not.
I've drafted a "no_transform=true" option above.
Could it be used for copying of the compressed file? You upload to a trusted server, get the compressed file and it's hash (when it's ready?) and then can publish the compressed hash, while client/server may reupload compressed version with "no_transform=true".
This assumes that the first server will not change the file again or serve different versions
Once the file is compressed, the hash is returned, you put this hash in your posts as the x tag and so clients check when they fetch this file that the hash is correct, the server cannot modify it again or it will break.
This is how i have implemented it in void-cat-rs i don't return the ox or use it at all, there is only 1 hash, and it is the hash of the file after compression.
This assumes that the first server will not change the file again or serve different versions
Once the file is compressed, the hash is returned, you put this hash in your posts as the
xtag and so clients check when they fetch this file that the hash is correct, the server cannot modify it again or it will break.This is how i have implemented it in
void-cat-rsi don't return theoxor use it at all, there is only 1 hash, and it is the hash of the file after compression.
As far as I understand (please correct me if I'm wrong), @fishcakeday would like to serve dynamic variants of the same file, which I can see why it would be useful (especially for video). Your approach only works if the server only compresses the file exactly once and serves that forever.
@sant0s12 on the subject of versions, i don't really see how this can work, to me each version is its own upload, if you want multiple versions then you need to upload using the type or another spec should be created to ask the server to resize the image for you.
Your approach only works if the server only compresses the file exactly once and serves that forever.
Yes this is how it should work, you cannot modify it after the fact as some sort of optimization because the hash will not be the same, the point of this spec is to avoid the media rugpull, you cannot serve whatever you want it has to be the same media that the user uploaded in the first place.
@sant0s12 @nostrband @fishcakeday this is completely off topic for this PR btw, i was going to make another PR to add the no_transform option later but here we are lol
@sant0s12 @nostrband @fishcakeday this is completely off topic for this PR btw, i was going to make another PR to add the
no_transformoption later but here we are lol
LOL! I was thinking the same, we complete this PR and open a new one for the no_transformation purposes.
As for the variants, NB (nostr.build) already shares the links to the transformed versions that are not widely used by present in the response. And for immutable content, you can still redirect the user to the optimized version and offer the original. Let say a user uploads video.mov (MOV is shit for streaming as a container but here we are) and we offer video.mov/optimized.mp4 and video.mov/poster.jpg video.mov/thumbnails.png video.mov/stream.m3u8 video.mov/stream.mpd etc. Note that original is still available for download if needed, with disposition for that purpose.
https://github.com/nostrbuild/nostr.build/blob/52c1b78aa8b67c1b70231abb8c4e5e0f6fac179e/api/v2/helper_functions.php#L192 - is where the specs are described