nostr-protocol
allow NIP-44 to encrypt more than 65535 bytes
This is backwards-compatible and no one has to change anything if they do not care about bigger messages. The only reasonable case where this necessity has shown up so far was signing of big kind:3 lists via NIP-46.
There is no need for the 0 length signal. You can just check if the whole message is bigger than 65603 and go for the 4-byte length.
Meaning.. current systems will misread the length bytes and will not be able to decrypt anyway. So why bother.
There is no need for the 0 length signal. You can just check if the whole message is bigger than 65603 and go for the 4-byte length.
Sounds good to me. Is that number 65603 exact?
I am not sure. I got from the documentation.
We need to get some tests for -1, 0, and +1 lengths from the 2-byte limit to ensure everyone is on the same page
@vitorpamplona I think the current proposal is slightly easier to understand and the code for it looks clearer, and it's easier to debug for legacy implementations (you get a length=0, it raises some eyebrows).
So even if it wastes 2 bytes I think we should go with it. Yay or nay?
Makes sense. Has anybody coded this? I am still stuck in my outbox refactoring :(
Has anybody coded this?
Yes, this is the diff:
I am still stuck in my outbox refactoring :(
That's good to hear!
Coding this, I am realizing a few things:
- The max continuous bytearray in any JVM-based system is 2GB (signed int)
- Most Android phones will limit memory to 512MB. Some temporarily bump it up to 1GB. Regardless, it is likely that apps will crash when dealing with bigger payloads.
The recommendation for anything over 50MB is to make the Websocket interface append individual frames directly to a memory-mapped file (which I don't think any higher-level websocket library actually offers) and first decode the base64 into a binary file and then decrypt that file either via inputstreams or memory maps
The max continuous bytearray in any JVM-based system is 2GB (signed int) Most Android phones will limit memory to 512MB. Some temporarily bump it up to 1GB. Regardless, it is likely that apps will crash when dealing with bigger payloads.
that's an issue I didn't foresee, not a huge problem, can be addressed by a version bump and a slightly different payload structure.
we could chunk the data to 128kB blocks and give each block its own 32-byte HMAC tag.
That way you only need to keep 128kb in memory at a time. The only downside is an additional 32 bytes per 128kB.
That change removes the "must-buffer-everything" requirement while keeping every cryptographic primitive that NIP-44 v2 already uses.
The recommendation for anything over 50MB is to make the Websocket interface append individual frames directly to a memory-mapped file (which I don't think any higher-level websocket library actually offers) and first decode the base64 into a binary file and then decrypt that file either via inputstreams or memory maps
this is a non issue, it's relatively simple to do, some nostr libraries may have to be adapted, however.
EDIT: we may not even need a version bump if we do it every 64kb rather than 128kb, and we get the advantage that clients that do not support higher payloads can just stop reading after the first 64kb!
@vitorpamplona interesting, the additional complexity of avoiding memory limits (on both server and client, along with any attacks that deliberately craft huge payloads to crash implementations) makes the low limit more of a feature that supports protocol simplicity rather than a bug. Maybe the current limit is ok, or we could find some other lowest common denominator.
I am not opposed to growing the size.
Still, implementers need to be aware that decrypting takes four to five times the size of the payload in memory, given the multiple decoding and byte array slicing needs of the algorithm.
Maybe we should just not allow people to have such big lists and institute 1800 as a kind of soft-limit for the protocol.
Most relays won't even accept events this big anyway (and they shouldn't!).
And no human can really "follow" that many people (sure, they can click a button that many times, which isn't the same as really following).
I agree to a point, but the extended nip-44 can have other uses as well (such as larger gift wraps).
For follow lists if we just encode the bytes directly you can follow twice the amount of people for the same amount of data.
❯ nak req -k 3 -a 66675158e6338fe89fda418e42a0bf2a7a2b132504dd347f015a18971b644430 wss://relay.arx-ccn.com 2>/dev/null | wc -c
18513
❯ nak req -k 3 -a 66675158e6338fe89fda418e42a0bf2a7a2b132504dd347f015a18971b644430 wss://relay.arx-ccn.com 2>/dev/null | nostr2bin | wc -c
9477
Agreed. I don’t think any of the points above are good reasons to keep NIP-44’s size limitations.
- ChaCha20 is a stream cipher, and it seems both the padding and the MAC can be handled on a block-by-block basis (correct me if I’m wrong). Despite the Python example code, as far as I can tell nothing in this algorithm requires large arrays or holding more than a small buffer in memory for encryption or decryption (again, correct me if I’m wrong).
- We’ve already seen practical cases of people exceeding 64 KB follow lists, with NIP-44 breaking Nostr functionality as a result. On social media, "following" often equates to "connecting", and I personally had more than 1.8k connections on multiple platforms. Yes, we need to review how the Outbox model should work at scale, but that’s a separate problem. Also, this is a non-argument since NIP-44 is not used exclusively for follow lists (e.g., all other NIP-51 lists and sets, NIP-17 messages, etc).
- Nostr, as a protocol, shouldn’t stop me from encrypting a payload of whatever size I want from my own client to my own relay. Limitations of popular Nostr software, including relays and clients, shouldn’t dictate protocol specs or encryption mechanisms.
- While I agree that size limitations make things easier for developers (since they don’t have to define their own policies). I don’t think convenience should be the primary driver for encryption design. In a way, this is similar to saying “NAT is just convenient”. Nostr needs a proper way to encrypt payloads larger than 64 KB. I also wouldn’t worry too much about performance here. I've used ChaCha20 to encrypt and decrypt multiple near real-time video streams simultaneously on consumer-grade hardware without hitting CPU limits. Setting a more reasonable limit for payloads shouldn’t have a major performance impact on clients. Even raising the cap to somewhere between 256 KB and 1024 KB would be a huge improvement in usability.
Nostr, as a protocol, shouldn’t stop me from encrypting a payload of whatever size I want from my own client to my own relay. Limitations of popular Nostr software, including relays and clients, shouldn’t dictate protocol specs or encryption mechanisms.
I don't actually think this is true. If some given payload gets rejected by 99% of relays in the wild, that should be considered a constraint of the "protocol". Keeping constraints relatively tight encourages implementers to create payloads that are more likely to be accepted. Scenarios that require interoperability are distinct from those in which someone who doesn't want their event broadcasted, and are using nostr only incidentally, in which case they can use non-standard encryption on their private infrastructure. That's not to say we can't find a middle ground here, but 4GB probably isn't it.
That's not to say we can't find a middle ground here, but 4GB probably isn't it.
The question, of course, is what is reasonable for which use cases, and ultimately what the boundaries are for Nostr (the protocol). A 4 GB limit is certainly unreasonable for short notes, comments, and follow lists, which is what much of Nostr’s early-stage software is optimised for. And of course, we have Blossom for media. But I can imagine many use cases where Nostr, as a protocol, could serve as a means to discover, write to and share a huge stream of ChaCha-encrypted data, with ephemeral notes that may never touch today’s relays but could drive tomorrow’s killer applications for Nostr.
My point is that NIP-44 is about cryptography. We shouldn’t artificially limit a payload encryption scheme just because it is convenient for the “Decentralised Twitter” use case, even if that is what a lot of Nostr currently looks like. New types of relays and clients can define their own adequate limits for their intended use cases. And if we want to define hard limits as part of the standard, I would prefer that it be done on a kind by kind basis, so we at least have more reasonable input to decide what’s adequate for each use case.
For nip44, 2^32 is fine. I updated my implementation.
For nostr events, gossip limits websocket messages to 1 MiB, so that sets a practical limit there. Tunstenite library limit is 64 MiB. Long ago I had heard people with >2k followers were hitting a 128 kb limit somewhere.
So, 3 implementations and looks like we sorta kinda have an agreement. Merge? :)
Looks ok, but would be great to understand precisely how padding works with big payloads: performance impact, memory overhead etc. Does it make sense to use it for 4GB payload?
FYI max msg size for chacha is 256GB (2**38-64).
We shouldn’t artificially limit a payload encryption scheme
@aaccioly the limit was primarily added because of padding. It's not clear how beneficial the padding is for such large messages.
Assuming bumping to version 3 is not an option? Bumping would allow to not add those "workarounds" (if 0000 then ... else ...); which are not 100% backwards-compatible, contrary to what's mentioned in OP-post.
Assuming bumping to version 3 is not an option? Bumping would allow to not add those "workarounds" (if 0000 then ... else ...); which are not 100% backwards-compatible, contrary to what's mentioned in OP-post.
The version bump would cause issues because some clients would start sending out v3 and some others would only accept v2 then the messages would fail even though the message was small and would have worked under v2 just fine.
Well, unless the v3-capable clients chose to use v2 when messages are small and v3 when they're big, but that sounds like an ugly workaround too, and more confusing because the numbers will imply that v3 is strictly better than v2 or something, and will say nothing about sizes.
Anyway, my opinion now is that we shouldn't change anything and let people with too many follows suffer.
Anyway, my opinion now is that we shouldn't change anything and let people with too many followers suffer.
This is similar to the Knots vs Core debate. Filtering at the encryption level doesn't make much sense. Relays will filter payload sizes on their end anyway. That's the final "block size" limit.
Even if we do this, people with too many follows will still suffer.
But getting rid of this limitation will open the encryption to be used in other things. Maybe even in large Blossom encrypted files.
test vectors are not defined for >64KB messages, it may be useful to add them to ensure implementations produce the same input / output.
since including 0.1-1GB test vectors is useless, they should be generated at test run-time
Any solution that overloads a event kind's payload with a differently encryption scheme (whether v3 or fiatjaf's hack) will result in unsupporting clients failing to read the data, and potentially nuking it if the user attempts to write to it. This is ok for RPC stuff (since no one is writing to a replaceable), but would be pretty bad for NIP 51 lists. Another reason why replaceables were a mistake.
Do we need to create new event kinds for use with the new encryption scheme? That would be madness, so I figure we'll just end up forcing everyone to upgrade to support the new encryption standard. @fiatjaf's approach is the only one that actually avoids the problem of dropping data.
Anyway, my opinion now is that we shouldn't change anything and let people with too many follows suffer.
And people using using NIP-17 or NIP-EE encryption for any messages bigger than 64k, and any kind of big encrypted lists or sets, and... Everyone gets to suffer 🤣
I'm ok with having a potentially "wasteful" backwards compatible "patch" to v2 now and then have a properly audited more thought out v3 that clients can slowly adopt.