nips
nips copied to clipboard
nostr-protocol
NIP-04 considered harmful
Hi - just nostr seems promising, and end to end encrypted communication is a very important part of it, but the NIP-04 spec as written should not be implemented.
There is another issue here: https://github.com/nostr-protocol/nips/issues/72 pointing out the non-uniform AES key. That issue has been closed but probably shouldn't be -- the spec still has the unhashed / truncated result of the DH as the AES key.
Another, I think more severe problem is that protocol as described uses aes-256-cbc with no message authentication. This means that messages can be undetectably altered in transit. Anyone relaying the message can change the message, and the receiver can't tell that it was changed. I would remove NIP-04 before people start trying to use it.
There are other encryption protocols that are used with the secp256k1 keys used in nostr that could be adapted for direct messages, such as BIP324 (https://github.com/bitcoin/bips/pull/1378) or lightning's bolt 8 (https://github.com/lightning/bolts/blob/master/08-transport.md). There are also ratcheting protocols which have forward secrecy like the one used by signal, but those have other trade-offs.
The BIP324 and bolt 8 protocols aren't for discrete messages; they are for encrypted & authenticated communication channels. That means it doesn't directly replace NIP-04; the bolt 4 (https://github.com/lightning/bolts/blob/master/04-onion-routing.md) onion messages would be a closer fit. But encrypted transport between nodes is also important.
The code from BIP324 and LN is available in different languages and open licenses so I think that's the best bet for getting some code that's been used and tested.
I have to agree with @adiabat here. NIP04 is potentially harmful.
However, @fiatjaf's comment on the non-uniform AES key issue seems to be a reasonable approach. There is just the risk that people start to use NIP04 with sensitive information without knowing the downsides.
By the way NIP-04 was never supposed to be the thing in the first place, it was made as a prototype to show it was possible and expecting someone to supersede it with a better protocol (I personally think we must still find some other more p2p more private way to send direct messages between Nostr contacts that doesn't involve broadcasting to relays).
IMO NIP04 should at least state its infancy and thus risks of metadata exposure (who sent to who, when and how many bytes, etc.), missing message authentication (alteration by relays) and potential vulnerable crypto (no key derivation after DH). I would even go a step further and put a paragraph into NIP04 that clients implementing is, MUST warn users of the its infancy and risks.
NIP04 is probably not production ready which can lead to harm,... especially with faster adoption of nostr in general.
Of course fixing the crypto part (use AES-GCM, run HKDF after DH, etc.) is probably the trivial part (new NIP). But what needs more thinking is how much users have to trust a nostr relay and if so, could clients be force to send certain EVENTS to only relays controlled by a trusted operator (receiver himself), etc.
A more p2p-ish approach where clients connect to each other directly sound interesting at first, but seems not to be practical because not having to be "online" 24/7 is a major feature of the nostr protocol. Relays could proxy connections to avoid having to build true incoming connections to clients (with the bandwidth downside for relays).
At the end, I guess its very important to give users a clear sense of what level of security the get with DMs on nostr. ... we still use emails without PGP for a lot of important communication.
@adiabat wrote
This means that messages can be undetectably altered in transit. Anyone relaying the message can change the message, and the receiver can't tell that it was changed.
@jonasschnelli wrote
NIP04 should at least state its infancy and [...] missing message authentication (alteration by relays)
I want to double-check something about those warnings. Nostr events are all signed. Is it really possible to alter encrypted message content without making signature checks fail?
I know NIP-04 messages are signed by the sender's pubkey, and not the shared secret between the sender and receiver, but I don't think this would allow any relays or third parties to alter message content while keeping the signature valid.
I guess I could be missing something, but the content is AES encrypted with shared secret of sender/receiver. So there are only 2 keys that can read the content. Sender, receiver. No?
The Event is then signed by the sender. As long as client is validating signature of received Event, I don’t see how the content can be altered?
What am I missing?
@adiabat wrote
This means that messages can be undetectably altered in transit. Anyone relaying the message can change the message, and the receiver can't tell that it was changed.
@jonasschnelli wrote
NIP04 should at least state its infancy and [...] missing message authentication (alteration by relays)
I want to double-check something about those warnings. Nostr events are all signed. Is it really possible to alter encrypted message content without making signature checks fail?
I know NIP-04 messages are signed by the sender's pubkey, and not the shared secret between the sender and receiver, but I don't think this would allow any relays or third parties to alter message content while keeping the signature valid.
the content IS encrypted with shared secret.
Did I say anywhere that it's not encrypted by the shared secret?
Yeah in the bottom part of your comment. 😅
he's talking about the signature, not the content
Ahhh... I misunderstood. Sorry @gkbrk
Anyways, I think maybe we are on the same page, but as I see it, the Event can be verified on the client that it's not been altered and as far as I see the encrypted content can only be decrypted by the sender/receiver as its encrypted with the shared secret from those keys.
Just trying to understand what I am missing.
Anyways, I think maybe we are on the same page, but as I see it, the Event can be verified on the client that it's not been altered and as far as I see the encrypted content can only be decrypted by the sender/receiver as its encrypted with the shared secret from those keys.
Just trying to understand what I am missing.
Yes, I agree with those. I don't see any way to alter the message content anywhere without being able to sign arbitrary Nostr events without having a private key. If the way we do schnorr signatures is broken, we have bigger problems anyway.
even if nip-04 isn't broken it still sucks, either party leaking their private key and having their entire convo history public is concerning. we should be brainstorming ratchet-like specs so that if a root key is leaked you're not screwed. would be happy to sponsor anyone who wants to take on that task.
even if nip-04 isn't broken it still sucks, either party leaking their private key and having their entire convo history public is concerning. we should be brainstorming ratchet-like specs so that if a root key is leaked you're not screwed. would be happy to sponsor anyone who wants to take on that task.
Yup, I don't disagree here, but the original premise of the issue here seems to be misguided unless Im missing something.
Im totally fine with using another approach, but just wanted to point out that the current NIP can be used if implemented correctly and as long as each key holder doesn't leak their key 😅
Events which encapsulate the NIP-04 messages being signed helps, and prevents the straightforward attacks, but ... why tempt fate? aes128-gcm is available pretty much wherever aes-256-cbc is, it's faster, and prevents these vulnerabilities if, for example, someone relays an encrypted message for someone else.
This may be out of scope, but for sending encrypted messages, it also feels sub-optimal that the protocol generates publicly verifiable signatures on top of all the encrypted messages it sends. Why does Alice need to prove to everyone that she's sent encrypted messages to Bob?
While hiding metadata is hard, and the perfect is the enemy of the good etc etc, there are straightforward, already implemented ways to avoid having a publicly verifiable record of all encrypted messages (including source and destination keys). If the system relies on the top level clear signing as the only way to maintain authenticity, and encryption and authentication aren't linked, it will be hard to improve metadata privacy.
It is probably better to make a parallel protocol that reuses Nostr's key aspects but optimized for direct communication. It could be basically the same relay infrastructure, but the event format would be an encrypted binary blob that only the receiver would be able to read, relays would be programmed to not leak anything except to the intended receiver, senders would not be identifiable at all and receivers would only be identifiable by a temporary decoy id.
The Event is then signed by the sender. As long as client is validating signature of received Event, I don’t see how the content can be altered?
I guess this is a fair observation. While the NIP04 encryption itself is missing a MAC, the envelope signature done by the same key might provide a similar result (but again, this is cooking your own crypto scheme and should be reviewed carefully).
This is a bit more of a high-level question, but is message authentication necessarily a desideratum? A lot of e2ee messengers don't sign the messages for the explicit purpose of plausible deniability. It might be worth considering the possibility that a potentially alterable message might actually be preferable to the alternative, should the decryption key fall into the wrong hands.
I think there's two components here -- one is if a stranger wants to start a direct message with you how do you notice that without being swamped by spam, and the other is maintaining a private conversation with someone you know over a public broadcast medium (which is not really any different to broadcasting over radio or similar anyway)...
For unsolicited messages from strangers, you could do a bunch of things: rate limit by requiring proof of work or an advance payment, have a foaf network and require an introduction first and just not allow messages from anyone who is too many degrees of separation away.
I think once you've done that, you can just focus on the second part -- I have a pubkey A, you have a pubkey B, we both know those things, and both of us want to communicate via a (semi-trusted?) relay R without anyone else knowing what we're saying, how much / how often we're saying things, possibly that we're communicating at all, or being able to confirm after the fact when A or B's private keys have been obtained, that any particular thing was communicated?
That is, I think you want (at least) two event kinds -- an introduction event that you can listen for to see if strangers want to talk to you, and the actual direct messages that can focus more on privacy, and could have single-use pubkeys and lossy filters to make it hard for even the relay you're using to do traffic analysis.
NIP 4 makes it easy to see who's talking to each other, how often/much they're talking, and lets anyone who compromises the keys later see past history. Might be worth updating the NIP to make it clear that it's a prototype and is known to have those flaws?
I think once you've done that, you can just focus on the second part -- I have a pubkey A, you have a pubkey B, we both know those things, and both of us want to communicate via a (semi-trusted?) relay R without anyone else knowing what we're saying, how much / how often we're saying things, possibly that we're communicating at all, or being able to confirm after the fact when A or B's private keys have been obtained, that any particular thing was communicated?
Some two years ago I was working on a simple protocol (not completely sure it's original, you guys probably might tell) that looks useful as the second part -- it was based on a public database that stores values indexed by public keys. I had no idea anyone was going to implement such a database (Nostr relays seem to be an implementation of such databases).
Basic idea is:
- Alice and Bob have a shared secret S1 (by whatever shared secret generation scheme).
- Alice and Bob both derive the same public/private key pair P1 from S1.
- Alice generates a second secret S2 (or both Alice and Bob might have generated multiple secrets from the start).
- Alice posts publicly a message M1 (Nostr note in this case) encrypted symmetrically (using S1 as key), and use P1 as the public key of the "user". The message includes info for Bob to generate S2, and a payload.
- Bob queries relay for messages that matches the public key P1.
- Bob decrypts M1, and obtains the payload and S2.
- Bob restarts the process from (1), using the new secret S2 as the shared secret.
As long as one of Alice or Bob interacts with the relay in a perfectly anonymous way (eg using Tor and with no authentication), the two ends of the communication can't be matched. With fixed rate decoy messages and queries, it becomes impossible to know when/how much each end is communicating.
Messages in each sequence can't be linked to one another unless the secrets are leaked.
If any of the secrets is leaked, you still can't say who wrote the messages, much less prove it. The secrets aren't linked to identities.
It's impossible for the relay to censor any specific exchange without stopping all communications of that type within the relay.
There are many other possible ways to implement it, but the fact that Nostr lets you query relays by public key makes a shared sequence of throw-away key pairs the obvious choice.
Please note this would still be secure and censorship-resistant (in the sense of not being possible to censor specific exchanges) even with just one relay available. The relay has also plausible deniability as they have no idea what's being sent (basically like a router in an ISP). For further reduction of legal liability, messages could be dropped after a fixed TTL.
Could also use "remailers" within this protocol to further enhance privacy. The remailer could either use this new scheme or just plain old NIP4, receiving an event encrypted, then posting it. Chaining multiple remailers means network analysis can't work as long as at least one remailer doesn't collude with the relay. This can remove the dependency on Tor (but needs new infra-structure).
I think that's actually a better use-case for Nostr relays than public messages (which isn't censorship resistant in any strong sense, likely in any practical sense at all).
Events which encapsulate the NIP-04 messages being signed helps, and prevents the straightforward attacks, but ... why tempt fate?
aes128-gcmis available pretty much whereveraes-256-cbcis, it's faster, and prevents these vulnerabilities if, for example, someone relays an encrypted message for someone else.
Using a library such as aes-gcm is indeed a better choice than the current aes-cbd scheme as the MAC part has been taken care of.
If the key is based on Curve25519, you can use the mature TweetNaCl library's authenticated encryption for end-to-end encrypted communication.
noble-secp author here. just want to provide some information:
- Topicstarter is right: nip04 is harmful
- aes-cbc also needs padding, and bad padding can be pretty bad.
- aes-gcm-128 is not a future-proof option. Use at least aes-gcm-256 or even better: xchacha20-poly1305.
- curve25519 (@xz-cn) is not cool because: 1) it needs a separate elliptic curve, which can be pretty big if written from scratch 2)
Ideally: secp256k1 (perhaps x-only ECDH) + HKDF + XChaCha20-Poly1305; will still need an audit and comparison to protocols like the one Signal is using. It's important to not implement just anything from scratch without consensus first.
At least, if you deploy something like this, make it versionable. If protocol gets broken, you'll at least be able to upgrade it.
I think that crypto is about right. I pointed out the lack of an HKDF a while back.
As for protocol versioning, if the crypto is ever broken we can just create a new event kind with a new form of DM. In fact, that is what we have to do if we were to make changes, because NIP-04 is implemented in many places we can't go changing it now. So a new DM spec would be a new event kind.
BUT, I'm not in favor of DMs over nostr. I'm in favor of advertising an endpoint in your metadata, and using out-of-band messaging.
I think that crypto is about right. I pointed out the lack of an HKDF a while back.
As for protocol versioning, if the crypto is ever broken we can just create a new event kind with a new form of DM. In fact, that is what we have to do if we were to make changes, because NIP-04 is implemented in many places we can't go changing it now. So a new DM spec would be a new event kind.
BUT, I'm not in favor of DMs over nostr. I'm in favor of advertising an endpoint in your metadata, and using out-of-band messaging.
In that case, you will need external support for the DM function... I guess that is not the idea of the Nostr protocol.
What we can do is indeed to add a new type of event. Which not only makes the messaging part safer, but also includes the protection of the DM metadata. In that case, other people won't know to whom you have DMed and when. I understand there is a repo working on this already? https://github.com/SebastiaanWouters/emon
I think @jb55 https://github.com/nostr-protocol/nips/issues/107#issuecomment-1363215100 and @fiatjaf https://github.com/nostr-protocol/nips/issues/107#issuecomment-1363261759 made very good points, and @paulmillr https://github.com/nostr-protocol/nips/issues/107#issuecomment-1426854522 on the crypto. I offer my concurrence with them and as I won't be leading this effort, I have nothing else to offer so I'll bow out of this issue. I look forward to one day a new way to do DMs more securely.
- with the signature, the need for a mac isn't really present - message is already tamper proof, gcm is fine tho.
- versioning is key
- the dm-spec, as is, leaks no more public keys than are already leaked in the existing protocol, fixing that is a separate issue
- doing dm's "more securely" is challenging. leaking the main key, even with discarded channel keys, results in a full loss of metadata privacy. and most clients would keep channel keys for a long enough time that the additional security isn't notable. only decent way is to establish a more "direct" connection with a DM, then at least you have pure deniability. which, of course, you can use nip-4 style shared secret to establish.
- i think "considering it harmful" isn't really considering what its intended use case is: "persistent censorship resistant e2e encrypted messages". if you don't like the persistence (which can always leads to metadata leakage), don't use nostr
- any "dm" system should probably be layered on nip-4 shared secret as the initial channel creator anyway. maybe just change nip-4 wording to say "don't use this for dm's... use it at a lower layer in a secure dm system"
@ajtowns I was thinking something similar in regards to two events kinds, a request and then messages.
@lucash-dev I had a similar thought, looking for messages at an agreed upon shared secret "location".
Event Kind A:
A request for Direct Message. To prevent spam there is either a NIP-57 private payment, proof-of-work, or is a known existing pubkey that is followed. For example a Diffie-Hellman key exchange can be made for all pubkeys that are followed to be "locations" to watch for incoming messages.
Event Kind B:
After confirming the Event Kind A, messages are sent using Event Kind B. The identities of the messages are not known and encrypted. How each party knows where to find the messages is based on information exchanged in Event Kind A. The messages do not need to stay on relays until after they have been received, or in the case that both are online they can be sent directly peer-to-peer. Read receipts can be optional for confirming the delivery.
@ajtowns I was thinking something similar in regards to two events kinds, a request and then messages.
@lucash-dev I had a similar thought, looking for messages at an agreed upon shared secret "location".
Event Kind A:
A request for Direct Message. To prevent spam there is either a NIP-57 private payment, proof-of-work, or is a known existing pubkey that is followed. For example a Diffie-Hellman key exchange can be made for all pubkeys that are followed to be "locations" to watch for incoming messages.
Event Kind B:
After confirming the Event Kind A, messages are sent using Event Kind B. The identities of the messages are not known and encrypted. How each party knows where to find the messages is based on information exchanged in Event Kind A. The messages do not need to stay on relays until after they have been received, or in the case that both are online they can be sent directly peer-to-peer. Read receipts can be optional for confirming the delivery.
Precisely. I think much of the details could be just copied from existing protocols, eg based on Double Ratchet like Signal. The only thing that needs to be "invented" is the generation of the "location", which might be as trivial as using DH, hashing the result, and using that as a private key for the "author" of the new events, then with each new secret generated just using its hash as a private key.
However, I'm also intrigued by the possibility of using "private zaps" (receipts of LN payments with encrypted data) for encrypted direct messages. By the pace of development it might be ready well before a good replacement for NIP-4 is figured out.
Maybe both might be combined, with a "private zap" used as Kind A to jump start the protocol.
Yeah, Private Zaps for Kind A to jump start sounds useful.
The location could be a DH location, seems simple. Could be useful to implement to see how it might work in practice.
Those locations could also be shared among many recepients that might not be involved. AES-GSM could be used to determine which messages can be decrypted and intended for the recepient (this would have some additional CPU and bandwidth cost for stored messages at-least). Peer-to-peer would then optimize bandwidth and CPU usage. Specific relays should likely also be part of the defined location from Kind A.
Want to look more into Double Ratchet and how Signal is working with groups.
Simplest incrementally better thing would be to just take pretty much exactly NIP-4, but derive properly three keys from the DH (using prefixed hashes):
- encryption key
- "fake" author key
- "fake" recipient key
then send it as a nip-4 event. it would look like just a regular nip4 thing and relays don't need to be aware of anything different.
(might use each of the "fake" keys for one of the users, just to make it easier for clients as it is closer to nip4, though it leaks a bit more info).
clients need only know what public keys to use for DH (maybe just everybody you follow) and request them from relays.
disadvantages:
- still leaks when a specific "location" (pair of "fake" author/recipient keys) is used
- users have to always use same location for talking to same people
- no forward secrecy
- relays can still figure out which users are talking to each other by connections and requests for the location (not sure how much that could be avoided anyway).
- sort of a hack
much easier to implement than a proper private messaging protocol though. however, like I said, by the time anyone has time to do it, a usable alternative might already be available.
still leaks when a specific "location" (pair of "fake" author/recipient keys) is used
It's association would only be known to the two parties though (mostly).
users have to always use same location for talking to same people
The "location" could be random after the initial DH exchange (e.g Event Kind A).
relays can still figure out which users are talking to each other by connections and requests for the location
This is where being able to use short location queries could be useful. Let's take for example, if the full location was 3bf0c63fcb93463407af97a5e5ee64fa883d107ef9e558472c4eb9aaaefa459d. It could be possible to query it by using only the leading bytes, for example 3b. If there were too many results, 3bf0 could be used and so forth.
The "location" could be random after the initial DH exchange (e.g Event Kind A).
Yes, but that’s harder to specify and implement.
I was just trying to spell out the a solution that would require the minimum effort to implement. You could do the above client-side only, and basically just tell your client to follow some weird keys and add some transformation when displaying DMs from those keys and sending to them.