nips
nips copied to clipboard
nostr-protocol
NIP-104: Double Ratchet (End-to-End Encrypted) DMs
This NIP adds several events and details a protocol to add E2EE DMs to Nostr that use a double ratchet (based on the Signal protocol, but without the centralized servers).
Nice!
I think this idea of using a double ratchet inside GiftWraps is neat. It basically creates a second encryption layer, with different keys that are not stored inside relays and thus cannot be decrypted even if the main key leaks, while keeping the GiftWrap's routing mechanism intact (poor man's explainer for Backward and Forward Secrecy)
I'd call this PR Double Ratchet DMs just to differentiate from all the other Private, end-to-end DM PRs around here. :)
I would love to get @paulmillr's take on this.
Any effort to improve DMs is good. I didn't do the ratcheting/hierarchy of keys in NIP-44, because it's too complicated and has many non-obvious caveats.
I have been discussing this with @erskingardner directly. Have been waiting for a working demo that allows outside researchers to audit how it would work in real-world scenario.
@erskingardner does the demo contain full functionality now?
Hello, I am new and pardon me to ask stupid question, how about Pre-Shared Key? so Alice and Bob could just use the specify agreement key without something Diffie-Hellman exchange handshake. That could maybe more secure or keep secret?
this is great, just want to add, look at finding a way to do MLS as well and the misery of nostr DM will at last be over
@mleku just take a look at RFC 9420 and try to comprehend it. It's complicated beyond any reasonable means. Which means it won't be implemented simply by volunteers. Which means it's much worse. It has also been focused on group chats - a different beast.
Initial thoughts:
- This is, as mentioned in doc, scheme unique to device pairs. User won't be able to read his messages on PC when a conversation was made from a phone.
- UIs will need to accept the fact that some users would accidentally accept conversations which would need to be moved to other devices
- The move would be done with initializing a new conversation
- Are other nostr collaborators fine with the fact that there would be no synchronization? I remember some conversations about that.
- Bob sends 5 messages to Alice, then loses his connection and sends 10 more. Meanwhile Alice sends 4 messages to Bob, one of which arrives correctly "in the middle of these 10". Then Bob gets back online after 3 days. 5 of the remaining keys are sent to alice, which are created on top of Alice's message.
- What happens in this case? How would software know which keys should be used to which messages? Would it simply brute-force every combination of keys?
- How does this work with multiple relays?
- Which relays would store pre-keys?
- Are prekeys same for every relay?
- Prekey exhaustion (mentioned in doc) is a real problem in nostr. Signal has sybil protection with phones and server-side rate limits. In nostr, a bad actor would simply iterate over all users and send conv requests with all available prekeys.
Are other nostr collaborators fine with the fact that there would be no synchronization?
No, I think synchronization needs to be solved for this to become the de-facto messaging standard. But I'm ok with deferring that discussion until later.
Nice work btw @erskingardner, very excited for the prospect of NIP 17 eventually being improved upon.
Signal doesn't allow multiple devices. Their web version proxies stuff from phone, which acts as a main device.
It doesn't seem like syncing could be solved in a simple way. Whatsapp allows multiple devices, may be worth looking into.
Keep in mind that, in Nostr, DMs should work not only multiple devices, but also multiple Clients in the same device.
Signer Apps can be used to share pre-key private keys among many Clients, however, the big question on my mind is that if all clients and all devices get a copy of all these pre-keys, doesn't it break the Backward and Forward Secrecy just because the prekeys are being inserted into all the places that the main private key also is? Meaning... wouldn't all keys also leak if the private key leaks? And if so, what's the advantage of this scheme over the basic NIP-17?
It feels like we need a scheme where each app/device has a pre-key and can talk to all other pre-keys in other devices.
Glad to see this is finally happening. We attempted to solve this problem a while back but there was no engagement from anyone in the community so it fizzled out: https://github.com/nostr-protocol/nips/pull/410 closing in favor of this.
Should we also have an inbox relays event kind that allows users to specify which relays they want used? That definitely makes it more obvious that these GW events are DMs though.
This must be done. The only alternative to having this is centralization among a few big relays.
does the demo contain full functionality now?
@paulmillr Yes, it has the basic full functionality. I need to build more in the demo so I can test out of order and lost messages but in theory they should already work.
It feels like we need a scheme where each app/device has a pre-key and can talk to all other pre-keys in other devices.
@vitorpamplona the prekeys are basically per "device-client". I'm imagining that the clients themselves manage the prekey but I hadn't actually thought through the management of the private key of the prekey. You're right, that a remote signer like nsecbunker or otherwise could handle that but it's not ideal. An easy way to solve this would be to make these PRE instead of simple replaceable events. That way each device-client could manage it's own prekey. There is a can of worms there though on which prekey a user trying to connect would use for the initial conversation request event... I'll think about this a bit more.
My initial thought here was that these sorts of DMs would be single purpose client instead of a another feature in every client out there. But maybe that is too narrow a view.
Prekey exhaustion (mentioned in doc) is a real problem in nostr. Signal has sybil protection with phones and server-side rate limits. In nostr, a bad actor would simply iterate over all users and send conv requests with all available prekeys.
With signal, there are two types of prekeys, basic prekeys and one-time use prekeys. The basic prekeys are rotated on a regular basis but aren't onetime use so they can't be exhausted. The addition of a one-time prekey does give you better replay attack protection but I chose to leave that out to keep things more manageable for now.
How does this work with multiple relays?
I'm not sure what this means, it doesn't matter how many or which relays you publish to, as long as the other user shares some relays (or they use outbox) then they shuold be able to find you.
Bob sends 5 messages to Alice, then loses his connection and sends 10 more. Meanwhile Alice sends 4 messages to Bob, one of which arrives correctly "in the middle of these 10". Then Bob gets back online after 3 days. 5 of the remaining keys are sent to alice, which are created on top of Alice's message.
@paulmillr The clients aren't syncing keys with each other they're always deriving them all from the initial shared root key (secret key). And bob can't "send" message while offline. If the client can't publish the event to a relay it should fail immediately so bob knows that those messages weren't ever sent. retrying at a later date with those messages would have to create a new 444 event encrypted off the latest keys in the chain instead of trying to just republish.
For generally missed messages (either they arrive out of order or anything else) the incoming messages give the pubkey (root key), the message number, and the previous chain's length of messages. The receiving client can use that to ratchet through the right chains to create and store the message keys for use later when (hopefully) the missing messages arrive. If they never arrive, the client could show a message to the user "missing message" if it wanted to.
This must be done. The only alternative to having this is centralization among a few big relays.
@fiatjaf what about the fact that it's more obvious to people watching that you're receiving DMs? I guess it's still speculation but it is a degree more information about what's going on than a bystander would otherwise have...
My initial thought here was that these sorts of DMs would be single purpose client instead of a another feature in every client out there. But maybe that is too narrow a view.
A possible solution could use time-window keys and sharing those keys with all of the other clients: Build another ratchet that creates a new pre-key every month. You can share that prekey with other clients, but never the main ratchet key. Meaning, if the inner key leaks, it leaks only a window of time. Within that time, all clients can encrypt and decrypt. Users would need to periodically reload all clients with new keys.
Hello, I am new and pardon me to ask stupid question, how about Pre-Shared Key? so Alice and Bob could just use the specify agreement key without something Diffie-Hellman exchange handshake. That could maybe more secure or keep secret?
DH is the most secure way to agree on a shared secret value and it also authenticates the user's to each other in the process.
@fiatjaf what about the fact that it's more obvious to people watching that you're receiving DMs? I guess it's still speculation but it is a degree more information about what's going on than a bystander would otherwise have...
People can select relays that will hide events targeting you from non-AUTHed users, same as in NIP-17. For that reason it is better to allow people to select specialized DM relays other than their default NIP-65 relays.
Great work but unfortunatelly as other people said, there is the lack of multi-device support problem, which is a noticeable downgrade from NIP-04. The secrecy features of this NIP rely in part on keeping extra privkey state stored on a single device. It is unclear how it would be possible to securely add multi-device support later, considering the text says it is beyond the NIP scope.
I think the threat of total device compromise is far greater than the threat of a cryptographic key being derived via cryptanalysis. In that much more common scenario, whatever your device can do the attacker can do. If you can read historic messages, so can eve. If you can't, well that's quite a usability shortcoming.
I'm not against this double ratchet NIP. I think this is very cool. I just want to (if I'm right) make people aware of how limited of an improvement all this complexity is providing. It ticks a marketing box and limits the damage caused by certain kinds of remote cryptographic breaks. But that threat was never as great IMHO as device compromise is.
Your devices have ~hundreds of~ a dozen ways to break into them right now, most are not publicly known. I spent too many years in that field to have any hope left that devices are sometimes secure (It led me to rust, and towards writing a RISC-V os in rust, ... but then I found nostr).
@mikedilger This is the tradeoff that ALL e2ee messaging services have accepted. They encrypt the conversation data and all the key material in the most secure way possible on the device (secure enclave, etc. etc.) but fundamentally, that's a tradeoff that this method takes onboard (again, Signal, Telegram, Whatsapp, etc. all use the exact same model).
It's also important to be clear. Device compromise breaks ALL forms of DM encryption that we could come up with. If your device is truly compromised then there is no line of defense for any keys that touch the device. So I'm not really sure the point you're making here.
People can select relays that will hide events targeting you from non-AUTHed users, same as in NIP-17. For that reason it is better to allow people to select specialized DM relays other than their default NIP-65 relays.
Good point. I do make a note about using only relays that support AUTH but I can make that more clear here. Will update.
So I'm not really sure the point you're making here.
First, I think we should do the best we can, even with all the complexity it adds.
My point is just that we should be careful in our communications. If we ever say "this is the maximum security possible" in our pride, we should also be very clear that device security matters more and that we can't do anything about that.
I don't want non-techies thinking this is super secure way to do DMs and then communicating about really dangerous topics, and then getting executed by their state because their devices weren't secure.
I agree @mikedilger - it's really important that people don't think this is some sort of panacea solution that makes their comms perfect.
I agree @mikedilger - it's really important that people don't think this is some sort of panacea solution that makes their comms perfect.
the model of relays introduces a trust problem, in herently, and at minimum, a signals intelligence metadata leak problem
any discussion about "what we can do" on this protocol should omit any pointless handwringing about the fact that it is inherently not trustless
if you want a trustless relay network, you need to create a trustless way to pay for relay service, and you need to enable clients to sync state with each other... these are the two problems that a trustless p2p network faces, and any discussion about what nostr can do that pecks at the problem of what it inherently can't do is a total waste of time
@erskingardner what is the threat model you’re trying to work in, with this improvement? Is it different than the previous one?
@erskingardner what is the threat model you’re trying to work in, with this improvement? Is it different than the previous one?
The user I have had in mind since starting this work is activists in hostile jurisdictions. People that are working under authoritarian rule and need to have secure communications that can't just be blocked by the regime in power.
The goal of this NIP is to create a scheme for secure direct messages between two individuals on Nostr that does three main things:
- Makes it much harder for an attacker to access the content of messages; focused mostly improving on the into-the-future and back-into-the-past guarantees that are lacking in other NIPs (e.g. NIP-17). I think the encryption scheme of NIP-44 does a great job with the actual encryption. But having the double ratchets completes the puzzle and makes conversations both forward and post-compromise secure.
- Improve the deniability significantly for all conversations. This is actually also covered by NIP-17, given the way GW's work. I think there are a few further tweaks that we could make (publishing ephemeral keys potentially) to further improve deniability but this is a good start.
- I believe having Signal-level encryption + forward & post-compromise security + metadata obfuscation + deniability coupled with a huge number of potential relays is actually a HUGE benefit over centralized services like Signal. If the US govt. decided tomorrow that Signal is a terrorist org they could force them to shut down their servers and hence Signal would stop working. It would be vastly harder to turn off access to double ratchet e2ee DMs on Nostr given the distributed nature of the "servers". It would be an un-winnable game of whack-a-mole.
The tradeoffs inherent here are: (if anyone has others, please speak up)
- IP address leakage. This is a general problem with the internet. There are solutions, if people care to use them.
- Observers are able to see GW events being sent to you. In theory, this is opens you up to some degree of timing correlation analysis (from attackers and/or relays). That said, DM messages aren't the only type of event that is gift wrapped and I expect more kinds to be GWed in the future. This can also be easily obfuscated with dummy GW events.
- Device compromise likely means total compromise. Securing decrypted messages, unused keys (to decrypt out of order messages), and destroying DH + ephemeral key data isn't a trivial task for client developers. There are many ways that a client dev could screw up the implementation and totally ruin the privacy benefits gained in this spec. Even if it's implemented perfectly, the conversation itself has to be stored on the device. All that said, I believe this point is pretty much true for ANY encryption/messaging setup. If your phone or computer is compromised I don't think Signal would stand up to much either. Again, maybe I'm wrong here?
- Speaking to @mleku's point about relays: this is why the spec is clear about "inbox" relays and leaves that choice to the uesrs. User's have to decide which relays they want these messages sent to. We're trying to minimize trust, there's no way to completely get rid of it.
I do think that @mikedilger's point is fair. The chances of device compromise is likely the largest risk that any of us face. It's also the one that is most understandable to a nontechnical user). Having as unimpeachable as possible encryption and metadata protection coupled with an unstoppable network goes a long way towards giving people piece on mind on the things that they can't control and leaves them to worry about the things they can control (their own security and device practices).
Thanks, that’s thoughtful.
What do you think about incorporating ML-KEM aka Kyber to gain additional pq security? I wanted to do it as v3 of nip44 at some point.
Signal already added it.
The user I have had in mind since starting this work is activists in hostile jurisdictions
Not to discourage this PR, but I fail to see how this PR improves the situation for that specific user. I initially thought this was for other types of users.
As described before, this PR only provides benefits IF Ratchet keys are kept separate from the private key. If they are kept together, in the same client (both leak at the same time), this PR provides exactly the same privacy and security as NIP-17.
I think we all agree that NIP-44 encryption is nearly unbreakable, so there are two main threats for activists:
- Physical: The attacker has confiscated their phone. This is probably the most common threat.
- Virtual: The attacker found a bug in the client OR found ways to capture the private key (and ratchet information) from it.
The use of ratchets by themselves doesn't solve either 1 or 2.
But, if this PR could force clients to use security chips to store the ratchet information in such a way that no one can take them out of the phone/browser, then it would solve it. But that is a big IF that is barely mentioned in the PR.
However, if that way of securing secrets can be coded, the same privacy gains could be achieved with burner Nostr accounts that use similar inability to export protections. And it would be a simpler implementation.
Improve the deniability significantly for all conversations.
This is also very unclear to me. If the previous point is not solved, there is no additional deniability provided. If private keys and rachet info can be accessed by attackers 1 or 2, then this is exactly like NIP-17, but more complex to code.
After all, GiftWraps with the user's main pubkey are still flying back and forth. NIP-17 key aliases could solve that, but we don't really explore that option neither here, not in NIP-17 (because it also needs its own little protocol to let friends know which keys to message).
But again, even Key aliases are just the same thing as a rotating Nostr burner account. Their gains are small considering the alternatives.
So, unless I am missing something, the use of rachets alone does not address the problem you are trying to solve. It's how you manage those rachet states, or other crypto primitives, that might solve it. But if that is the problem, then we might not need ratchets at all.
--
Now to the tradeoffs, I actually think they are the biggest problem we face today.
- IP address leakage & Relay tracking.
This can be solved with either Tor or our own GiftWrap routing network using DVMs to forward GiftWraps to the next relay/DVM until it gets to the user. I like building our own GiftWrap routing network better just because it solves privacy (of transferring a GiftWrap with the user's main key listed as p-tag) and IP tracking at the same time.
- Observers are able to see GW events being sent to you.
This is a problem that we tried to solve with key aliases on the GiftWrap. It works, but we need a protocol to create a priority inbox where users can send giftwraps to and other users would be listening. The protocol can rotate keys constantly, with a ratchet, or just randomly. It's not difficult, it's just more work.
- Device compromise likely means total compromise.
If by "compromised" we mean confiscated, I do think the ratchet here is a good solution, provided that we can keep the Ratchet's key away from people who have physical access to the phone.
If by "compromised" we mean some code is running in the phone or desktop to read these messages, there is literally nothing we can do at the NIP level. Clients can make things harder for non-root agents, but it is up to each client to code these things correctly.