[Tracking issue] End-to-End encrypted Broadcast Channels

[Hocuri]

Here I will collect all information around implementing broadcast channels, similar to the ones known from Telegram/WhatsApp. As oppsosed to Telegram/WhatsApp, channels will be end-to-end encrypted in Delta Chat. Some other features known from Telegram/WhatsApp will be missing.

Implementation

Show broadcast channels in their own, proper "Channel" chat

I checked all checkboxes which I implemented in a PR, even if it's not merged yet.

Core work (https://github.com/chatmail/core/pull/6901):

• [x] Add new chat type InBroadcastChannel and OutBroadcastChannel for incoming / outgoing channels, where the former is similar to a Mailinglist and the latter is similar to a Broadcast (which is removed)
  • Consideration for naming: InChannel/OutChannel (without "broadcast") would be shorter, but less greppable because we already have a lot of occurences of channel in the code. Consistently calling them BcChannel/bc_channel in the code would be both short and greppable, but a bit arcane when reading it at first. Opinions are welcome; if I hear none, I'll keep with BroadcastChannel.
• [x] api: Add create_broadcast_channel(), deprecate create_broadcast_list() (or create_channel() / create_bc_channel() if we decide to switch)
  • Adjust code comments to match the new behavior.
• [x] ASK Desktop developers what they use is_broadcast field for, and whether it should be true for both outgoing & incoming channels (or look it up myself)
  • I added is_out_broadcast_channel, and deprecated is_broadcast, for now
• [x] When the user changes the broadcast channel name, immediately show this change on receiving devices
• [x] Allow to change brodacast channel avatar, and immediately apply it on the receiving device
• [x] Make it possible to block InBroadcastChannel
• [x] Make it possible to set the avatar of an OutgoingChannel, and apply it on the receiving side
• [x] DECIDE whether we still want to use the broadcast icon as the default icon or whether we want to use the letter-in-a-circle
  • We decided to use the letter-in-a-circle for now, because it's easier to implement, and I need to stay in the time plan
• [x] chat.rs: Return an error if the user tries to modify a InBroadcastChannel
• [x] Add automated regression tests
• [ ] Grep for broadcast and see whether there is any other work I need to do
• [x] Bug: Don't show ~ in front of the sender's same in broadcast listsr

UI work (https://github.com/deltachat/deltachat-android/pull/3783):

• [x] Rename "broadcast list" to "channel" or "broadcast channel" both in UI and code (depending on whether it's important to be concise in the particular place). There are new stock strings in which broadcast_list is replaced with channel
• [ ] Adapt to the new chat types InBroadcastChannel and OutBroadcastChannel for incoming / outgoing channels, where the former is similar to a Mailinglist and the latter is similar to a Broadcast (which was removed)
• [ ] Adapt to the new create_broadcast_channel() API

UI Testing:

• Creating a channel
• Adding/removing members from a channel
• Set a channel avatar, and check that receivers see it after sending a message
• Change the channel name, and check that receivers see it after sending a message
• On the receiving side of a channel, check that it's not possible to send a message
• On the receiving side of a channel, check that it's possible to block the chat
• On the sending side of a channel, check that it's possible to add and remove members, and that the shown texts are correct (e.g. "Remove X from Channel?" instead of "Remove X from Chat?")
• Both on the receiving and sending side of a channel:
  • In that chat, open the 3-dots menu, and check that things look fine
  • In the chat, check that the subtitle is correct
  • Open the profile page for a channel, and check that things look fine
  • Check that ephemeral messages actually disappear

Symmetrically encrypt broadcast channels

Channels need to be symmetrically encrypted, because if we encrypted them asymetrically, like we do for all other messages:

• with some technical expertise, it would be possible to find out who the other recipients are
• when there are a lot of recipients, the messages would be very big.

This will need a lot of work (like new QR codes) which I'll write about later.

• [x] Talk to some PGP experts that the general idea is fine

[iequidoo]

• Consideration for naming

Maybe {In,Out}Broadcast? If we remove the current Broadcast lists, these don't conflict with anything. BroadcastChannel is quite long and will cause extra line wrapping. OTOH just Channel would be ambiguous.

• chat.rs: Return an error if the user tries to modify a ReadonlyChannel

I think it should be possible to rename incoming broadcast channels the same way it's possible for contacts. Btw, how is ReadonlyChannel different from an incoming one?

Channels need to be symmetrically encrypted

They can be asymmetrically encrypted, but the private key should be shared across contacts. Anyway, messages should be signed with the sender's key (before encryption is done). I don't suggest anything right now, but maybe asymmetric encryption fits better into PGP or at least the current code. Can you provide any reference where the symmetric encryption scheme is described?

[Hocuri]

Maybe {In,Out}Broadcast? If we remove the current Broadcast lists, these don't conflict with anything. BroadcastChannel is quite long and will cause extra line wrapping. OTOH just Channel would be ambiguous.

Not sure - then, the name in the code and the name in the UI don't have anything to do with each other. OTOH, it's true that BroadcastChannel is quite long

I think it should be possible to rename incoming broadcast channels the same way it's possible for contacts.

I would assume that this leads to user confusion, also since it's not possible to rename channels in other messengers. But anyways, in the first iteration I will try and keep things simple, which means leaving away non-essential features like renaming of incoming channels

Btw, how is ReadonlyChannel different from an incoming one?

Not at all, ReadonlyChannel is just another name which I used in a previous version of my TODO list. I renamed it now.

Can you provide any reference where the symmetric encryption scheme is described?

https://www.rfc-editor.org/rfc/rfc9580.html#seipd, encrypt_device_token() at https://github.com/chatmail/core/blob/main/src/push.rs#L76

[iequidoo]

Not sure - then, the name in the code and the name in the UI don't have anything to do with each other. OTOH, it's true that BroadcastChannel is quite long

"Bcast" is the common shortening for "broadcast". But IMO the most important thing is to have similar naming in the core and UI's code, how things are translated in the UI is secondary, maybe some languages don't have obvious translation for "broadcast" at all.

https://www.rfc-editor.org/rfc/rfc9580.html#seipd, encrypt_device_token() at https://github.com/chatmail/core/blob/main/src/push.rs#L76

Well, encrypt_device_token() encrypts to a public encryption subkey (i.e. asymmetrically) and doesn't sign the message. Would be nice to know how broadcast messages are authenticated, but if you'll write about that later, i can wait

[Hocuri]

I meant this:

    let mut msg = pgp::composed::MessageBuilder::from_bytes("", padded_device_token).seipd_v2(
        &mut rng,
        SymmetricKeyAlgorithm::AES128,
        AeadAlgorithm::Ocb,
        ChunkSize::C8KiB,
    );

And then, there is the function symm_encrypt():

        let mut rng = thread_rng();
        let s2k = StringToKey::new_default(&mut rng);
        let builder = MessageBuilder::from_bytes("", plain);
        let mut builder = builder.seipd_v1(&mut rng, SYMMETRIC_KEY_ALGORITHM);
        builder.encrypt_with_password(s2k, &passphrase)?;

        let encoded_msg = builder.to_armored_string(&mut rng, Default::default())?;

which actually symmetrically encrypts, but with seipd_v1, which is worse than seipd_v2 IIUC. So, I will probably do the same as symm_encrypt(), but using seipd_v2, but I will talk to a PGP expert first.

[iequidoo]

I guess using seipd_v2 with encryption to a password is possible, but note that the message must be signed with the sender's key first. Signing isn't needed in case of Autocrypt Setup Message because the password isn't shared. But my concerns are that overall such a scheme isn't standardized by OpenPGP and it would require more coding and more difficult audits while creation a shared asymmetric key (even w/o uid) is a bit more computationally expensive, but allows reusing the existing code. Also i'm not sure that with symmetric encryption broadcast messages won't be distinguishable from group messages for passive eavesdroppers.

EDIT: Took a look at the code, indeed, it looks possible to add another version of pk_encrypt() which uses seipd_v2() and encrypt_with_password() instead and that should be fine according to OpenPGP. As for protecting the message type (group/broadcast) from passive eavesdroppers, this doesn't look technically feasible anyway, particularly for large broadcast channels because we don't want the message size to grow with the number of recipients.

[btcbobby]

Is this a place I can add a broadcast channel feature request?

1. I like the "Reply Privately" for feedback but it is often not easily found in the UI.
2. Reaction emoji's back to the owner of the list would be a quick way to provide feedback more than just the read receipt.

[iequidoo]

"Reply Privately" indeed should work the same way it works for groups, i think it doesn't even need additional development. Reactions (if only shown to the broadcast owner and the reaction sender) would need some efforts, but not a lot and mostly in Core.

[Hocuri]

There are two decisions I have to make for symmetric encryption:

1. How can recipients be added? (1a) Should we continue to support the UI flow of the channel owner adding a recipient (1b) Or should it only be possible to join a channel via a QR code?
2. How to propagate the shared secret? (2a) The QR code works like today's group-invite QR codes, just that the inviter adds you to a channel in the end, sending the channel-shared-secret in an asymetrically encrypted message (2b) The QR code directly contains the shared secret used to encrypt and decrypt messages

If I go for option 1b, then I can't go for option 2a.

~~I chose to pursue option 1b and 2b for now.~~ I chose to pursue option 2a for now, see my comment below

Thoughts 1. How can recipients be added?

There are surprisingly many problems with the owner adding members to broadcast channels:

• If the member-addition message gets lost, the member won't be able to receive any messages, because they don't have the shared secret. This problem could be ignored for now, but to make this really nice, the invited member has to send a confirmation message when the user clicks "Accept" on the contact request, and the channel owner's device shows them as a member only then (not sure about the exact details, but anyways nothing for now to discuss).
• Implementation is not straightforward, because the member-added/removed message has to be sent to only self and this one member, and because we don't really have a header for this: We have ChatGroupMember{Added|Removed}, but it only contains the email address (rather than the fingerprint), and today, it's only used for showing info messages. We have ChatGroupMemberTimestamps and ChatGroupMemberFpr, which are overkill, and which would reveal the other member's fingerprints. The possible solutions for this are:
  • send a separate sync message to self, and send the member-added message only to the new member, but still, it has to be done.
  • Put the fingerprint into ChatGroupMember{Added|Removed}, rather than the address (as we are doing today). Not the best option because it will probably lead to confusion.
  • Add a new header ChatGroupMember{Added|Removed}Fpr
• Again on the implementation side: When promoting a channel, all we have to send member addition messages to all members. Alternatively, in order to introduce less complexity and make the implementation easier, the channel could be promoted right after creation (rather than when sending a message, as for groups). Not sure if this discrepancy between channels and groups is a usability problem; OTOH, this is the behavior users know from other messengers, who don't have promotion. [^1]

Still, all of these problem can be solved or postponed. With Chatmail, messages usually don't get lost, implementation would be possible as far as I can judge right now, and we can deem the added code complexity worth it.

I'm not sure how much users expect to be able to directly add recipients to their channels, rather than sending them QR invite codes; there could be UI for easily sending a QR invite code (which would need adaptions in all UIs), or prospective new members could be sent an invite code (which could be done purely in core).

Thoughts on 2. How to propagate the shared secret?

Pro (2a) send a member-added message:

• Same code path for group-invite and channel-invite, which is better for security
• The shared secret is not exposed in cleartext -> If a purely passive attacker gets access to both the email server and the QR code, then they can't read messages (although this is a very specific attach scenario).
• ~~It's possible to withdraw a QR code~~ Actually, it is equally possible to withdraw QR codes with option 2b

Pro (2b) include the shared secret in the QR code:

• More quantumn-resistant: If the QR code is transferred out-of-band, then all encryption is symmetric, i.e. resistant against quantumn computer attacks.
• If I want to choose option 1b, then I also have to choose this option.

Decision

After some online&offline discussions, I now decided that I will go for the 1b and 2b option, because it's simpler, and from a usability perspective it's not even clear whether it's good if someone can add you to a channel without your consent. People will just have to send a link in order to add someone to a channel; we might even decide to show invitation codes in a nice way in the UI, which would also be a UX advantage for groups invite links.

[^1]: After creation, a group is in unpromoted state. This means, you can add or remove members, change the name, the group image and so on without messages being sent to all group members. This changes as soon as the first message is sent to the group members and the group becomes promoted. After that, all changes are synced with all group members by sending status message.

[iequidoo]

Thoughts on 2. How to propagate the shared secret?

A third option is to revert the protocol and say that the broadcast QR is already a vg-request message. Then:

• Broadcast Owner's fingerprint isn't needed in the QR and the QR AUTH can be bigger.
• Subscriber sends vg-auth-required auth-encrypted with the AUTH. It contains the INVITENUMBER + Subscriber's Autocrypt key as usually.
• Owner sends vg-request-with-auth Autocrypt-encrypted to Subscriber's key. It contains AUTH + the actual symmetric key + Owner's Autocrypt key.
• Subscriber sends vg-member-added Autocrypt-encrypted to Owner's key which in fact confirms receipt of the keys.

Anyway at least 3-step protocol is needed so that Subscriber confirms receipt of Owner's Autocrypt key (if we don't put it into QR).

[Hocuri]

In reply to my previous comment:

After a discussion with @hpk42, I now tend to change the approach and go for option 2a:

• This way, the secrets won't be in clear-text in the QR code. Otherwise, a server operator could just scrape the web for QR codes and try to decrypt as many messages as possible; with option 2a,
• We may want to switch the normal securejoin to use symmetric encryption. With option 2a, we will already have most of the code that is necessary for this.
• QR codes will be smaller, because they only need to contain one symmetric secret and no additional AUTH token.
• It will be possible to completely withdraw QR codes; with option 2b, the QR code wouldn't have worked anymore, but would still have contained the secret
• It will be easier to rotate the secret in the future when removing a member
• can be made quantumn-resistant by additionally encrypting the member-addition message with the symmetric key from the QR code, or by only encrypting the member-addition message with the symmetric key from the QR code.

In the UI, I still only want to make it possible to add a member via a QR code, at least for now. This way, there is less risk of being added to channels you don't want to be in (sure, you can be added to groups, but it's an additional potential annoyance), and if the member-added message gets lost, it's more straightforward for the user how to solve the problem (just scan the code / click the link again, rather than removing and adding the member).

About the mentioned implementation problem:

Implementation is not straightforward, because the member-added/removed message has to be sent to only self and this one member, and because we don't really have a header for this

It turns out that I would have to solve this in some way, anyways; I will talk to @link2xt what's the best way forward.

I actually have a TODO for this in https://github.com/chatmail/core/pull/7042/:

        // TODO this may lookup the wrong contact if multiple contacts have the same email addr.
        // We can send sync messages instead,
        // lookup the fingerprint by gossip header (like it's done for groups right now)
        // or add a header ChatGroupMemberAddedFpr.

[Hocuri]

Benchmarking

Decryption is potentially slow because we need to try out a lot of secrets, so, I wrote some benchmarks:

1. Switching the string2key algorithm from the default (IteratedAndSalted) to Salted saved a lot.

Before, with 20 secrets: Decrypt/Decrypt symmetrically encrypted
time: [300.08 ms 428.44 ms 553.47 ms] Decrypt/Decrypt pk encrypted
time: [115.78 µs 116.28 µs 117.12 µs]

After, with 500 secrets: Decrypt/Decrypt symmetrically encrypted
time: [334.13 µs 763.25 µs 1.2905 ms] Decrypt/Decrypt pk encrypted
time: [162.34 µs 162.65 µs 163.14 µs]

You can also see here that, even with 500 secrets, both public-key and symmetric-secret decryption is still reasonably fast.

2. Loading and trying out the symmetric secrets doesn't make receiving pk-encrypted messages measurably slower.

(for 50 secrets) Before the first run, replaced the loading of shared secrets with vec![]. Before the second run, I reverted this change. You can see that there is hardly any change in performance:

Decrypt/Receive a public-key encrypted message
time: [9.9283 ms 9.9433 ms 9.9556 ms] change: [-0.6166% -0.4294% -0.2464%] (p = 0.00 < 0.05) Change within noise threshold.

3. Even with 500 secrets, symmetric-secret decryption now makes the whole process of parsing a message only 11% slower.

1 secret vs 500 secrets: Decrypt/Receive a symmetrically encrypted message
time: [8.7576 ms 8.8209 ms 8.8771 ms] change: [+9.3010% +11.292% +13.013%] (p = 0.00 < 0.05) Performance has regressed.

[Hocuri]

The technical design

I moved this post to the PR description of https://github.com/chatmail/core/pull/7042, because the issue here is getting long, and posts are getting hard to find.

[iequidoo]

FINGERPRINT is the OpenPGP fingerprint of the channel owner, which will allow the Bob to verify that the messages come from the correct sender, and that there is no MitM attack.

Owner's fingerprint in the QR is needed because we want to support multi-use/-scan QRs AFAIU. For one-use QRs a MITM isn't possible. Just for the record.

Also you suggest a 2-step protocol, but it doesn't solve the problem that you mentioned:

• If the member-addition message gets lost, the member won't be able to receive any messages, because they don't have the shared secret. This problem could be ignored for now, but to make this really nice, the invited member has to send a confirmation message when the user clicks "Accept" on the contact request, and the channel owner's device shows them as a member only then

If we want to redesign the protocol, better to add this 3rd step to avoid the mentioned problem and other problems in the future. Now we have a 4-step protocol, so this will be an improvement anyway.

[Hocuri]

Also you suggest a 2-step protocol, but it doesn't solve the problem that you mentioned:

If the member-addition message gets lost, the member won't be able to receive any messages [...]

The 2-step protocol is completely independent of the addition-message-lost problem. The 2-step protocol has these advantages over the old 4-step protocol:

• Less complexity
• Faster (fewer messages need to be exchanged)
• Slightly smaller QR codes (no need for a separate INVITE token)
• No unencrypted messages at all

If the 2-step protocol turns out to work fine for broadcast channels, we can also use them for group and 1:1 invites.

The addition-message-lost problem is mitigated (tough not solved) by this:

In the UI, I still only want to make it possible to add a member via a QR code, at least for now. If the member-added message gets lost, it's more straightforward for the user how to solve the problem (just scan the code / click the link again, rather than removing and adding the member).

to make this really nice, the invited member has to send a confirmation message when the user clicks "Accept" on the contact request, and the channel owner's device shows them as a member only then

If we want to redesign the protocol, better to add this 3rd step to avoid the mentioned problem and other problems in the future. Now we have a 4-step protocol, so this will be an improvement anyway.

Adding a confirmation step to the end of the securejoin protocol is independent of the rest of the protocol, and does not need to happen at the same time as switching to v2. However, I'm hesitant to actually do this now, because there are a lot of lower-hanging fruits to solve first.

[iequidoo]

2. In the encrypted part, this message contains Bob's own fingerprint Bob_FP in the header Secure-Join-Fingerprint.

So, even if we put Subscriber's (Bob's) Autocrypt key into the encrypted part (we have Config::ProtectAutocrypt for this), subscribers are able to know about each other in case of multi-use QR codes because they share AUTH. This is fine for groups, but i'd say not for broadcasts and contact requests. So either QR codes should be single-use in these cases or (in case of broadcasts) Subscriber should generate a separate asymmetric key (which makes the protocol not quantum-resistant again, but maybe that's acceptable, at least we protect from external attackers), or it should be at least documented that Owner is responsible for not sharing the same QR code with potential passive eavesdroppers. Pointing to this because initially it was stated that

Channels need to be symmetrically encrypted, because if we encrypted them asymetrically, like we do for all other messages:

• with some technical expertise, it would be possible to find out who the other recipients are

[Hocuri]

Bob_FP is in the encrypted part of a message Bob sends to Alice; the other recipients never see this message. If one of the other recipients has access to Bob's or Alice's server and has seen the same QR code as Bob, then yes, they can find out Bob's cryptographic identity. But I think that this is a very rare scenario, and also, if an attacker has access to the server, they can just track who receives messages from Alice; still, I will write it on my list of potential problems, and I will write regenerating QR codes every time a QR code is shown on my list of possible improvements.

With asymmetric encryption, subscribers would only need to look at the message they received, and would find out who the other subscribers are (because they can see whom the message was encrypted to). So, the symmetric encryption still is a very big step forward in this regard.

[iequidoo]

1. In a symmetrically encrypted message, it's not visible which secret was used to encrypt without trying out all secrets. If this does turn out to be too slow in the future

Probably we should just try keys starting from recently used ones. This should be fast enough most of the time. And if the user is subscribed to too many broadcasts so it's indeed slow to try all keys, we should consider subscribing using different transports (which can be just aliases).

2. A DOS attacker could send a message with a lot of encrypted session keys

Does Delta Chat ever need to create messages with multiple session keys?

Adding a confirmation step to the end of the securejoin protocol is independent of the rest of the protocol, and does not need to happen at the same time as switching to v2.

While the confirmation step may look independent of the protocol, it's actually needed in all practical cases. SecureJoin v1's vc-contact-confirm/vg-member-added are actually confirmation steps and v2 suggests not to add anything instead. Even vg-member-added, while carrying key gossips, can be lost w/o significant harm -- the new member continues to receive decryptable messages and when re-gossiping occurs, can also send to the group, so the protocol is recoverable. v2 will be irrecoverable in the sense that side effects remain ("hung" subscriber, undecryptable messages eating traffic), QR code re-scan won't always happen:

• Users often just scan QR codes with camera, they don't take a photo that can be rescanned.
• Some users subscribe to many channels w/o actually checking the result. Then they don't follow particular channels, but just use global search to look for information in all channels, even i do so.

[Hocuri]

Probably we should just try keys starting from recently used ones. This should be fast enough most of the time. And if the user is subscribed to too many broadcasts so it's indeed slow to try all keys, we should consider subscribing using different transports (which can be just aliases).

As you can see in the benchmark results I posted above, even trying 500 keys for decryption is fast enough. So, for now, there is no need to optimize by sorting by recently used or subscribing using an alias.

Does Delta Chat ever need to create messages with multiple session keys?

No, that's why with this PR, Delta Chat doesn't attempt to decrypt such messages.

About the confirmation step: Lost messages are very, very rare, and with the current concept, even if the message with the key is lost, nothing terrible will happen. I think it's good enough for now; we can see if this turns out to be a problem in the real world, and reiterate if yes. Nevertheless, I wrote this on the "Wishlist" for channels.

[Hocuri]

Channels are now released in version 2.26.0 :tada:!