Open community, send messages

[D4nte]

As a user, I want to be able to send messages in the chat.

DoD

• [ ] Implementation to send and receive messages for a Status Community
  • [x] Send and receive messages in a public chat.
  • [x] Helper to import a Community and chat from Status (e.g using invite link) (pending on https://github.com/status-im/status-react/issues/12548 or https://github.com/status-im/status-desktop/issues/3410)
  • [x] Being able to set type (image, etc) for messages
  • [ ] Have the messenger handle the retrieving of previous messages in the background
  • [ ] Provide API that returns all seen messages (received both over store and relay)
  • [ ] Test against Status Community created with Status App.

[D4nte]

Starting to look at protobuf:

1. Is an "open community" one with NO_MEMBERSHIP?
2. In ChatMessage it is mentioned that timestamp is not used, is it still correct?
3. In what unit is ChatMessage.clock? I assume milliseconds (same than timestmamp).
4. What is the easiest way to verify interoperability of this SDK with status-go. Is there an API to "create a community", "send a message", etc?
5. Similar to 4, is there a way to enable Waku v2 in either Status app.
6. On ChatMessage, for open community, are message types COMMUNITY_CHAT or PUBLIC_GROUP?
7. On ChatMessage, for open community, what is the chatId? Same than for public chats? Looks slightly different by pre-pending the community id. Is the community id in hex format (pub key I presume)?
8. On ChatMessage, for open community, what is the community value? Community pubkey (byte array)?

@cammellos

[cammellos]

@D4nte

1. Yes
2. I don't believe we use it, correct, though is best to set it, just in case for backward compatibility (also it relies on waku 1 usage, I am not sure waku 2 has an equivalent)
3. Yes
4. Yes, the tests should give you an example of usage https://github.com/status-im/status-go/blob/06d4163de873df1a135044ba570741017880a8d0/protocol/communities_messenger_test.go#L183
5. That's based on the fleet, if you change the fleet there's should be a waku 2 fleet in there @siphiuel can add more details
6. COMMUNITY_CHAT

Is the community id in hex format (pub key I presume)?

Yes, compressed key in hex format with 0x prefix

Same than for public chats?

Not sure I understand what you mean exactly, it's the same as basically any other chat (the chat id), though the format as you pointed out is different (community-id + random uuid)

8. That's only used to share communities, it's the encoded protobuf of the community description, but it's not needed for you at this stage likely (only if you want to share a community with another user)

Let me know if something is not clear or I misunderstood anything

[D4nte]

@cammellos

9. Is the topic for a community chat following the same spec https://specs.status.im/spec/10#public-chats where it is based from the chat name? Is the chat name anything that the community admin would set or does it follow some rule (e.g. prefix with community name). ~I assume the chat name and chat id are different.~ chat id is used to calculate topic

[D4nte]

@cammellos

Looking at ChatMessage: https://github.com/status-im/status-go/blob/06d4163de873df1a135044ba570741017880a8d0/protocol/protobuf/chat_message.proto#L58

I do not see any reference to the sender. How are senders identified?

Are sender only identified via the signature done on the Waku envelope?

Is it compatible with Waku v2 envelope? https://rfc.vac.dev/spec/26/

Edit: Also, I understand messages are encrypted with a symmetric key derived from the chat id, can you please clarify what is the key? I have difficulties to understand this from the go code.

Edit: I can see it adds a Bundle https://github.com/status-im/status-go/blob/b2d8f7f416a276f1e03282cd95cf43b187e12c5f/protocol/encryption/protocol_message.pb.go#L417

But I can also see that this ProtocolMessage protobuf is defined inline and not in a proto file. Is ProtocolMessage the payload of the waku message (to be encrypted and signed)?

[cammellos]

@D4nte

There are 2 signatures:

One is at the "application level". The message is wrapped in https://github.com/status-im/status-go/blob/06d4163de873df1a135044ba570741017880a8d0/protocol/protobuf/application_metadata_message.proto#L6 and signed. https://specs.status.im/spec/6#payload-wrapper This signature is the one we consider when assigning an author to a message.

The second is at the transport level (i.e waku), and it's the signature of the "relayer" of the message (not necessarily the original sender). At this level is also the encryption

The relevant code is here: https://github.com/status-im/status-go/blob/a6e7ff6ddde4819c091a2cd73899214df9c12b58/protocol/transport/filters_manager.go#L534

So, first we derive a topic from the chatID, then we pass the chatID to waku for generating a "filter" https://github.com/status-im/status-go/blob/a6e7ff6ddde4819c091a2cd73899214df9c12b58/waku/waku.go#L916 which basically generates a sym key (algorithm is in the code above), which will be used to encrypt/decrypt messages.

Edit: I can see it adds a Bundle https://github.com/status-im/status-go/blob/b2d8f7f416a276f1e03282cd95cf43b187e12c5f/protocol/encryption/protocol_message.pb.go#L417 But I can also see that this ProtocolMessage protobuf is defined inline and not in a proto file. Is ProtocolMessage the payload of the waku message (to be encrypted and signed)?

This is mostly relevant only for one-to-one, it's for the double ratchet encryption, which is probably not what you want

Let me know if anything is unclear or I should explain some things further

[D4nte]

So, first we derive a topic from the chatID

I understand that this is only relevant to then create a Waku v1 filter. right? Not really for the encryption part.

[D4nte]

The second is at the transport level (i.e waku), and it's the signature of the "relayer" of the message (not necessarily the original sender). At this level is also the encryption

In what scenarios is the relayer not the message author?

[D4nte]

@cammellos what store/mailserver strategy do you currently use? Do you retrieve messages when starting the app? when retrieving connectivity? Do you then merge/order messages to display properly? Do you periodically retrieve from the store/mail in case a message is missed via flood/gossip sub?

[D4nte]

@cammellos also, what is the exact convention of a public chat id of an open community please?

[cammellos]

I understand that this is only relevant to then create a Waku v1 filter. right? Not really for the encryption part.

Yes, in waku v1 filter + encryption are coupled (a filter is a combination of a encryption key, a topic, pow value etc).

The second is at the transport level (i.e waku), and it's the signature of the "relayer" of the message (not necessarily the original sender). At this level is also the encryption

In most cases it is, but the datasync layer allows for that https://rfc.vac.dev/spec/2/ through the OFFER/REQUEST mechanism

what store/mailserver strategy do you currently use?

We retrieve messages anytime we switch from an offline state to an online state so:

• when the app starts
• after connectivity is restored from an offline state
• joining new chats

Do you then merge/order messages to display properly? The client side does that, if they are in the current view, otherwise we just save them in the database, we order them by clock-value+message-id to break ties

Do you periodically retrieve from the store/mail in case a message is missed via flood/gossip sub?

Not quite, if you are say always online, we would not be fetching messages ever from mailserver. But say that you open the app at 9 AM, then you go offline at 10 AM and go back online at 11 AM, we would be fetching messages from 9 AM -> 11 AM.

what is the exact convention of a public chat id of an open community please

That would be communityID + chatID, so just straight append

[D4nte]

I can see that for communities, chat id is a new UUID v4:

https://github.com/status-im/status-go/blob/89251e841655082076ef304791b9ee3145f337c2/protocol/communities/manager.go#L301

I could not find a way to extract this UUID from Status Mobile & Desktop app.

I also noticed I get an error when trying to create two chats with the same name in a given community.

Can I safely assume that community public key + chat name uniquely identify a chat?

At first the SDK will allow joining a single chat. I intend to allow the developer to initiliaze the SDK with a community public key + chat name then the SDK will fetch the chat id from waku.

[D4nte]

@cammellos Regarding the message timestamp for display purposes, do you use clock or timestamp (a comment says it's not used) on ChatMessage or do you use the timestamp on the envelope?

[D4nte]

At first the SDK will allow joining a single chat. I intend to allow the developer to initialiaze the SDK with a community public key + chat name then the SDK will fetch the chat id from waku.

@cammellos so actually I can't seem to be able to do that.

When looking at the CommunityDescription protobuf:

message CommunityDescription {
  uint64 clock = 1;
  map<string,CommunityMember> members = 2;
  CommunityPermissions permissions = 3;
  ChatIdentity identity = 5;
  map<string,CommunityChat> chats = 6;
  repeated string ban_list = 7;
  map<string,CommunityCategory> categories = 8;
}

I can see that it contains a map of chats. Can you please clarify what s the string key of the map? is it the full chat id of the chat in value, ie, community key + uuid ?

[cammellos]

Can you please clarify what s the string key of the map? is it the full chat id of the chat in value, ie, community key + uuid ?

It should be only the uuid, without the prefixed community key, that's appended locally

[cammellos]

@cammellos Regarding the message timestamp for display purposes, do you use clock or timestamp (a comment says it's not used) on ChatMessage or do you use the timestamp on the envelope?

Neither actually, timestamp is a legacy field, but it's not validated and it's user controlled. clock is a logical clock (lamport timestamp), though it could be used as a timestamp as it's hinted with the current unix time

But we actually use the timestamp of the envelope, which we have some guarantees to be +/- 20 seconds from real time (ish), as waku nodes wants to have their clock synchronized.

[D4nte]

@cammellos on ChatMessage. When is the content type EMOJI used? When the `text is a single emoji?

Also, for sticker, image and audio messages. Can the text field be defined or does it need to be empty?

[cammellos]

@cammellos on ChatMessage. When is the content type EMOJI used? When the `text is a single emoji?

Correct, yes

Also, for sticker, image and audio messages. Can the text field be defined or does it need to be empty?

It can be defined, generally we set it to a message that invites users to upgrade the app, as it might be displayed textually, but you can set it to anything, though it won't be displayed by most clients

[D4nte]

@Szymx95 see the box I updated. I'll handle the retrieval of messages next week.

[D4nte]

@cammellos I have difficulties generating the symmetric key for a community to decode the community description.

I can confirm that I was able to probably generate the content topic as I do find messages for a community I created in status desktop.

What Is the password for https://github.com/status-im/status-go/blob/a6e7ff6ddde4819c091a2cd73899214df9c12b58/waku/waku.go#L916 for a community? I tried the community public key, I also tried appending -contact-code to no avail.

[D4nte]

Ok I have also noticed that my generation of sym key does not match status go. checking on that.

[D4nte]

@cammellos I am able to find messages on a community topic I am also able to decrypt them

However, I have issue decoding the protobuf.

Are CommunityDescription messages wrapped in something more than the waku envelope?

[D4nte]

Successfully retrieved and decoded CommunityDescription message: https://github.com/status-im/dappconnect-chat-sdk/pull/83

[D4nte]

Successfully retrieved messages from community chat.

However, the messages does not match messages sent in Desktop App when using app StatusIm-210919-232959-7f0e39-pr3513.AppImage.

Whenever I start the desktop App I can see a "resend" red error text so at this stage, I don't think our test can be conclusive.

Let's move forward as it is and re test once there is a new status desktop release.

[D4nte]

Successfully retrieved messages from community chat.

However, the messages does not match messages sent in Desktop App when using app StatusIm-210919-232959-7f0e39-pr3513.AppImage.

Whenever I start the desktop App I can see a "resend" red error text so at this stage, I don't think our test can be conclusive.

Let's move forward as it is and re test once there is a new status desktop release.

Issue is actually that messages are not decrypted/encrypted correctly.

[D4nte]

@cammellos I was able to decrypt a CommunityDescription Messsage but I am now struggling to decrypt chat messages.

For a community public chat, is the password to generate the symmetric key the full chat id <0x communitypublic key><chat uuid> e.g 0x0262c65c881f5a9f79343a26faaa02aad3af7c533d9445fb1939ed11b8bf4d2abd615cb08b-ec64-41f3-9175-3376031ab343

Or the uuid only, e.g. 615cb08b-ec64-41f3-9175-3376031ab343

Or something else? (I tried both above with no luck)

[D4nte]

Playing with status-go tests, it looks like the password is indeed 0x0262c65c881f5a9f79343a26faaa02aad3af7c533d9445fb1939ed11b8bf4d2abd615cb08b-ec64-41f3-9175-3376031ab343.

It seems that Status desktop is not stable enough to confirm whether or not the chat sdk implementation conforms to status encryption.

Will need to wait for a status desktop release to confirm.

[D4nte]

Tracking status desktop issue with: https://github.com/status-im/status-desktop/issues/3899