MSC2545: Image Packs (Emoticons & Stickers)

[Sorunome]

Rendered

Signed-off-by: Sorunome [email protected]

Author: @Sorunome Shepherd: @anoadragon453

Future MSCs

• Image pack cross-room referencing / suggesting (https://github.com/matrix-org/matrix-spec-proposals/pull/2545#discussion_r753830655)
• Additional metadata for stickers and emotes to find their pack (https://github.com/matrix-org/matrix-spec-proposals/pull/2545#discussion_r753881475)
• Image pack ordering (https://github.com/matrix-org/matrix-spec-proposals/pull/2545#discussion_r910188726)
• Multiple shortcodes for a single image (https://github.com/matrix-org/matrix-spec-proposals/pull/2545#discussion_r598354072)
• E2EE support (https://github.com/matrix-org/matrix-spec-proposals/pull/2545#discussion_r677009720)
• Image-Pack rooms (needing a room type) (https://github.com/matrix-org/matrix-spec-proposals/pull/2545#discussion_r910190308)

Current implementations

Emote rendering (rendering of the <img> tag)

• element-web
• revolution (even handles data-mx-emoticon)
• nheko
• fluffychat (even handles data-mx-emoticon)
• neochat
• cinny

Sending, using the mentioned events here

• revolution (emoticons)
• fluffychat (emoticons, stickers)
• neochat (emoticons, at least partial)
• nheko (emoticons, stickers)
• cinny (emoticons, stickers)
• kazv (stickers)

Bridges

• mx-puppet-discord
• matrix-appservice-discord (partially, sending data-mx-emoticon only)

Implementation PRs

FluffyChat

Data model: https://gitlab.com/famedly/company/frontend/libraries/matrix_api_lite/-/merge_requests/26 SDK: https://gitlab.com/famedly/company/frontend/famedlysdk/-/merge_requests/726 Emoticons: https://gitlab.com/famedly/fluffychat/-/merge_requests/433 Stickers: https://gitlab.com/famedly/fluffychat/-/merge_requests/452

Nheko

Stickers: https://github.com/Nheko-Reborn/nheko/pull/648 Sticker editor: https://github.com/Nheko-Reborn/nheko/pull/669 Choosing emoticons: https://github.com/Nheko-Reborn/nheko/commit/ea6b19b3077dad0280e32aa762137728432e01f4

Cinny

Emoticons and Stickers: https://github.com/cinnyapp/cinny/pull/686

kazv

Creating and sending stickers: https://lily-is.land/kazv/kazv/-/merge_requests/71

---

SCT:

FCP tickyboxes

MSC Checklist

[anoadragon453]

People wanting to try out an implementation of custom emotes on the web can use their existing homeserver account through this riot-web instance.

[MTRNord]

People wanting to try out an implementation of custom emotes on the web can use their existing homeserver account through this riot-web instance.

Fluffychat Android works too :)

[MatMaul]

What about adding a mandatory level of pack indirection ?

im.ponies.room_emotes would only contains something like:

{
  "packs": ["im.ponies.room_emotes.mypack1", "im.ponies.room_emotes.mypack2"]
}

and then use what is proposed here inside im.ponies.room_emotes.mypackX state events.

This allows bundling and updating packs independently, and overcome the 65k limit by quite a margin.

[Sorunome]

What about adding a mandatory level of pack indirection ?

im.ponies.room_emotes would only contains something like:

{
  "packs": ["im.ponies.room_emotes.mypack1", "im.ponies.room_emotes.mypack2"]
}

and then use what is proposed here inside im.ponies.room_emotes.mypackX state events.

This allows bundling and updating packs independently, and overcome the 65k limit by quite a margin.

Not quite sure what you mean, here. That, for room emotes, you can specify other state keys that just extend that one pack?

While not explicitly stated in this MSC (yet?), if multiple packs have the same pack.short they are expected to be merged together - that way you can also overcome the 65k limit, however, when adding it to your account data you will have to add multiple entries for just a single pack. Maybe it would be better to explicitly add related state keys?

[MatMaul]

Ok nevermind, I misread. I thought im.ponies.room_emotes was the state key, not the type.

[theotheroracle]

how hard would it be to add versioning and uuids to emoji packs like for example if i'm in a room with a sticker pack that matches the UUID and version of the pack i'm using, then it won't add those emotes to my emote list

[Sorunome]

how hard would it be to add versioning and uuids to emoji packs like for example if i'm in a room with a sticker pack that matches the UUID and version of the pack i'm using, then it won't add those emotes to my emote list

You already have versioning with state events - or do you mean something like v1.0? It would be trivial to add more information like that, or a uuid, to the pack object.

That being said, what if multiple packs have the same uuid but different emotes? Merge them? What if you spread one pack over multiple state events to overcome the 65k chars limit and then give them all the same uuid? Should they appear as one pack? How should that work with emote_rooms?

[theotheroracle]

how hard would it be to add versioning and uuids to emoji packs like for example if i'm in a room with a sticker pack that matches the UUID and version of the pack i'm using, then it won't add those emotes to my emote list

You already have versioning with state events - or do you mean something like v1.0? It would be trivial to add more information like that, or a uuid, to the pack object.

That being said, what if multiple packs have the same uuid but different emotes? Merge them? What if you spread one pack over multiple state events to overcome the 65k chars limit and then give them all the same uuid? Should they appear as one pack? How should that work with emote_rooms?

if emotes are added to a pack, then the version number should be changed, and yeah i meant like a 1.0 thing, if multiple packs have the same uuid and the same version number, but different emotes, then the conflict should be reported to the user, after thinking about it might make more sense to have a last modified date instead of a version number, and it just uses the newest one, because if you want to remove an emote from a pack, then it shouldn't add it back from an older version of the pack, and i don't see how any of this is affected by it being in a room or a user account

[Sorunome]

and i don't see how any of this is affected by it being in a room or a user account

emote_rooms is where you say that you want emotes in a room (by room id and by state key) visible everywhere for yourself. How to handle now, if you spread one pack over multiple state events by putting the same uuid? Do you have to enable all the state keys? Should it autodetermine? If the later, wouldn't that be complex for client development?

and yeah i meant like a 1.0 thing, if multiple packs have the same uuid and the same version number, but different emotes, then the conflict should be reported to the user

Would a user care about the version of an emote pack? How to determine which version is newer? Grammer for version numbers would need to be defined, etc.

[Sorunome]

Like, maybe the uuid and versioning stuff could be added in a separate msc building onto this - the goal of this was to keep things as simple as possible

[theotheroracle]

emote_rooms is where you say that you want emotes in a room (by room id and by state key) visible everywhere for yourself. How to handle now, if you spread one pack over multiple state events by putting the same uuid? Do you have to enable all the state keys? Should it autodetermine? If the later, wouldn't that be complex for client development?

ok if one pack was spread over multiple events, i would have them share the uuid and also have part numbers, like "part":1 "part":2 etc for A, mainting the order of emotes, and B, to prevent them from overlapping eachother.

Would a user care about the version of an emote pack? How to determine which version is newer? Grammer for version numbers would need to be defined, etc.

state events have timestamps right, so you could just use the newest state event instead of bothering with version numbers

[Sorunome]

state events have timestamps right, so you could just use the newest state event instead of bothering with version numbers

origin_server_ts can't be trusted and can easily be forged

ok if one pack was spread over multiple events, i would have them share the uuid and also have part numbers, like "part":1 "part":2 etc for A, mainting the order of emotes, and B, to prevent them from overlapping eachother.

What would it need part for? It shouldn't matter if they are ordered correctly or not, emotes are an unordered set

[theotheroracle]

state events have timestamps right, so you could just use the newest state event instead of bothering with version numbers

oh you could just put a modified tag in then with like unixtime or something

What would it need part for? It shouldn't matter if they are ordered correctly or not, emotes are an unordered set

the most important reason for the part value is to distinguish continuations of the pack from updates and being able to order the emotes could matter to some pack maintainers, i would want to be able to do that for example

[Sorunome]

oh you could just put a modified tag in then with like unixtime or something

In a federated system there is no one true timesource. That is a mathematically unsolvable problem.

the most important reason for the part value is to distinguish continuations of the pack from updates and being able to order the emotes could matter to some pack maintainers, i would want to be able to do that for example

Updates would replace the previous state event. You don't need any "part" attribute for that. As for ordering, you could say they are ordered alphanumerically by the state key.

[theotheroracle]

In a federated system there is no one true timesource. That is a mathematically unsolvable problem.

right ok, could just be an integer like, every time a pack is modified, the version number is just bumped

Updates would replace the previous state event. You don't need any "part" attribute for that. As for ordering, you could say they are ordered alphanumerically by the state key.

oh right ok not important then

[theotheroracle]

another thing about packs split over multiple state events, is that there maybe should be some sort of indicator that the pack is a slave pack and not the like, source pack or the pack where the avatar and name are held, with the possibility of adding more data to a pack later

[Sorunome]

Changed the format from

{
  "short": {
    ":emote:": "mxc://blah/blubb"
  }
}

to

{
  "emoticons": {
    ":emote": {
      "url": "mxc://blah/blubb"
    }
  }
}

[Sorunome]

Updated the MSC to

1. include stickers (usage parameter)
2. change shortcodes from :emote: to emote, the client can wrap in : or () or whatever it wants to
3. define a bit more metadata that proved to be useful during discussions
4. renamed the state event again, the unstable prefix is still the same
5. user image pack can have the pack key now
6. remove the overrides for room image packs, leaving it to be an empty object for greater flexibility

[BBaoVanC]

shouldn't the discord appservice bridge be listed as one of the bridges with support?

[deepbluev7]

If anyone is interested, I made a quick and dirty hack on top of @tulir's sticker-picker, so that it can create state events from this MSC: https://github.com/deepbluev7/stickerpicker (And implemented sending stickers in Nheko, as mentioned in the description now)

[Sorunome]

Added now to also suggest image packs in claimed parents (m.space.parent)

[deepbluev7]

Please comment such things in a thread. But in general I don't see, why this explicitly should support e2ee. You can already send encrypted stickers (although the media itself wouldn't be encrypted, just the sticker event). But none has any idea yet how to do encrypted state events or how to encrypt this in account data (maybe SSSS?). I don't see why this MSC specifically would be blocked by that. The only thing the user leaks is which sticker packs they use and if they upload a pack in a room, the pack is available unencrypted. This is the same as the room topic, room name, room membership, room avatar, etc. Those 4 are much more critical than what sticker packs you created, especially since people will probably subscribe to public packs more often than create private packs that shouldn't be seen by the server operators.

I don't see why stickers of all things should be blocked behind E2EE state events.

[turt2live]

This is believed to be ready for FCP, though I haven't personally looked at it.

[BBaoVanC]

Shouldn't fluffychat say "stickers and emoticons" under the "Sending, using the mentioned events here" section (under Current implementations)

[richvdh]

I'm removing this from "proposed for FCP readiness" for now, as I see a lot of unresolved comment threads, and no progress for several months. I hope it manages to get some love soon though!

[AndrewRyanChama]

@turt2live @richvdh I see that there are a lot of comments on this proposal, which is understandable due to the large scope and complexity, but my understanding is that it's effectively complete except for small clarifications and readability improvements. @anoadragon453 would you agree with this? If not, it would be helpful to call out specifically the issues that must be addressed for this.

Also, it seems that the original author may have abandoned this PR, as both the author's website and demo element instance have been taken down. If there is substantiate change needed here is it OK if I created a new PR for this? Would this need a new MSC number?

[kate-shine]

Also, it seems that the original author may have abandoned this PR, as both the author's website and demo element instance have been taken down. If there is substantiate change needed here is it OK if I created a new PR for this? Would this need a new MSC number?

Small clarification, original author is still interested in this MSC being finished, she's just unable to find energy to contribute to it right now for personal reasons. Demo element (revolution) instance stopped working for different reason, but implementations in Fluffychat and Nheko are still fully functional.

[turt2live]

This is still on my personal todo list to figure out, though is blocked behind some larger moving pieces. Once things like Extensible Events clear out, I'll be able to take a look at this.

For adding implementations, please just leave a comment and we'll associate them. This does not require a new MSC.

For comments about the MSC, please use line comments on the diff. Otherwise, replies/comments are not considered during review.

And finally, for further questions please visit #matrix-spec:matrix.org so we can answer with a bit more clarity :)

[AndrewRyanChama]

@kate-shine I guess that is a misunderstanding on my part then. I'm happy to wait on the original author if that is so desired, but I can also offer assistance in editing this if it would help expedite progress here.

[Sorunome]

Soru went through and closed threads which are future MSCs that build ontop of this and linked them for reference in the top post. Please say if soru missed any!