MSC2530: Body field as media caption

[tulir]

Rendered

Implementations:

• mautrix-whatsapp (https://github.com/mautrix/whatsapp/commit/aa0daceb85fa41fb57c75c431d32ba95670aeeb4 + https://github.com/mautrix/whatsapp/commit/794975eaa004f051675da126774d7890857a750f)
• mautrix-telegram
• matrix-react-sdk (rendering: https://github.com/maunium/matrix-react-sdk/commit/b6024aa697ca87c5f781ca4b8c44fbd6744905a4, sending: TODO)

This is an alternative proposal to #2529

[theotheroracle]

you mentioned some clients already suport this behaviour ? which clients are those ?

[MurzNN]

There are already several MSC about adding text to images (at least this one, MSC2881, MSC2674), but we can split their ideas to two types:

1. Classic behavior: media + caption: this is about adding text description, that comments the media. It suit well for articles, where we already have a long text and media (image, video, file) is just an item of the whole story, and it needs a short text description. In that cases it can be named "media caption" or "media description".

2. Chat behavior: text + media: This is a common usage of media in chat style conversations, where we have short text messages and media that adds some information to that short text. The examples can be:

• Here is my photos from last weekend [image1, image2]
• My payment is rejected, here is the file with the rejection message: [file.pdf]
• I've got the "Access denied" error, screenshot is attached. How can I solve this problem?

Also very often we have not a single media, but several ones (two or more photos, several following screenshots, etc) and the "media + caption" behavior doesn't suite for such cases at all!

So I want to discuss the whole idea about media caption implementation that it is not suit well for Matrix ecosystem, and the text + media(s) suits much better for chat style conversations. Please share your own thoughts about this idea!

And maybe discuss this in some better place, could anyone suggest it?

[turt2live]

Please use comments on the diff to receive replies.

[MurzNN]

Yeah, sorry, and seems this MSC is ok, so I've commented a diff in the most suitable MSC here: https://github.com/matrix-org/matrix-spec-proposals/pull/2529#discussion_r896423860

[tulir]

The proposal text has been refactored and it should be ready for review. The only functional change from the initial revision is that body isn't treated as a caption if the value is the same as filename. The list of implementations in the PR description has also been updated.

This has been implemented in Beeper Desktop and iOS for a while, and was also finally shipped in Beeper Android last week. Captions were the primary reason many bridges had to split some remote messages into multiple Matrix events. Now that they're bridged as a single event, points of potential confusion like which message to reply/react to are gone. Since Beeper clients are proprietary and the implementation can't be verified by looking at the source, here's a fancy video of the desktop app:

caption.webm

[cvwright]

Adding "filename" is a good idea, but I still like the idea of having an explicit "caption" field too.

Maybe it makes sense for the "body" to be more like "alt text" anyway. Then you could have a single m.image with body, caption, and filename, and they could all be different.

[SpiritCroc]

Added some rendering support to schildichat-next (Element X-based), which will probably be refined later, but here's some initial implementation commits:

• schildichat-android-next
• matrix-rust-sdk
• ruma

(FWIW the old SchildiChat-android based on Element supported rendering these too)

[turt2live]

I've tested the Beeper Desktop implementation with Element Web, and it seems to work well and as described.

@mscbot fcp merge

[mscbot]

Team member @turt2live has proposed to merge this. The next step is review by the rest of the tagged people:

• [x] @dbkr
• [x] @uhoreg
• [x] @turt2live
• [x] @ara4n
• [ ] @anoadragon453
• [x] @richvdh
• [ ] @erikjohnston
• [x] @KitsuneRal
• [x] @tulir
• [x] @clokep

Once at least 75% of reviewers approve (and there are no outstanding concerns), this will enter its final comment period. If you spot a major issue that hasn't been raised at any point in this process, please speak up!

See this document for information about what commands tagged team members can give me.

[mscbot]

:bell: This is now entering its final comment period, as per the review above. :bell:

[mscbot]

The final comment period, with a disposition to merge, as per the review above, is now complete.

[turt2live]

Spec PR: https://github.com/matrix-org/matrix-spec/pull/1731

[turt2live]

Merged 🎉