Add support for URL previews in matrix-rust-sdk

[satrujit11]

Description

We need to add support for URL previews in the matrix-rust-sdk, specifically using the matrix_sdk::ruma::api::client::media::get_media_preview endpoint. URL previews are essential for clients like Fractal, and these previews should be handled server-side as per the Matrix specification.

The API should handle two different endpoints, depending on the Matrix version supported by the homeserver (similar to how get_media_content works). This ensures backward compatibility with older versions of the Matrix protocol.

Requirements

• Implement API support for URL previews using the get_media_preview endpoint.
• Handle compatibility with multiple Matrix versions by supporting both relevant endpoints (similar to how get_media_content is handled).
• Ensure that URL previews are done server-side, not client-side, as specified by the Matrix protocol.

Additional Context

This feature is important for integrating URL preview functionality into Fractal, so it would be ideal to implement the API in the matrix-rust-sdk first and then use it within Fractal. This ensures a proper and standardized implementation of URL previews.

Thanks @afranke for the help with this suggestion.

[satrujit11]

Hi Everyone,

I am currently working on a binding for the get_media_preview functionality. I will be utilizing two endpoints: api::client::authenticated_media::get_media_preview and api::client::media::get_media_preview depending oon the matrix version.

My plan is to handle the extraction of URLs from messages using the SDK. Then, I will use the API to get previews for one or more links and send that information back in a single call.

Could you please advise whether I should use regex for URL extraction, or are there specific methods or crates available in the codebase that would be more appropriate?

Additionally, I welcome any suggestions regarding the path I intend to follow.

Thank you!

[zzorba]

@satrujit11 curious how your effort on this is proceeding, we are also interested in this feature.

[satrujit11]

@zzorba actually i couldn't work on it due to ofcourse insufficient knowledge on this. But can revisit this on next weekend.

[zzorba]

So I'm definitely not an expert on this either, but the way that media works (images/video/audio) is its a new entrypoint at the client level. Even though media is referenced by Messages, the decision to fetch and load it is handled by the end-user of the library.

I think the equivalent for URL previews would be to decide if a link should be previewed or not, and then would call get_url_preview or something on the client to load that information from the URL previews.

[Johennes]

Diving deeper into details of a possible implementation, all clients I've seen this feature in so far, have a room-wide setting to en- or disable loading previews. We can achieve this by using a setting on timelines.

For loading and communicating the previews themselves, I can think of these options:

1. Previews as new type of virtual timeline entries below the event that contains the URLs.
2. Previews on timeline items using TimelineDetails as a wrapper to load them on demand. The SDK would initially only expose the URLs, if any. A client could then decide for itself when to load the previews.

https://github.com/matrix-org/matrix-rust-sdk/blob/ce65317ab892cc3b820e316e14b1f9fb5afc3c53/crates/matrix-sdk-ui/src/timeline/event_item/mod.rs#L581-L594

3. Previews in a central (in-memory?) cache. Like in 2, the SDK would only expose the contained URLs on timeline items. Clients could then call a method to load the previews from the cache when needed and a cache miss would trigger a request to the server.

Personally, option 2 seems like a good compromise between complexity and ease of use to me. I'm curious if there's any preference among the SDK maintainers?