Update WebXDC instance in-place

[adbenitez]

for the new appstore bot, there is the need to upgrade the appstore.xdc app itself in-place, reasons:

• to avoid spamming the chat with new message with new instance causing confusion and needing to switch app instances
• the new sendToChat() API facilitated a new approach of sending the apps inside the store instead of in-chat and using sendToChat() to start the app in the desired chat, downloaded apps are cached in localStorage of the instance, if upgrades of the store would need to move to a new instance, downloaded apps and potential local settings are lost
• solving the problem for the appstore this way will pave the way for future instance upgrading for all other webxdc apps

how should it work?

for the requirements of the store, it would be enough a simple "edit" approach: just replacing the old webxdc file/attachment of the instance with a a new webxdc file received in a new "special hidden message" type. Probably only the sender of the instance should be able to upgrade/change it (this is also the case for the "re-send webxdc" feature)

a more complex and safe approach that might be needed in the future, mainly when upgrading apps is possible by users in DC clients: app ID and version should be taken in consideration

[link2xt]

I think replacement message should use the same In-Reply-To as the original message. If it is the first message in a chat, which is unlikely in case of a bot but nevertheless, the replacement message will still be linked into thread thanks to References. This way we do not have to modify existing handling of In-Reply-To and think about the code paths related to quoting.

We then need to somehow indicate that the message is an replacement message.

Chat-Version extension

One option is to use the same Message-ID for the replacement as the original message. Then both the original and all replacement messages have the same Message-ID, but we need to somehow make prefetch not skip replacement messages as the duplicate. For this we can add a version somewhere in the prefetched headers, e.g. extend the Chat-Version.

What I do not like about this trick is that the server can see which messages are replacements, because they have the same Message-ID but different contents and size, and the Chat-Version is not encrypted.

Supersedes header

There is a standard header Supersedes. Its value contains Message-IDs of the messages that are replaced by the new one.

With this approach the replacement message has its own Message-ID. The msgs table contains only one rfc724_mid for each message bubble. I think we need to keep it even after replacing the message, so all replies to the message contain the original Message-ID regardless of whether they were sent before or after the replacement.

If we take this approach, there is a question of how to track replacement message in the database. One option is to create entries in the imap table with the Message-ID from Supersedes header. This way the Message-ID of the replacing message never appears in the database and if the user deletes the message bubble, both original and replacement messages will be deleted from IMAP.

Handling messages which don't have original message

If there is no original message in the database when we receive a replacement, we should probably create a new message. However, if the original message is in the trash chat, we should trash the replacement as well and delete it on IMAP. There is a problem that trashed messages (tombstones) are collected by housekeeping almost immediately, maybe we should open an issue about it.

[hpk42]

a supersedes protected header sounds good. If the receiving delta chat ignores the supersedes header it will.just appear as a new message which makes sense.

[hpk42]

So if i understand correctly, @link2xt, it could work like this:

• the superseded message remains unmodified in the msgs table but its blob file is in-place rewritten. If you resend the superseded message, you'll send the replaced blob file.

• the superseding message is not put into the msgs table but rather in the imap table to avoid re-downloading it (IIUC you also suggested this)

• when receiving a superseding blob-replace message we need to check that the sender of the superseding and the original superseded message are the same, that the original message had an xdc/blob file etc.

• if the superseded message does not exist and is not in the trash a new message is created normally (basically this is what would happen with current core that doesn't know about supersession)

All of this is only available on the Rust-API for now and maybe the only thing needed from UIs would be that they signal this message-change somehow to a running webxdc but i guess we could live without it even. On next restart of the app the new blob file will be used (and the app can internally do its own updating/migrations as needed, that's outside core scope).

In any case, let's not target full "message-editing" (including text parts etc.) and not expose this replace-xdc-blob file api to CFFI or jsonrpc before we have thoroughly tested it for the webxdc store-update use case.

[hpk42]

outcome from a quick discussion with @link2xt and @hocuri was that we better not replace the blob file but create a new one and change the params-blobfile link to the new file in the superseded message. Also a dc_msg_send_updated_xdc(msg_id, blobfile) or similar CFFI would in fact be good for python testing with an "experimental" marker in the docstring :)

In the fullness of time, and for more general usage from UIs, we can think about a "message-edit" API (clone existing message into a draft new one etc.) but no need to discuss this in the context of this issue.

[link2xt]

One drawback of not updating rfc724_mid or storing Message-ID of the replacement message anywhere is that prefetch_should_download will not be able to catch that the message was already downloaded if it somehow gets duplicated. This can be fixed by adding another SQL table to matche Message-IDs of replacement messages with Message-IDs of original messages, but I am not sure it is worth the effort to maintain this table because it also has to be cleaned up once the database does not have the Message-ID stored anymore.

[link2xt]

If the superseded message does not exist and is not in the trash a new message is created normally (basically this is what would happen with current core that doesn't know about supersession)

This is actually tricky because in this case receiver device will have no updates (I assume that replacement message is always sent with only the .xdc file and no webxdc updates attached, resending previously sent apps on update will result in too large message), while the sender will have old updates still stored in the database and associated with existing message.

In addition to Supersedes messages, we have resent messages and forwarded messages. When WebXDC message is resent, it is sent with all existing updates attached, but with old Message-ID. When WebXDC message is forwarded, it is sent without any updates and becomes a fresh instance. But with Supersedes messages old devices will keep the updates, while new devices which see the message for the first time will have the same application associated to the same Message-ID but without any updates.

This is likely not much of a problem for the store.xdc with its request-response usage of WebXDC updates, but we should keep in mind that this way it is easy to get inconsistent state on different devices.

[link2xt]

If the receiving delta chat ignores the supersedes header it will.just appear as a new message which makes sense.

This will not work properly because the bot will think it is a single message, while the receiver will think there are two messages with their own WebXDC update histories. Only the original store will work, bot will not reply to this new store because it does not have it as a separate message in the database.

Edit: current solution to this is to modify the core to include delta chat version in download request updates and make it possible for the bot to detect if the download request is sent from the new core. If the core is old, just send a new XDC instead of using send_webxdc_replacement() API.

[link2xt]

Closing currently as there are no plans to add webxdc upgrade API. xstore works without it and is stable without the need for upgrade, while apps are added to it from time to time.