Inconsistent behaviour after enabling use_mam_for_storage option

[logicwonder]

Environment

• ejabberd version: 22.05
• Erlang version: 12.3.1
• OS: Linux (Debian)
• Installed from: source

Errors from error.log/crash.log

No errors

Configuration

mod_mam: assume_mam_usage: true

mod_offline: use_mam_for_storage: true

Errors from error.log/crash.log

No errors

Bug description

I am currently using mod_offline with spool table (SQL) for offline message storage. I am planning to migrate my client to use message archive instead of spool for offline message retrievel. As part of the transition, I enabled the option use_mam_for_storage to true. As per my understanding, this will force mod_offline table to use archive instead of spool table.

During testing offline delivery for chat messages, I observed inconsistent results as given below. The test involved sending 5 chat messages to an offline user and the receiver comes back online after the stream management timeout.

1. With use_mam_for_storage=false => All 5 chat messages gets stored into the spool table after stream management timeout. And the messages are received at client on reconnection. - Success
2. With use_mam_for_storage=true => No offline chat messages are stored in spool table and messages are received at client after stream management timeout (from archive). - Success
3. With use_mam_for_storage=true => The first offline chat messages gets stored in spool table and only this message is received at client after stream management timeout (from archive). - Fail
4. With use_mam_for_storage=true => No offline chat messages are stored in spool table and no messages are received at client after stream management timeout. - Fail

What could be the reason for this inconsistent behaviour?

[licaon-kter]

Which clients are used for testing?

[logicwonder]

I have tried in my existing clients based on Smack and XMPP framework. The missing message issue does not happen if there is only offline one-to-one chat messages. In case any group delivery reciept gets inserted in spool table after the one-to-one chat messages, then one-to-one messages go missing after the user comes back online.

Sample group delivery reciept stored in spool is given below: Resent

And there is no loss when tested with Gajim (Linux) and Conversation (Android).

As per my understanding, this configuration only changes the way offline messages are retrieved at server. Is there any specific changes required in the client side also to ensure that all messages are delivered. Like explicitly pulling messages from MAM based on the timestamp of message received from spool.

[licaon-kter]

then one-to-one messages go missing after the user comes back online.

As expected

And there is no loss when tested with Gajim (Linux) and Conversation (Android).

So your custom client is missing something then? It can handle both Offline and MAM?

[logicwonder]

I did not miss offline messages before enabling these options. My expectation was that existing users would continue to receive all offline messages while the new users could start leveraging MAM for offline messages.

I have analysed the raw offline message stanzas received from the server and the one-to-one messages are not found in it.

Is it possible that the Gajim (Linux) and Conversation (Android) are pulling these messages explicitly from MAM?

Any idea what could be missing at my end?

[logicwonder]

I tried the case in Psi+ XMPP client and only a single one-to-one chat message (out of 4) is received.

[licaon-kter]

users could start leveraging MAM for offline messages.

The storage is transparent (in theory), clients pull Offline or MAM, ejabberd does not tell them where it stored them.

[licaon-kter]

Can you post the whole config on https://gist.github.com ?

[logicwonder]

The storage is transparent (in theory), clients pull Offline or MAM, ejabberd does not tell them where it stored them.

Thanks for clarifying this. My custom client does not sync offline messages from archive. But I am confused with the following documentation:

use_mam_for_storage: true | false This is an experimental option. Enabling this option, mod_offline uses the mod_mam archive table instead of its own spool table to retrieve the messages received when the user was offline. This allows client developers to slowly drop XEP-0160 and rely on XEP-0313 instead. It also further reduces the storage required when you enable MucSub. Enabling this option has a known drawback for the moment: most of flexible message retrieval queries don’t work (those that allow retrieval/deletion of messages by id), but this specification is not widely used. The default value is false to keep former behaviour as default.

My current client version uses XEP-0160 for offline message retrieval. I am planning to rollout the following in the new client version:

1. Implement MucSub
2. Drop XEP-0160 in the next client version and add support to XEP-0313.

On doing this the users with the old client must continue to use XEP-0160. After all users are upgraded to the latest version, I need to remove XEP-0160.

My current configuration: https://gist.github.com/logicwonder/8339390f54dcf19ae9ed7edd6ef070ea

[logicwonder]

@licaon-kter Did you find anything from my configuration?

[PasqualePuzio]

Hi,

we're having the same "issue", although we're not affected as we only use MAM for retrieving any kind of old messages (including those sent when the user was offline).

However we would like to understand what's the expected behavior.

Why does the first offline message get stored also in the spool table if we enable mam for mod_offline?

Thanks

[licaon-kter]

@PasqualePuzio detail your setup, gist of config, etc

[badlop]

Why does the first offline message get stored also in the spool table if we enable mam for mod_offline?

That was first implemented in https://github.com/processone/ejabberd/commit/571a786b9b6e18a861de4767df3ca7fe1147631f. Looking at the commit description, the first message is stored in the offline table to later read its timestamp, and know what messages to read from the MAM table: the ones after that timestamp.

2. With use_mam_for_storage=true => No offline chat messages are stored in spool table and messages are received at client after stream management timeout (from archive). - Success

Remember that use_cache is enabled by default. This means that the table offline_msg may be empty in the database, but not empty in the cache. You can try to disable that toplevel option and repeat the experiments.

3. With use_mam_for_storage=true => The first offline chat messages gets stored in spool table and only this message is received at client after stream management timeout (from archive). - Fail

I didn't see that behaviour, using ejabberd 22.05 and 22.10. In my case it works correctly: the first offline message is stored in offline_msg (and of course in archive_msg). When the user logins, all the messages are sent correctly, not only the first one.

[PasqualePuzio]

Thank you for these details @badlop

Just to make sure we got it right: will the message in the spool table be automatically deleted as soon as the recipient is online again? Or do we have to perform some operation?

Thanks

[badlop]

Right, it gets deleted.

When the receiver of the messages becomes online (with a positive priority), the offline message is sent to him (and deleted from offline_msg table as usual), then ejabberd sends the remaining messages which were stored in archive_msg (and those ones are not deleted, obviously).

[PasqualePuzio]

Right, it gets deleted.

When the receiver of the messages becomes online (with a positive priority), the offline message is sent to him (and deleted from offline_msg table as usual), then ejabberd sends the remaining messages which were stored in archive_msg (and those ones are not deleted, obviously).

Thank you for the clarification @badlop Does this happen only when the user sends his presence packet after authenticating ? Or is it enough that the user connects and successfully authenticates with the server ?

[prefiks]

It's done when we server sends offline messages, so when we get initial presence

[PasqualePuzio]

It's done when we server sends offline messages, so when we get initial presence

Thank you @prefiks So do you confirm that if the user only connects and authenticates without sending the presence stanza, he won't receive the offline messages (therefore they won't be removed from the spool table) ?

Thanks

[badlop]

Thank you @prefiks So do you confirm that if the user only connects and authenticates without sending the presence stanza, he won't receive the offline messages (therefore they won't be removed from the spool table) ?

I mentioned it in https://github.com/processone/ejabberd/issues/3859#issuecomment-1354714650 :

When the receiver of the messages becomes online (with a positive priority)

Prefiks has again mentioned it in https://github.com/processone/ejabberd/issues/3859#issuecomment-1354775771 :

when we get initial presence

You still need another re-confirmation? :)

Try it yourself, it's easy to reproduce. Some clients allow to login and send presence with custom priority. For example, Tkabber allows to set initial presence priority, and you can set it to -1 or whatever: in that case, offline messages are not delivered to the user.

Or check the specification, https://xmpp.org/extensions/xep-0160.html#flow says:

When the recipient next sends non-negative available presence to the server, the server delivers the message to the resource that has sent that presence.

[PasqualePuzio]

Sorry I'm not a Jabber expert :) I just wanted to make sure that my understanding was correct and double-check if it was required to send the presence stanza to delete the messages in the spool table. By the way, everything is clear now, thank you very much for the support.

[badlop]

@logicwonder do you think this issue can be closed, or is there still something to investigate or solve?

[logicwonder]

Sure we can close this. Thanks @badlop