MUC room not sending messages to a subscriber | Intermittent | Happened only for few messages in the session

[jnmayur]

Before creating a ticket, please consider if this should fit the discussion forum better: https://github.com/processone/ejabberd/discussions

Environment

• ejabberd version: 20.04
• Erlang version: `Eshell V11.0.3
• OS: Linux (Debian)
• Installed from: source

Configuration (only if needed):

##
###              ejabberd configuration file
###
### The parameters used in this configuration file are explained at
###
###       https://docs.ejabberd.im/admin/configuration
###
### The configuration file is written in YAML.
### *******************************************************
### *******           !!! WARNING !!!               *******
### *******     YAML IS INDENTATION SENSITIVE       *******
### ******* MAKE SURE YOU INDENT SECTIONS CORRECTLY *******
### *******************************************************
### Refer to http://en.wikipedia.org/wiki/YAML for the brief description.
###
hosts:
  - host
loglevel: info
## If you already have certificates, list them here
certfiles:
  --removed from pubic
    
#certfiles:
#   	- removed from pubic
#  - removed from pubic
oom_watermark: 50
listen:
  -
    port: 5222
    ip: "::"
    module: ejabberd_c2s
    max_stanza_size: 262144
    shaper: c2s_shaper
    access: c2s
    starttls_required: true
  -
    port: 5269
    ip: "::"
    module: ejabberd_s2s_in
    max_stanza_size: 524288
  -
    port: 8443
    ip: "::"
    module: ejabberd_http
    tls: true
    #certfile: "removed from pubic"
    request_handlers:
      /websockets: ejabberd_http_ws
      /bosh: mod_bosh
      /admin: ejabberd_web_admin
  -
    port: 5281
    ip: "::"
    module: ejabberd_http
    request_handlers:
      /api: mod_http_api
  -
    port: 5280
    ip: "::"
    module: ejabberd_http
    request_handlers:
      /bosh: mod_bosh
      /websockets: ejabberd_http_ws
      /admin: ejabberd_web_admin
     # /.well-known/acme-challenge: ejabberd_acme
  -
    port: 3478
    transport: udp
    module: ejabberd_stun
    use_turn: true
    ## The server's public IPv4 address:
    # turn_ip: 203.0.113.3
  -
    port: 1883
    ip: "::"
    module: mod_mqtt
    backlog: 1000
s2s_use_starttls: optional
acl:
  admin:
    user:
      - "admin": "xyz"
      - "bot_admin": "xyz"
    ip: "0.0.0.0"
  local:
    user_regexp: ""
  loopback:
    ip:
      - "127.0.0.1/8"
      - "::1"
access:
  configure:
    admin: allow
access_rules:
  local:
    allow: local
  c2s:
    deny: blocked
    allow: all
  announce:
    allow: admin
  configure:
    allow: admin
  muc_create:
    allow: loopback
  pubsub_createnode:
    allow: local
  trusted_network:
    allow: loopback
api_permissions:
  "console commands":
    from:
      - ejabberd_ctl
    who: all
    what: "*"
  "admin access":
    who:
      - access:
          - allow:
            - acl: admin
      - oauth:
        - scope: "ejabberd:admin"
        - access:
          - allow:
              - acl: admin
    what:
      - "*"
      - "!stop"
      - "!start"
  "public commands":
    who: all
    what:
      - "status"
      - "register"
      - "connected_users_number"
      - "create_room"
      - "set_room_affiliation"
      - "subscribe_room"
      - "unregister"
      - "unsubscribe_room"
      - "create_room_with_opts"
      - "send_stanza"
shaper:
  normal: 1000
  fast: 50000
shaper_rules:
  max_user_sessions: 10
  max_user_offline_messages:
    5000: admin
    100: all
  c2s_shaper:
    none: admin
    normal: all
  s2s_shaper: fast
modules:
  mod_adhoc: {}
  mod_admin_extra: {}
  mod_announce:
    access: announce
  mod_avatar: {}
  mod_blocking: {}
  mod_bosh: {}
  mod_caps: {}
  mod_carboncopy: {}
  mod_client_state: {}
  mod_configure: {}
  mod_disco: {}
  mod_fail2ban: {}
  mod_http_api: {}
  mod_http_upload:
    put_url: https://@HOST@:5443/upload
  mod_last: {}
  mod_mam:
    ## Mnesia is limited to 2GB, better to use an SQL backend
    ## For small servers SQLite is a good fit and is very easy
    ## to configure. Uncomment this when you have SQL configured:
    db_type: sql
    assume_mam_usage: true
    default: always
  mod_mqtt: {}
  mod_muc:
    access:
      - allow
    access_admin:
      - allow: admin
    access_create: muc_create
    access_persistent: muc_create
    access_mam:
      - allow
    default_room_options:
      mam: true
      persistent: true
      allow_subscription: true
    db_type: sql
    preload_rooms: false
  mod_muc_admin: {}
  mod_offline:
    store_empty_body: unless_chat_state
    store_groupchat: true
    access_max_user_messages: max_user_offline_messages
  mod_ping:
    send_pings: true
    ping_interval: 5
    ping_ack_timeout: 5
    timeout_action: kill
  mod_privacy: {}
  mod_private: {}
  mod_proxy65:
    access: local
    max_connections: 5
  mod_pubsub:
    access_createnode: pubsub_createnode
    plugins:
      - flat
      - pep
    force_node_config:
      ## Avoid buggy clients to make their bookmarks public
      storage:bookmarks:
        access_model: whitelist
  mod_push: {}
  mod_push_keepalive: {}
  mod_register:
    ## Only accept registration requests from the "trusted"
    ## network (see access_rules section above).
    ## Think twice before enabling registration from any
    ## address. See the Jabber SPAM Manifesto for details:
    ## https://github.com/ge0rg/jabber-spam-fighting-manifesto
    ip_access: trusted_network
  mod_roster:
    versioning: true
    db_type: sql
  mod_s2s_dialback: {}
  mod_shared_roster: {}
  mod_stream_mgmt:
    resume_timeout: 60 sec
    resend_on_timeout: if_offline
  mod_stun_disco: {}
  mod_vcard: {}
  mod_vcard_xupdate: {}
  mod_version:
    show_os: false
  mod_chatevents:
    auth_token: "secret"
    post_url: "someurl"
    confidential: true
  mod_recommendation:
    auth_token: "secret"
    post_url: "someurl"
    confidential: true
  mod_offline_http_post:
    auth_token: "secret"
    post_url: "someurl"
    confidential: true
  mod_read_markers:
    auth_token: "secret"
    post_url: "someurl"
    confidential: true
sql_type: mysql
sql_server: "dbserver"
sql_database: "abc"
sql_username: "bcd"
sql_password: "pass"
sql_port: port
default_db: sql
auth_method: sql
### Local Variables:
### mode: yaml
### End:

Errors from error.log/crash.log

No errors

Bug description

Two people are added to a muc_room and are chatting with each other. When User A goes offline and User B sends messages, some of them do not get delivered offline and hence do not also get stored in archive through MAM for that User A. Please note that this is not happening for all messages in the chat from User B, it's happening intermittently. Looking for potential reasons.

We've made sure that

1. There are no error logs at the server for User A.
2. User A is subscribed to the muc_room for messages
3. Some messages are captuured in archive and also sent offline.
4. Spool doesn't have any entries.

[licaon-kter]

So the users set for their accounts MAM as "always" ?

You mean version 20.04, right?

Which clients&versions are used?

[jnmayur]

@licaon-kter Yes 20.04. Corrected now.

Yes default:always for mod_mam. Also mentioned in above yaml.

Also, the real issue we're debugging is that since message is not triggered to User A(assume he's offline), his offline_message_hook is also not triggered. We're evaluating if loffline_message_hook will always be triggered in case user is offline. We're dependent on these events for business use cases.

[jnmayur]

@badlop can you also take a look and share your inputs please.

[badlop]

I don't have much clue what's going, I can just offer some generic ideas for you to try:

The fact that it's intermittent obviously makes it harder to investigate. Also the fact that you are using a fairly old ejabberd version, custom modules, a slightly custom configuration.

I installed ejabberd 20.04, with default config, and sending 20 offline messages, they are stored and recovered correctly. Obviously, the conditions to provoke the problem were not so easy to reproduce.

What reproduction rate do you get? 1 every 10 messages is lost aprox? 1 every 100? Depends on time of day? On server load?

Did you try to disable some custom modules (those not included in ejabberd)?

Are you using ejabberd clustering? Or any other feature that a simple "install + start + register + login" does not take into account?

You can simplify your setup to the barely minimum to reproduce the problem: barely modified configuration file, without additional modules, or options.

Maybe you prefer to install ejabberd in another dev/test machine, try to reproduce the problem. Then configure it slowly to suit your requirements, each time starting it and trying to reproduce the problem. Once you get the problem... what changes did you make?

[badlop]

Hi again! I wonder if this problem is reproducible in any way, or if it ever happened again. Otherwise, I propose to close the issue as "Cannot reproduce", and maybe in the future it gets reopened by anybody with new details.