Guidance on tracking down missed messages / mismatched subscriptions

[ColinSullivan1]

I'm not sure where to put this right now so am opening an issue. We should provide guidance on how to track down missing messages / mismatched subscriptions.

Check for Errors

First search the server logs for authorization errors.

Test the subscription

http://demo.nats.io:8222/subsz?subs=1&test=yoursubject

If there's a listener on your subject, you'll get a list returned like:

{
  "server_id": "NCPRDHZCF4V54OQWYICZIFNCXECVZGS7WITJ7UVIFDT6IJZRSSMOMCPV",
  "now": "2021-01-25T11:29:03.914444926-05:00",
  "num_subscriptions": 39,
  "num_cache": 7,
  "num_inserts": 3643482,
  "num_removes": 3643443,
  "num_matches": 77212,
  "cache_hit_rate": 0.45173240075124665,
  "max_fanout": 1,
  "avg_fanout": 0.5714285714285714,
  "total": 1,
  "offset": 0,
  "limit": 1024,
  "subscriptions_list": [
    {
      "account": "$G",
      "subject": "yoursubject",
      "sid": "0",
      "msgs": 0,
      "cid": 68352
    }
  ]
}

If there are no active subscriptions on the supplied subject then no subscriptions_list field will be returned.

Inspect subscriptions

via connections endpoint: http://demo.nats.io:8222/connz?subs=1 via subscriptions endpoint: http://demo.nats.io:8222/subsz?subs=1

Check recently closed connections

You may also want to check closed connections to see if there was a disconnect at the time you are missing messages. http://demo.nats.io:8222/connz?state=closed

If you named the connection in the application, you should be able to easily track down that application's connection closed events and the reason.

Watch published messages

nats-sub your.subject

Watch all published messages and look for a subject mismatch

nats-sub ">"

Debug/Verbose Mode

Run the server with -DV flags and watch protocol messages (or reload with debug:true verbose:true). This is intrusive and impacts performance so should be used sparingly in production.