stan.py icon indicating copy to clipboard operation
stan.py copied to clipboard

Published 20 hours ago • verified publisher icon nats-io

[DBG] Skipping redelivery to subid=<> due to missed client heartbeat

Open nswarup14 opened this issue 3 years ago • 0 comments

Greetings to the community, I am currently using NATS streaming service as a means to communicate (stream) packets of data between two flask applications. The entire architecture is deployed in individual docker containers, and I'm using docker-compose as a means to start all the containers at once. The flow of packets is as follows:-

  1. I receive my data packets via an MQTT broker to the first flask app. It does minor processing, and publishes to a NATS Streaming topic. I am publishing the packets via a synchronous publish, with no ack_handler.
  2. The second flask app subscribes to the same topic and is supposed to receive the data packets.

The examples provided were procedural, but for my task, the code has classes and the subscription happens within one of the class methods. Consequently, I have defined my subscription callback_handler as a async class method.

I am able to publish successfully, as I can verify with the acks received, but on the subscription side, there seems to be a problem. The packets do not seem to be reaching the second flask at all. I have posted a tracelog of the entire operation for your reference.

payload-nats  | [1] 2020/12/16 07:42:55.957773 [INF] STREAM: Starting nats-streaming-server[nats-streaming] version 0.19.0
payload-nats  | [1] 2020/12/16 07:42:55.957981 [INF] STREAM: ServerID: 0oEnqgEDaVUGRy2HXfESUq
payload-nats  | [1] 2020/12/16 07:42:55.958002 [INF] STREAM: Go version: go1.14.10
payload-nats  | [1] 2020/12/16 07:42:55.958017 [INF] STREAM: Git commit: [c658000]
payload-nats  | [1] 2020/12/16 07:42:56.362184 [INF] Starting nats-server version 2.1.9
payload-nats  | [1] 2020/12/16 07:42:56.362249 [DBG] Go build version go1.14.10
payload-nats  | [1] 2020/12/16 07:42:56.362263 [INF] Git commit [7c76626]
payload-nats  | [1] 2020/12/16 07:42:56.363065 [INF] Starting http monitor on 0.0.0.0:8222
payload-nats  | [1] 2020/12/16 07:42:56.363783 [INF] Listening for client connections on 0.0.0.0:4222
payload-nats  | [1] 2020/12/16 07:42:56.363816 [INF] Server id is NAEAXOASIBLWTG3YSNDQLWE4DET2Q3X3RGAC3AXDM6TZQRDHVFIKNRHN
payload-nats  | [1] 2020/12/16 07:42:56.363826 [INF] Server is ready
payload-nats  | [1] 2020/12/16 07:42:56.363844 [DBG] Get non local IPs for "0.0.0.0"
payload-nats  | [1] 2020/12/16 07:42:56.364535 [DBG]  ip=172.18.0.2
payload-nats  | [1] 2020/12/16 07:42:56.370679 [DBG] 127.0.0.1:60288 - cid:1 - Client connection created
payload-nats  | [1] 2020/12/16 07:42:56.523465 [DBG] 127.0.0.1:60290 - cid:2 - Client connection created
payload-nats  | [1] 2020/12/16 07:42:56.526379 [DBG] 127.0.0.1:60292 - cid:3 - Client connection created
payload-nats  | [1] 2020/12/16 07:42:56.527809 [INF] STREAM: Recovering the state...
payload-nats  | [1] 2020/12/16 07:42:56.528476 [INF] STREAM: No recovered state
payload-nats  | [1] 2020/12/16 07:42:56.779720 [DBG] STREAM: Did not detect another server instance
payload-nats  | [1] 2020/12/16 07:42:56.780147 [DBG] STREAM: Discover subject:           _STAN.discover.nats-streaming
payload-nats  | [1] 2020/12/16 07:42:56.780203 [DBG] STREAM: Publish subject:            _STAN.pub.0oEnqgEDaVUGRy2HXfESYT.>
payload-nats  | [1] 2020/12/16 07:42:56.780224 [DBG] STREAM: Subscribe subject:          _STAN.sub.0oEnqgEDaVUGRy2HXfESYT
payload-nats  | [1] 2020/12/16 07:42:56.780238 [DBG] STREAM: Subscription Close subject: _STAN.subclose.0oEnqgEDaVUGRy2HXfESYT
payload-nats  | [1] 2020/12/16 07:42:56.780255 [DBG] STREAM: Unsubscribe subject:        _STAN.unsub.0oEnqgEDaVUGRy2HXfESYT
payload-nats  | [1] 2020/12/16 07:42:56.780272 [DBG] STREAM: Close subject:              _STAN.close.0oEnqgEDaVUGRy2HXfESYT
payload-nats  | [1] 2020/12/16 07:42:56.780989 [INF] STREAM: Message store is FILE
payload-nats  | [1] 2020/12/16 07:42:56.781030 [INF] STREAM: Store location: /data/msg
payload-nats  | [1] 2020/12/16 07:42:56.781250 [INF] STREAM: ---------- Store Limits ----------
payload-nats  | [1] 2020/12/16 07:42:56.781286 [INF] STREAM: Channels:                  100 *
payload-nats  | [1] 2020/12/16 07:42:56.781347 [INF] STREAM: --------- Channels Limits --------
payload-nats  | [1] 2020/12/16 07:42:56.781359 [INF] STREAM:   Subscriptions:          1000 *
payload-nats  | [1] 2020/12/16 07:42:56.781371 [INF] STREAM:   Messages     :       1000000 *
payload-nats  | [1] 2020/12/16 07:42:56.781381 [INF] STREAM:   Bytes        :     976.56 MB *
payload-nats  | [1] 2020/12/16 07:42:56.781394 [INF] STREAM:   Age          :     unlimited *
payload-nats  | [1] 2020/12/16 07:42:56.781407 [INF] STREAM:   Inactivity   :     unlimited *
payload-nats  | [1] 2020/12/16 07:42:56.781419 [INF] STREAM: ----------------------------------
payload-nats  | [1] 2020/12/16 07:42:56.781433 [INF] STREAM: Streaming Server is ready
payload-nats  | [1] 2020/12/16 07:42:58.648880 [DBG] 127.0.0.1:60290 - cid:2 - Client Ping Timer
payload-nats  | [1] 2020/12/16 07:42:58.670886 [DBG] 127.0.0.1:60288 - cid:1 - Client Ping Timer
payload-nats  | [1] 2020/12/16 07:44:58.649107 [DBG] 127.0.0.1:60290 - cid:2 - Client Ping Timer
payload-nats  | [1] 2020/12/16 07:44:58.649490 [DBG] 127.0.0.1:60290 - cid:2 - Delaying PING due to remote ping 2s ago
payload-nats  | [1] 2020/12/16 07:44:58.671378 [DBG] 127.0.0.1:60288 - cid:1 - Client Ping Timer
payload-nats  | [1] 2020/12/16 07:44:58.671457 [DBG] 127.0.0.1:60288 - cid:1 - Delaying PING due to remote ping 2s ago
payload-nats  | [1] 2020/12/16 07:44:58.768837 [DBG] 127.0.0.1:60292 - cid:3 - Client Ping Timer
payload-nats  | [1] 2020/12/16 07:44:58.768915 [DBG] 127.0.0.1:60292 - cid:3 - Delaying PING due to remote ping 2s ago
payload-nats  | [1] 2020/12/16 07:46:58.649814 [DBG] 127.0.0.1:60290 - cid:2 - Client Ping Timer
payload-nats  | [1] 2020/12/16 07:46:58.649898 [DBG] 127.0.0.1:60290 - cid:2 - Delaying PING due to remote ping 2s ago
payload-nats  | [1] 2020/12/16 07:46:58.671891 [DBG] 127.0.0.1:60288 - cid:1 - Client Ping Timer
payload-nats  | [1] 2020/12/16 07:46:58.671969 [DBG] 127.0.0.1:60288 - cid:1 - Delaying PING due to remote ping 2s ago
payload-nats  | [1] 2020/12/16 07:46:58.769443 [DBG] 127.0.0.1:60292 - cid:3 - Client Ping Timer
payload-nats  | [1] 2020/12/16 07:46:58.769529 [DBG] 127.0.0.1:60292 - cid:3 - Delaying PING due to remote ping 2s ago
payload-nats  | [1] 2020/12/16 07:48:58.650487 [DBG] 127.0.0.1:60290 - cid:2 - Client Ping Timer
payload-nats  | [1] 2020/12/16 07:48:58.650555 [DBG] 127.0.0.1:60290 - cid:2 - Delaying PING due to remote ping 2s ago
payload-nats  | [1] 2020/12/16 07:48:58.672299 [DBG] 127.0.0.1:60288 - cid:1 - Client Ping Timer
payload-nats  | [1] 2020/12/16 07:48:58.672383 [DBG] 127.0.0.1:60288 - cid:1 - Delaying PING due to remote ping 2s ago
payload-nats  | [1] 2020/12/16 07:48:58.769951 [DBG] 127.0.0.1:60292 - cid:3 - Client Ping Timer
payload-nats  | [1] 2020/12/16 07:48:58.770107 [DBG] 127.0.0.1:60292 - cid:3 - Delaying PING due to remote ping 2s ago
payload-nats  | [1] 2020/12/16 07:50:58.650975 [DBG] 127.0.0.1:60290 - cid:2 - Client Ping Timer
payload-nats  | [1] 2020/12/16 07:50:58.651055 [DBG] 127.0.0.1:60290 - cid:2 - Delaying PING due to remote ping 2s ago
payload-nats  | [1] 2020/12/16 07:50:58.672658 [DBG] 127.0.0.1:60288 - cid:1 - Client Ping Timer
payload-nats  | [1] 2020/12/16 07:50:58.672714 [DBG] 127.0.0.1:60288 - cid:1 - Delaying PING due to remote ping 2s ago
payload-nats  | [1] 2020/12/16 07:50:58.770561 [DBG] 127.0.0.1:60292 - cid:3 - Client Ping Timer
payload-nats  | [1] 2020/12/16 07:50:58.770732 [DBG] 127.0.0.1:60292 - cid:3 - Delaying PING due to remote ping 2s ago
payload-nats  | [1] 2020/12/16 07:51:23.540769 [DBG] 172.18.0.4:60104 - cid:4 - Client connection created
payload-nats  | [1] 2020/12/16 07:51:23.645746 [DBG] STREAM: [Client:decryption-nats-client] Connected (Inbox=_INBOX.8b8afd43c20f1a2a73c5f0)
payload-nats  | [1] 2020/12/16 07:51:23.650223 [INF] STREAM: Channel "transfer-frames" has been created
payload-nats  | [1] 2020/12/16 07:51:23.710991 [DBG] STREAM: [Client:decryption-nats-client] Started new subscription, subject=transfer-frames, inbox=_INBOX.9a3c1a083c32638f183c0c, subid=1, sending new-only, seq=1
payload-nats  | [1] 2020/12/16 07:51:25.854088 [DBG] 172.18.0.4:60104 - cid:4 - Client Ping Timer
payload-nats  | [1] 2020/12/16 07:51:30.708514 [DBG] 172.18.0.5:58202 - cid:5 - Client connection created
payload-nats  | [1] 2020/12/16 07:51:30.760001 [DBG] STREAM: [Client:reception-nats-client] Connected (Inbox=_INBOX.d5a4d3f2e9745132d4c1a4)
payload-nats  | [1] 2020/12/16 07:51:32.833547 [DBG] 172.18.0.5:58202 - cid:5 - Client Ping Timer
payload-nats  | [1] 2020/12/16 07:52:02.724329 [TRC] STREAM: [Client:reception-nats-client] Received message from publisher subj=transfer-frames guid=43d30895023671eac648a4
payload-nats  | [1] 2020/12/16 07:52:02.803399 [TRC] STREAM: [Client:decryption-nats-client] Delivering msg to subid=1, subject=transfer-frames, seq=1
payload-nats  | [1] 2020/12/16 07:52:02.836905 [TRC] STREAM: [Client:reception-nats-client] Acking Publisher subj=transfer-frames guid=43d30895023671eac648a4
payload-nats  | [1] 2020/12/16 07:52:04.432645 [TRC] STREAM: [Client:reception-nats-client] Received message from publisher subj=transfer-frames guid=a7db8e894ded821891369f
payload-nats  | [1] 2020/12/16 07:52:04.472127 [TRC] STREAM: [Client:decryption-nats-client] Delivering msg to subid=1, subject=transfer-frames, seq=2
payload-nats  | [1] 2020/12/16 07:52:04.519354 [TRC] STREAM: [Client:reception-nats-client] Acking Publisher subj=transfer-frames guid=a7db8e894ded821891369f
payload-nats  | [1] 2020/12/16 07:52:06.472810 [TRC] STREAM: [Client:reception-nats-client] Received message from publisher subj=transfer-frames guid=775b8186d275dbc250058a
payload-nats  | [1] 2020/12/16 07:52:06.521575 [TRC] STREAM: [Client:decryption-nats-client] Delivering msg to subid=1, subject=transfer-frames, seq=3
payload-nats  | [1] 2020/12/16 07:52:06.560889 [TRC] STREAM: [Client:reception-nats-client] Acking Publisher subj=transfer-frames guid=775b8186d275dbc250058a
payload-nats  | [1] 2020/12/16 07:52:08.424016 [TRC] STREAM: [Client:reception-nats-client] Received message from publisher subj=transfer-frames guid=75d3bfb699bb8ce823a806
payload-nats  | [1] 2020/12/16 07:52:08.471177 [TRC] STREAM: [Client:decryption-nats-client] Delivering msg to subid=1, subject=transfer-frames, seq=4
payload-nats  | [1] 2020/12/16 07:52:08.499723 [TRC] STREAM: [Client:reception-nats-client] Acking Publisher subj=transfer-frames guid=75d3bfb699bb8ce823a806
payload-nats  | [1] 2020/12/16 07:52:10.573253 [TRC] STREAM: [Client:reception-nats-client] Received message from publisher subj=transfer-frames guid=bd6f8361bf853bb7459d34
payload-nats  | [1] 2020/12/16 07:52:10.612676 [TRC] STREAM: [Client:decryption-nats-client] Delivering msg to subid=1, subject=transfer-frames, seq=5
payload-nats  | [1] 2020/12/16 07:52:10.641562 [TRC] STREAM: [Client:reception-nats-client] Acking Publisher subj=transfer-frames guid=bd6f8361bf853bb7459d34
payload-nats  | [1] 2020/12/16 07:52:12.416695 [TRC] STREAM: [Client:reception-nats-client] Received message from publisher subj=transfer-frames guid=b06081d386030954397ef0
payload-nats  | [1] 2020/12/16 07:52:12.462441 [TRC] STREAM: [Client:decryption-nats-client] Delivering msg to subid=1, subject=transfer-frames, seq=6
payload-nats  | [1] 2020/12/16 07:52:12.491464 [TRC] STREAM: [Client:reception-nats-client] Acking Publisher subj=transfer-frames guid=b06081d386030954397ef0
payload-nats  | [1] 2020/12/16 07:52:12.803875 [DBG] STREAM: [Client:decryption-nats-client] Skipping redelivery to subid=1 due to missed client heartbeat
payload-nats  | [1] 2020/12/16 07:52:14.364721 [TRC] STREAM: [Client:reception-nats-client] Received message from publisher subj=transfer-frames guid=d86a9a5c120aa07aa2f6f4
payload-nats  | [1] 2020/12/16 07:52:14.404039 [TRC] STREAM: [Client:decryption-nats-client] Delivering msg to subid=1, subject=transfer-frames, seq=7
payload-nats  | [1] 2020/12/16 07:52:14.443453 [TRC] STREAM: [Client:reception-nats-client] Acking Publisher subj=transfer-frames guid=d86a9a5c120aa07aa2f6f4
payload-nats  | [1] 2020/12/16 07:52:16.309511 [TRC] STREAM: [Client:reception-nats-client] Received message from publisher subj=transfer-frames guid=3b5b04e4b4142e0ce2af00
payload-nats  | [1] 2020/12/16 07:52:16.353729 [TRC] STREAM: [Client:decryption-nats-client] Delivering msg to subid=1, subject=transfer-frames, seq=8
payload-nats  | [1] 2020/12/16 07:52:16.392479 [TRC] STREAM: [Client:reception-nats-client] Acking Publisher subj=transfer-frames guid=3b5b04e4b4142e0ce2af00
payload-nats  | [1] 2020/12/16 07:52:18.453142 [TRC] STREAM: [Client:reception-nats-client] Received message from publisher subj=transfer-frames guid=5301fe16ac096ba5538c9c
payload-nats  | [1] 2020/12/16 07:52:18.544103 [TRC] STREAM: [Client:decryption-nats-client] Delivering msg to subid=1, subject=transfer-frames, seq=9
payload-nats  | [1] 2020/12/16 07:52:18.593894 [TRC] STREAM: [Client:reception-nats-client] Acking Publisher subj=transfer-frames guid=5301fe16ac096ba5538c9c
payload-nats  | [1] 2020/12/16 07:52:20.403590 [TRC] STREAM: [Client:reception-nats-client] Received message from publisher subj=transfer-frames guid=6187b4946170be20b85146
payload-nats  | [1] 2020/12/16 07:52:20.444888 [TRC] STREAM: [Client:decryption-nats-client] Delivering msg to subid=1, subject=transfer-frames, seq=10
payload-nats  | [1] 2020/12/16 07:52:20.470434 [TRC] STREAM: [Client:reception-nats-client] Acking Publisher subj=transfer-frames guid=6187b4946170be20b85146
payload-nats  | [1] 2020/12/16 07:52:22.451509 [TRC] STREAM: [Client:reception-nats-client] Received message from publisher subj=transfer-frames guid=e41cb588bbe14ac480e83f
payload-nats  | [1] 2020/12/16 07:52:22.503058 [TRC] STREAM: [Client:decryption-nats-client] Delivering msg to subid=1, subject=transfer-frames, seq=11
payload-nats  | [1] 2020/12/16 07:52:22.542551 [TRC] STREAM: [Client:reception-nats-client] Acking Publisher subj=transfer-frames guid=e41cb588bbe14ac480e83f
payload-nats  | [1] 2020/12/16 07:52:22.804118 [DBG] STREAM: [Client:decryption-nats-client] Skipping redelivery to subid=1 due to missed client heartbeat
payload-nats  | [1] 2020/12/16 07:52:24.391571 [TRC] STREAM: [Client:reception-nats-client] Received message from publisher subj=transfer-frames guid=548971ca6e0478afd7551f
payload-nats  | [1] 2020/12/16 07:52:24.494543 [TRC] STREAM: [Client:decryption-nats-client] Delivering msg to subid=1, subject=transfer-frames, seq=12
payload-nats  | [1] 2020/12/16 07:52:24.521356 [TRC] STREAM: [Client:reception-nats-client] Acking Publisher subj=transfer-frames guid=548971ca6e0478afd7551f
payload-nats  | [1] 2020/12/16 07:52:26.336656 [TRC] STREAM: [Client:reception-nats-client] Received message from publisher subj=transfer-frames guid=b5a5ed6e4b514ee2a40139
payload-nats  | [1] 2020/12/16 07:52:26.377745 [TRC] STREAM: [Client:decryption-nats-client] Delivering msg to subid=1, subject=transfer-frames, seq=13
payload-nats  | [1] 2020/12/16 07:52:26.417236 [TRC] STREAM: [Client:reception-nats-client] Acking Publisher subj=transfer-frames guid=b5a5ed6e4b514ee2a40139
payload-nats  | [1] 2020/12/16 07:52:28.383872 [TRC] STREAM: [Client:reception-nats-client] Received message from publisher subj=transfer-frames guid=e19236e7738ab143a2154e
payload-nats  | [1] 2020/12/16 07:52:28.435839 [TRC] STREAM: [Client:decryption-nats-client] Delivering msg to subid=1, subject=transfer-frames, seq=14
payload-nats  | [1] 2020/12/16 07:52:28.475239 [TRC] STREAM: [Client:reception-nats-client] Acking Publisher subj=transfer-frames guid=e19236e7738ab143a2154e
payload-nats  | [1] 2020/12/16 07:52:30.561078 [TRC] STREAM: [Client:reception-nats-client] Received message from publisher subj=transfer-frames guid=61b933d0b001b651256f76
payload-nats  | [1] 2020/12/16 07:52:30.610855 [TRC] STREAM: [Client:decryption-nats-client] Delivering msg to subid=1, subject=transfer-frames, seq=15
payload-nats  | [1] 2020/12/16 07:52:30.640088 [TRC] STREAM: [Client:reception-nats-client] Acking Publisher subj=transfer-frames guid=61b933d0b001b651256f76
payload-nats  | [1] 2020/12/16 07:52:32.377026 [TRC] STREAM: [Client:reception-nats-client] Received message from publisher subj=transfer-frames guid=9f36ce6deaf8e133f9ed24
payload-nats  | [1] 2020/12/16 07:52:32.427303 [TRC] STREAM: [Client:decryption-nats-client] Delivering msg to subid=1, subject=transfer-frames, seq=16
payload-nats  | [1] 2020/12/16 07:52:32.458046 [TRC] STREAM: [Client:reception-nats-client] Acking Publisher subj=transfer-frames guid=9f36ce6deaf8e133f9ed24
payload-nats  | [1] 2020/12/16 07:52:32.804406 [DBG] STREAM: [Client:decryption-nats-client] Skipping redelivery to subid=1 due to missed client heartbeat
payload-nats  | [1] 2020/12/16 07:52:34.434049 [TRC] STREAM: [Client:reception-nats-client] Received message from publisher subj=transfer-frames guid=89e34a1d38dfe1183615d
payload-nats  | [1] 2020/12/16 07:52:34.493575 [TRC] STREAM: [Client:decryption-nats-client] Delivering msg to subid=1, subject=transfer-frames, seq=17
payload-nats  | [1] 2020/12/16 07:52:34.524989 [TRC] STREAM: [Client:reception-nats-client] Acking Publisher subj=transfer-frames guid=89e34a1d38dfe1183615d
payload-nats  | [1] 2020/12/16 07:52:36.372304 [TRC] STREAM: [Client:reception-nats-client] Received message from publisher subj=transfer-frames guid=e1bb9fced16c42eacc895b
payload-nats  | [1] 2020/12/16 07:52:36.410013 [TRC] STREAM: [Client:decryption-nats-client] Delivering msg to subid=1, subject=transfer-frames, seq=18
payload-nats  | [1] 2020/12/16 07:52:36.434339 [TRC] STREAM: [Client:reception-nats-client] Acking Publisher subj=transfer-frames guid=e1bb9fced16c42eacc895b
payload-nats  | [1] 2020/12/16 07:52:38.315697 [TRC] STREAM: [Client:reception-nats-client] Received message from publisher subj=transfer-frames guid=710086c7d507a258f8dcdf
payload-nats  | [1] 2020/12/16 07:52:38.384779 [TRC] STREAM: [Client:decryption-nats-client] Delivering msg to subid=1, subject=transfer-frames, seq=19
payload-nats  | [1] 2020/12/16 07:52:38.435229 [TRC] STREAM: [Client:reception-nats-client] Acking Publisher subj=transfer-frames guid=710086c7d507a258f8dcdf
payload-nats  | [1] 2020/12/16 07:52:40.569668 [TRC] STREAM: [Client:reception-nats-client] Received message from publisher subj=transfer-frames guid=90ccdfc14a3b8269fbb9a5
payload-nats  | [1] 2020/12/16 07:52:40.617892 [TRC] STREAM: [Client:decryption-nats-client] Delivering msg to subid=1, subject=transfer-frames, seq=20
payload-nats  | [1] 2020/12/16 07:52:40.657333 [TRC] STREAM: [Client:reception-nats-client] Acking Publisher subj=transfer-frames guid=90ccdfc14a3b8269fbb9a5
payload-nats  | [1] 2020/12/16 07:52:42.410943 [TRC] STREAM: [Client:reception-nats-client] Received message from publisher subj=transfer-frames guid=c2f0212b1256a3f3136138
payload-nats  | [1] 2020/12/16 07:52:42.459317 [TRC] STREAM: [Client:decryption-nats-client] Delivering msg to subid=1, subject=transfer-frames, seq=21
payload-nats  | [1] 2020/12/16 07:52:42.494603 [TRC] STREAM: [Client:reception-nats-client] Acking Publisher subj=transfer-frames guid=c2f0212b1256a3f3136138
payload-nats  | [1] 2020/12/16 07:52:42.804448 [DBG] STREAM: [Client:decryption-nats-client] Skipping redelivery to subid=1 due to missed client heartbeat
payload-nats  | [1] 2020/12/16 07:52:44.357699 [TRC] STREAM: [Client:reception-nats-client] Received message from publisher subj=transfer-frames guid=dd82580b54607e33df9062
payload-nats  | [1] 2020/12/16 07:52:44.400793 [TRC] STREAM: [Client:decryption-nats-client] Delivering msg to subid=1, subject=transfer-frames, seq=22
payload-nats  | [1] 2020/12/16 07:52:44.444773 [TRC] STREAM: [Client:reception-nats-client] Acking Publisher subj=transfer-frames guid=dd82580b54607e33df9062
payload-nats  | [1] 2020/12/16 07:52:46.408721 [TRC] STREAM: [Client:reception-nats-client] Received message from publisher subj=transfer-frames guid=b1a7a038e5cad7e68eefef
payload-nats  | [1] 2020/12/16 07:52:46.459339 [TRC] STREAM: [Client:decryption-nats-client] Delivering msg to subid=1, subject=transfer-frames, seq=23
payload-nats  | [1] 2020/12/16 07:52:46.486816 [TRC] STREAM: [Client:reception-nats-client] Acking Publisher subj=transfer-frames guid=b1a7a038e5cad7e68eefef
payload-nats  | [1] 2020/12/16 07:52:48.354389 [TRC] STREAM: [Client:reception-nats-client] Received message from publisher subj=transfer-frames guid=5b67d62010b2b7e7ba5cc1
payload-nats  | [1] 2020/12/16 07:52:48.425917 [TRC] STREAM: [Client:decryption-nats-client] Delivering msg to subid=1, subject=transfer-frames, seq=24
payload-nats  | [1] 2020/12/16 07:52:48.456144 [TRC] STREAM: [Client:reception-nats-client] Acking Publisher subj=transfer-frames guid=5b67d62010b2b7e7ba5cc1
payload-nats  | [1] 2020/12/16 07:52:50.402095 [TRC] STREAM: [Client:reception-nats-client] Received message from publisher subj=transfer-frames guid=5b0db86d26dc4d50008cd
payload-nats  | [1] 2020/12/16 07:52:50.458676 [TRC] STREAM: [Client:decryption-nats-client] Delivering msg to subid=1, subject=transfer-frames, seq=25
payload-nats  | [1] 2020/12/16 07:52:50.490188 [TRC] STREAM: [Client:reception-nats-client] Acking Publisher subj=transfer-frames guid=5b0db86d26dc4d50008cd
payload-nats  | [1] 2020/12/16 07:52:52.554838 [TRC] STREAM: [Client:reception-nats-client] Received message from publisher subj=transfer-frames guid=a90925c37f3d4437ad575e
payload-nats  | [1] 2020/12/16 07:52:52.591826 [TRC] STREAM: [Client:decryption-nats-client] Delivering msg to subid=1, subject=transfer-frames, seq=26
payload-nats  | [1] 2020/12/16 07:52:52.623628 [TRC] STREAM: [Client:reception-nats-client] Acking Publisher subj=transfer-frames guid=a90925c37f3d4437ad575e
payload-nats  | [1] 2020/12/16 07:52:52.804876 [DBG] STREAM: [Client:decryption-nats-client] Skipping redelivery to subid=1 due to missed client heartbeat
payload-nats  | [1] 2020/12/16 07:52:54.396984 [TRC] STREAM: [Client:reception-nats-client] Received message from publisher subj=transfer-frames guid=a558904aee10925aec153f
payload-nats  | [1] 2020/12/16 07:52:54.441861 [TRC] STREAM: [Client:decryption-nats-client] Delivering msg to subid=1, subject=transfer-frames, seq=27
payload-nats  | [1] 2020/12/16 07:52:54.490076 [TRC] STREAM: [Client:reception-nats-client] Acking Publisher subj=transfer-frames guid=a558904aee10925aec153f
payload-nats  | [1] 2020/12/16 07:52:56.731543 [TRC] STREAM: [Client:reception-nats-client] Received message from publisher subj=transfer-frames guid=79dfa4a48390b3cd1236a1
payload-nats  | [1] 2020/12/16 07:52:56.783429 [TRC] STREAM: [Client:decryption-nats-client] Delivering msg to subid=1, subject=transfer-frames, seq=28
payload-nats  | [1] 2020/12/16 07:52:56.822748 [TRC] STREAM: [Client:reception-nats-client] Acking Publisher subj=transfer-frames guid=79dfa4a48390b3cd1236a1
payload-nats  | [1] 2020/12/16 07:52:58.384819 [TRC] STREAM: [Client:reception-nats-client] Received message from publisher subj=transfer-frames guid=2ebbc0a3e3958f10de6d6c
payload-nats  | [1] 2020/12/16 07:52:58.432917 [TRC] STREAM: [Client:decryption-nats-client] Delivering msg to subid=1, subject=transfer-frames, seq=29
payload-nats  | [1] 2020/12/16 07:52:58.466321 [TRC] STREAM: [Client:reception-nats-client] Acking Publisher subj=transfer-frames guid=2ebbc0a3e3958f10de6d6c
payload-nats  | [1] 2020/12/16 07:52:58.651340 [DBG] 127.0.0.1:60290 - cid:2 - Client Ping Timer
payload-nats  | [1] 2020/12/16 07:52:58.651387 [DBG] 127.0.0.1:60290 - cid:2 - Delaying PING due to client activity 0s ago
payload-nats  | [1] 2020/12/16 07:52:58.672867 [DBG] 127.0.0.1:60288 - cid:1 - Client Ping Timer
payload-nats  | [1] 2020/12/16 07:52:58.672892 [DBG] 127.0.0.1:60288 - cid:1 - Delaying PING due to client activity 0s ago
payload-nats  | [1] 2020/12/16 07:52:58.771163 [DBG] 127.0.0.1:60292 - cid:3 - Client Ping Timer
payload-nats  | [1] 2020/12/16 07:52:58.771234 [DBG] 127.0.0.1:60292 - cid:3 - Delaying PING due to client activity 1m35s ago
payload-nats  | [1] 2020/12/16 07:53:00.437759 [TRC] STREAM: [Client:reception-nats-client] Received message from publisher subj=transfer-frames guid=93f5ffd19a3c08ff6e8b9b
payload-nats  | [1] 2020/12/16 07:53:00.491363 [TRC] STREAM: [Client:decryption-nats-client] Delivering msg to subid=1, subject=transfer-frames, seq=30
payload-nats  | [1] 2020/12/16 07:53:00.517582 [TRC] STREAM: [Client:reception-nats-client] Acking Publisher subj=transfer-frames guid=93f5ffd19a3c08ff6e8b9b
payload-nats  | [1] 2020/12/16 07:53:02.480424 [TRC] STREAM: [Client:reception-nats-client] Received message from publisher subj=transfer-frames guid=b6f0777f2063ef9d0fc2ca
payload-nats  | [1] 2020/12/16 07:53:02.532463 [TRC] STREAM: [Client:decryption-nats-client] Delivering msg to subid=1, subject=transfer-frames, seq=31
payload-nats  | [1] 2020/12/16 07:53:02.559954 [TRC] STREAM: [Client:reception-nats-client] Acking Publisher subj=transfer-frames guid=b6f0777f2063ef9d0fc2ca
payload-nats  | [1] 2020/12/16 07:53:02.805183 [DBG] STREAM: [Client:decryption-nats-client] Skipping redelivery to subid=1 due to missed client heartbeat
payload-nats  | [1] 2020/12/16 07:53:04.626294 [TRC] STREAM: [Client:reception-nats-client] Received message from publisher subj=transfer-frames guid=8a27b34579cb86bfcbba51
payload-nats  | [1] 2020/12/16 07:53:04.673963 [TRC] STREAM: [Client:decryption-nats-client] Delivering msg to subid=1, subject=transfer-frames, seq=32
payload-nats  | [1] 2020/12/16 07:53:04.702380 [TRC] STREAM: [Client:reception-nats-client] Acking Publisher subj=transfer-frames guid=8a27b34579cb86bfcbba51
payload-nats  | [1] 2020/12/16 07:53:06.580472 [TRC] STREAM: [Client:reception-nats-client] Received message from publisher subj=transfer-frames guid=99da69ee1c80277bb10770
payload-nats  | [1] 2020/12/16 07:53:06.623701 [TRC] STREAM: [Client:decryption-nats-client] Delivering msg to subid=1, subject=transfer-frames, seq=33
payload-nats  | [1] 2020/12/16 07:53:06.652928 [TRC] STREAM: [Client:reception-nats-client] Acking Publisher subj=transfer-frames guid=99da69ee1c80277bb10770
payload-nats  | [1] 2020/12/16 07:53:08.424349 [TRC] STREAM: [Client:reception-nats-client] Received message from publisher subj=transfer-frames guid=c9965c6b1d31f47696ab34
payload-nats  | [1] 2020/12/16 07:53:08.465282 [TRC] STREAM: [Client:decryption-nats-client] Delivering msg to subid=1, subject=transfer-frames, seq=34
payload-nats  | [1] 2020/12/16 07:53:08.497686 [TRC] STREAM: [Client:reception-nats-client] Acking Publisher subj=transfer-frames guid=c9965c6b1d31f47696ab34
payload-nats  | [1] 2020/12/16 07:53:10.471966 [TRC] STREAM: [Client:reception-nats-client] Received message from publisher subj=transfer-frames guid=dab30410224d6d398cc0dd
payload-nats  | [1] 2020/12/16 07:53:10.623764 [TRC] STREAM: [Client:decryption-nats-client] Delivering msg to subid=1, subject=transfer-frames, seq=35
payload-nats  | [1] 2020/12/16 07:53:10.656660 [TRC] STREAM: [Client:reception-nats-client] Acking Publisher subj=transfer-frames guid=dab30410224d6d398cc0dd
payload-nats  | [1] 2020/12/16 07:53:12.419842 [TRC] STREAM: [Client:reception-nats-client] Received message from publisher subj=transfer-frames guid=ec9d0547b212264c8eab97
payload-nats  | [1] 2020/12/16 07:53:12.481991 [TRC] STREAM: [Client:decryption-nats-client] Delivering msg to subid=1, subject=transfer-frames, seq=36
payload-nats  | [1] 2020/12/16 07:53:12.506775 [TRC] STREAM: [Client:reception-nats-client] Acking Publisher subj=transfer-frames guid=ec9d0547b212264c8eab97
payload-nats  | [1] 2020/12/16 07:53:12.805514 [DBG] STREAM: [Client:decryption-nats-client] Skipping redelivery to subid=1 due to missed client heartbeat
payload-nats  | [1] 2020/12/16 07:53:14.567589 [TRC] STREAM: [Client:reception-nats-client] Received message from publisher subj=transfer-frames guid=f6d579278eb0426517b8c9
payload-nats  | [1] 2020/12/16 07:53:14.614585 [TRC] STREAM: [Client:decryption-nats-client] Delivering msg to subid=1, subject=transfer-frames, seq=37
payload-nats  | [1] 2020/12/16 07:53:14.704266 [TRC] STREAM: [Client:reception-nats-client] Acking Publisher subj=transfer-frames guid=f6d579278eb0426517b8c9
payload-nats  | [1] 2020/12/16 07:53:16.733736 [TRC] STREAM: [Client:reception-nats-client] Received message from publisher subj=transfer-frames guid=b0c4b15b7dec73f2f4729f
payload-nats  | [1] 2020/12/16 07:53:16.772844 [TRC] STREAM: [Client:decryption-nats-client] Delivering msg to subid=1, subject=transfer-frames, seq=38
payload-nats  | [1] 2020/12/16 07:53:16.801999 [TRC] STREAM: [Client:reception-nats-client] Acking Publisher subj=transfer-frames guid=b0c4b15b7dec73f2f4729f
payload-nats  | [1] 2020/12/16 07:53:19.169750 [TRC] STREAM: [Client:reception-nats-client] Received message from publisher subj=transfer-frames guid=22a5509b75561c92a859ee
payload-nats  | [1] 2020/12/16 07:53:19.214135 [TRC] STREAM: [Client:decryption-nats-client] Delivering msg to subid=1, subject=transfer-frames, seq=39
payload-nats  | [1] 2020/12/16 07:53:19.262338 [TRC] STREAM: [Client:reception-nats-client] Acking Publisher subj=transfer-frames guid=22a5509b75561c92a859ee
payload-nats  | [1] 2020/12/16 07:53:21.217574 [TRC] STREAM: [Client:reception-nats-client] Received message from publisher subj=transfer-frames guid=42315bb1fff87806957bed
payload-nats  | [1] 2020/12/16 07:53:21.255729 [TRC] STREAM: [Client:decryption-nats-client] Delivering msg to subid=1, subject=transfer-frames, seq=40
payload-nats  | [1] 2020/12/16 07:53:21.280536 [TRC] STREAM: [Client:reception-nats-client] Acking Publisher subj=transfer-frames guid=42315bb1fff87806957bed
payload-nats  | [1] 2020/12/16 07:53:22.441525 [TRC] STREAM: [Client:reception-nats-client] Received message from publisher subj=transfer-frames guid=6735951bfe30c7f22e8473
payload-nats  | [1] 2020/12/16 07:53:22.488988 [TRC] STREAM: [Client:decryption-nats-client] Delivering msg to subid=1, subject=transfer-frames, seq=41
payload-nats  | [1] 2020/12/16 07:53:22.514339 [TRC] STREAM: [Client:reception-nats-client] Acking Publisher subj=transfer-frames guid=6735951bfe30c7f22e8473
payload-nats  | [1] 2020/12/16 07:53:22.805794 [DBG] STREAM: [Client:decryption-nats-client] Skipping redelivery to subid=1 due to missed client heartbeat
payload-nats  | [1] 2020/12/16 07:53:24.500405 [TRC] STREAM: [Client:reception-nats-client] Received message from publisher subj=transfer-frames guid=66cbcb0bad0c02c1e8d23e
payload-nats  | [1] 2020/12/16 07:53:24.546947 [TRC] STREAM: [Client:decryption-nats-client] Delivering msg to subid=1, subject=transfer-frames, seq=42
payload-nats  | [1] 2020/12/16 07:53:24.581559 [TRC] STREAM: [Client:reception-nats-client] Acking Publisher subj=transfer-frames guid=66cbcb0bad0c02c1e8d23e
payload-nats  | [1] 2020/12/16 07:53:25.854592 [DBG] 172.18.0.4:60104 - cid:4 - Client Ping Timer
payload-nats  | [1] 2020/12/16 07:53:25.854736 [DBG] 172.18.0.4:60104 - cid:4 - Delaying PING due to client activity 1s ago
payload-nats  | [1] 2020/12/16 07:53:26.443540 [TRC] STREAM: [Client:reception-nats-client] Received message from publisher subj=transfer-frames guid=5f29978b6481dca04c34c5
payload-nats  | [1] 2020/12/16 07:53:26.489072 [TRC] STREAM: [Client:decryption-nats-client] Delivering msg to subid=1, subject=transfer-frames, seq=43
payload-nats  | [1] 2020/12/16 07:53:26.515569 [TRC] STREAM: [Client:reception-nats-client] Acking Publisher subj=transfer-frames guid=5f29978b6481dca04c34c5
payload-nats  | [1] 2020/12/16 07:53:28.492752 [TRC] STREAM: [Client:reception-nats-client] Received message from publisher subj=transfer-frames guid=d3403830ac19fc511bddf7
payload-nats  | [1] 2020/12/16 07:53:28.538507 [TRC] STREAM: [Client:decryption-nats-client] Delivering msg to subid=1, subject=transfer-frames, seq=44
payload-nats  | [1] 2020/12/16 07:53:28.565947 [TRC] STREAM: [Client:reception-nats-client] Acking Publisher subj=transfer-frames guid=d3403830ac19fc511bddf7
payload-nats  | [1] 2020/12/16 07:53:30.380840 [TRC] STREAM: [Client:reception-nats-client] Received message from publisher subj=transfer-frames guid=102fd81b1858eea37a0d2
payload-nats  | [1] 2020/12/16 07:53:30.421683 [TRC] STREAM: [Client:decryption-nats-client] Delivering msg to subid=1, subject=transfer-frames, seq=45
payload-nats  | [1] 2020/12/16 07:53:30.461136 [TRC] STREAM: [Client:reception-nats-client] Acking Publisher subj=transfer-frames guid=102fd81b1858eea37a0d2
payload-nats  | [1] 2020/12/16 07:53:32.588232 [TRC] STREAM: [Client:reception-nats-client] Received message from publisher subj=transfer-frames guid=957e0b8265535ec89bc299
payload-nats  | [1] 2020/12/16 07:53:32.629640 [TRC] STREAM: [Client:decryption-nats-client] Delivering msg to subid=1, subject=transfer-frames, seq=46
payload-nats  | [1] 2020/12/16 07:53:32.658064 [TRC] STREAM: [Client:reception-nats-client] Acking Publisher subj=transfer-frames guid=957e0b8265535ec89bc299
payload-nats  | [1] 2020/12/16 07:53:32.806101 [DBG] STREAM: [Client:decryption-nats-client] Skipping redelivery to subid=1 due to missed client heartbeat
payload-nats  | [1] 2020/12/16 07:53:32.834025 [DBG] 172.18.0.5:58202 - cid:5 - Client Ping Timer
payload-nats  | [1] 2020/12/16 07:53:32.834101 [DBG] 172.18.0.5:58202 - cid:5 - Delaying PING due to client activity 0s ago
payload-nats  | [1] 2020/12/16 07:53:34.432198 [TRC] STREAM: [Client:reception-nats-client] Received message from publisher subj=transfer-frames guid=33959dcf52ed63b2a985b3
payload-nats  | [1] 2020/12/16 07:53:34.471026 [TRC] STREAM: [Client:decryption-nats-client] Delivering msg to subid=1, subject=transfer-frames, seq=47
payload-nats  | [1] 2020/12/16 07:53:34.499776 [TRC] STREAM: [Client:reception-nats-client] Acking Publisher subj=transfer-frames guid=33959dcf52ed63b2a985b3
payload-nats  | [1] 2020/12/16 07:53:36.577780 [TRC] STREAM: [Client:reception-nats-client] Received message from publisher subj=transfer-frames guid=97949909c90c19bf6954a3
payload-nats  | [1] 2020/12/16 07:53:36.612474 [TRC] STREAM: [Client:decryption-nats-client] Delivering msg to subid=1, subject=transfer-frames, seq=48
payload-nats  | [1] 2020/12/16 07:53:36.652132 [TRC] STREAM: [Client:reception-nats-client] Acking Publisher subj=transfer-frames guid=97949909c90c19bf6954a3
payload-nats  | [1] 2020/12/16 07:53:38.423492 [TRC] STREAM: [Client:reception-nats-client] Received message from publisher subj=transfer-frames guid=355efa268041d4dfa65165
payload-nats  | [1] 2020/12/16 07:53:38.462439 [TRC] STREAM: [Client:decryption-nats-client] Delivering msg to subid=1, subject=transfer-frames, seq=49
payload-nats  | [1] 2020/12/16 07:53:38.508983 [TRC] STREAM: [Client:reception-nats-client] Acking Publisher subj=transfer-frames guid=355efa268041d4dfa65165
payload-nats  | [1] 2020/12/16 07:53:40.574493 [TRC] STREAM: [Client:reception-nats-client] Received message from publisher subj=transfer-frames guid=7cf0eda1ac74e7ad479e8c
payload-nats  | [1] 2020/12/16 07:53:40.651108 [TRC] STREAM: [Client:decryption-nats-client] Delivering msg to subid=1, subject=transfer-frames, seq=50
payload-nats  | [1] 2020/12/16 07:53:40.684560 [TRC] STREAM: [Client:reception-nats-client] Acking Publisher subj=transfer-frames guid=7cf0eda1ac74e7ad479e8c
payload-nats  | [1] 2020/12/16 07:53:42.419887 [TRC] STREAM: [Client:reception-nats-client] Received message from publisher subj=transfer-frames guid=f1a1bd93ac18ec272533fb
payload-nats  | [1] 2020/12/16 07:53:42.453677 [TRC] STREAM: [Client:decryption-nats-client] Delivering msg to subid=1, subject=transfer-frames, seq=51
payload-nats  | [1] 2020/12/16 07:53:42.501990 [TRC] STREAM: [Client:reception-nats-client] Acking Publisher subj=transfer-frames guid=f1a1bd93ac18ec272533fb
payload-nats  | [1] 2020/12/16 07:53:42.806544 [DBG] STREAM: [Client:decryption-nats-client] Skipping redelivery to subid=1 due to missed client heartbeat
payload-nats  | [1] 2020/12/16 07:53:44.466004 [TRC] STREAM: [Client:reception-nats-client] Received message from publisher subj=transfer-frames guid=944739b7f720486489c891
payload-nats  | [1] 2020/12/16 07:53:44.511906 [TRC] STREAM: [Client:decryption-nats-client] Delivering msg to subid=1, subject=transfer-frames, seq=52
payload-nats  | [1] 2020/12/16 07:53:44.561597 [TRC] STREAM: [Client:reception-nats-client] Acking Publisher subj=transfer-frames guid=944739b7f720486489c891
payload-nats  | [1] 2020/12/16 07:53:46.383066 [TRC] STREAM: [Client:reception-nats-client] Received message from publisher subj=transfer-frames guid=7a9c5ec1e244a14d64dd6e
payload-nats  | [1] 2020/12/16 07:53:46.661633 [TRC] STREAM: [Client:decryption-nats-client] Delivering msg to subid=1, subject=transfer-frames, seq=53
payload-nats  | [1] 2020/12/16 07:53:46.687563 [TRC] STREAM: [Client:reception-nats-client] Acking Publisher subj=transfer-frames guid=7a9c5ec1e244a14d64dd6e
payload-nats  | [1] 2020/12/16 07:53:48.458261 [TRC] STREAM: [Client:reception-nats-client] Received message from publisher subj=transfer-frames guid=86405457ccbf63d12ffb1a
payload-nats  | [1] 2020/12/16 07:53:48.520005 [TRC] STREAM: [Client:decryption-nats-client] Delivering msg to subid=1, subject=transfer-frames, seq=54
payload-nats  | [1] 2020/12/16 07:53:48.546309 [TRC] STREAM: [Client:reception-nats-client] Acking Publisher subj=transfer-frames guid=86405457ccbf63d12ffb1a
payload-nats  | [1] 2020/12/16 07:53:50.814080 [TRC] STREAM: [Client:reception-nats-client] Received message from publisher subj=transfer-frames guid=500b2498e54e1fa7685975
payload-nats  | [1] 2020/12/16 07:53:50.877804 [TRC] STREAM: [Client:decryption-nats-client] Delivering msg to subid=1, subject=transfer-frames, seq=55
payload-nats  | [1] 2020/12/16 07:53:50.913031 [TRC] STREAM: [Client:reception-nats-client] Acking Publisher subj=transfer-frames guid=500b2498e54e1fa7685975
payload-nats  | [1] 2020/12/16 07:53:52.552506 [TRC] STREAM: [Client:reception-nats-client] Received message from publisher subj=transfer-frames guid=c4049487b1963fd35e86b
payload-nats  | [1] 2020/12/16 07:53:52.594419 [TRC] STREAM: [Client:decryption-nats-client] Delivering msg to subid=1, subject=transfer-frames, seq=56
payload-nats  | [1] 2020/12/16 07:53:52.630067 [TRC] STREAM: [Client:reception-nats-client] Acking Publisher subj=transfer-frames guid=c4049487b1963fd35e86b
payload-nats  | [1] 2020/12/16 07:53:52.806844 [DBG] STREAM: [Client:decryption-nats-client] Skipping redelivery to subid=1 due to missed client heartbeat
payload-nats  | [1] 2020/12/16 07:53:54.601870 [TRC] STREAM: [Client:reception-nats-client] Received message from publisher subj=transfer-frames guid=b8c12702f3709b2d94be15
payload-nats  | [1] 2020/12/16 07:53:54.644163 [TRC] STREAM: [Client:decryption-nats-client] Delivering msg to subid=1, subject=transfer-frames, seq=57
payload-nats  | [1] 2020/12/16 07:53:54.672444 [TRC] STREAM: [Client:reception-nats-client] Acking Publisher subj=transfer-frames guid=b8c12702f3709b2d94be15
payload-nats  | [1] 2020/12/16 07:53:56.407605 [TRC] STREAM: [Client:reception-nats-client] Received message from publisher subj=transfer-frames guid=2aa73fb7f8777424641145
payload-nats  | [1] 2020/12/16 07:53:56.443978 [TRC] STREAM: [Client:decryption-nats-client] Delivering msg to subid=1, subject=transfer-frames, seq=58
payload-nats  | [1] 2020/12/16 07:53:56.489222 [TRC] STREAM: [Client:reception-nats-client] Acking Publisher subj=transfer-frames guid=2aa73fb7f8777424641145
payload-nats  | [1] 2020/12/16 07:53:58.493213 [TRC] STREAM: [Client:reception-nats-client] Received message from publisher subj=transfer-frames guid=7bd953c27e7abf77763842
payload-nats  | [1] 2020/12/16 07:53:58.535492 [TRC] STREAM: [Client:decryption-nats-client] Delivering msg to subid=1, subject=transfer-frames, seq=59
payload-nats  | [1] 2020/12/16 07:53:58.564511 [TRC] STREAM: [Client:reception-nats-client] Acking Publisher subj=transfer-frames guid=7bd953c27e7abf77763842
payload-nats  | [1] 2020/12/16 07:54:00.538068 [TRC] STREAM: [Client:reception-nats-client] Received message from publisher subj=transfer-frames guid=a9235c5a0d89617b838c81
payload-nats  | [1] 2020/12/16 07:54:00.577013 [TRC] STREAM: [Client:decryption-nats-client] Delivering msg to subid=1, subject=transfer-frames, seq=60
payload-nats  | [1] 2020/12/16 07:54:00.616601 [TRC] STREAM: [Client:reception-nats-client] Acking Publisher subj=transfer-frames guid=a9235c5a0d89617b838c81
payload-nats  | [1] 2020/12/16 07:54:02.485987 [TRC] STREAM: [Client:reception-nats-client] Received message from publisher subj=transfer-frames guid=78cec9efb98d067392c622
payload-nats  | [1] 2020/12/16 07:54:02.526823 [TRC] STREAM: [Client:decryption-nats-client] Delivering msg to subid=1, subject=transfer-frames, seq=61
payload-nats  | [1] 2020/12/16 07:54:02.566261 [TRC] STREAM: [Client:reception-nats-client] Acking Publisher subj=transfer-frames guid=78cec9efb98d067392c622
payload-nats  | [1] 2020/12/16 07:54:02.807320 [DBG] STREAM: [Client:decryption-nats-client] Skipping redelivery to subid=1 due to missed client heartbeat
payload-nats  | [1] 2020/12/16 07:54:04.534052 [TRC] STREAM: [Client:reception-nats-client] Received message from publisher subj=transfer-frames guid=d5b168f25fd7af15df4a67
payload-nats  | [1] 2020/12/16 07:54:04.576670 [TRC] STREAM: [Client:decryption-nats-client] Delivering msg to subid=1, subject=transfer-frames, seq=62
payload-nats  | [1] 2020/12/16 07:54:04.624296 [TRC] STREAM: [Client:reception-nats-client] Acking Publisher subj=transfer-frames guid=d5b168f25fd7af15df4a67
payload-nats  | [1] 2020/12/16 07:54:06.664871 [TRC] STREAM: [Client:reception-nats-client] Received message from publisher subj=transfer-frames guid=a4b1c226e8a045dd4a1607
payload-nats  | [1] 2020/12/16 07:54:06.701433 [TRC] STREAM: [Client:decryption-nats-client] Delivering msg to subid=1, subject=transfer-frames, seq=63
payload-nats  | [1] 2020/12/16 07:54:06.741081 [TRC] STREAM: [Client:reception-nats-client] Acking Publisher subj=transfer-frames guid=a4b1c226e8a045dd4a1607
payload-nats  | [1] 2020/12/16 07:54:08.528285 [TRC] STREAM: [Client:reception-nats-client] Received message from publisher subj=transfer-frames guid=78a3292a880eb615045ca6
payload-nats  | [1] 2020/12/16 07:54:08.584581 [TRC] STREAM: [Client:decryption-nats-client] Delivering msg to subid=1, subject=transfer-frames, seq=64
payload-nats  | [1] 2020/12/16 07:54:08.624192 [TRC] STREAM: [Client:reception-nats-client] Acking Publisher subj=transfer-frames guid=78a3292a880eb615045ca6
payload-nats  | [1] 2020/12/16 07:54:10.472101 [TRC] STREAM: [Client:reception-nats-client] Received message from publisher subj=transfer-frames guid=6c39bd489d91cc5258c8b3
payload-nats  | [1] 2020/12/16 07:54:10.518106 [TRC] STREAM: [Client:decryption-nats-client] Delivering msg to subid=1, subject=transfer-frames, seq=65
payload-nats  | [1] 2020/12/16 07:54:10.557470 [TRC] STREAM: [Client:reception-nats-client] Acking Publisher subj=transfer-frames guid=6c39bd489d91cc5258c8b3
payload-nats  | [1] 2020/12/16 07:54:12.519767 [TRC] STREAM: [Client:reception-nats-client] Received message from publisher subj=transfer-frames guid=94e2c9db5fb0e74d49c491
payload-nats  | [1] 2020/12/16 07:54:12.559226 [TRC] STREAM: [Client:decryption-nats-client] Delivering msg to subid=1, subject=transfer-frames, seq=66
payload-nats  | [1] 2020/12/16 07:54:12.598758 [TRC] STREAM: [Client:reception-nats-client] Acking Publisher subj=transfer-frames guid=94e2c9db5fb0e74d49c491
payload-nats  | [1] 2020/12/16 07:54:12.807707 [DBG] STREAM: [Client:decryption-nats-client] Skipping redelivery to subid=1 due to missed client heartbeat
payload-nats  | [1] 2020/12/16 07:54:14.774669 [TRC] STREAM: [Client:reception-nats-client] Received message from publisher subj=transfer-frames guid=2b65c175d27a9c80d86b46
payload-nats  | [1] 2020/12/16 07:54:14.817429 [TRC] STREAM: [Client:decryption-nats-client] Delivering msg to subid=1, subject=transfer-frames, seq=67
payload-nats  | [1] 2020/12/16 07:54:14.856952 [TRC] STREAM: [Client:reception-nats-client] Acking Publisher subj=transfer-frames guid=2b65c175d27a9c80d86b46
payload-nats  | [1] 2020/12/16 07:54:16.513821 [TRC] STREAM: [Client:reception-nats-client] Received message from publisher subj=transfer-frames guid=59f772a314eb42c632f518
payload-nats  | [1] 2020/12/16 07:54:16.550413 [TRC] STREAM: [Client:decryption-nats-client] Delivering msg to subid=1, subject=transfer-frames, seq=68
payload-nats  | [1] 2020/12/16 07:54:16.590144 [TRC] STREAM: [Client:reception-nats-client] Acking Publisher subj=transfer-frames guid=59f772a314eb42c632f518
payload-nats  | [1] 2020/12/16 07:54:18.559730 [TRC] STREAM: [Client:reception-nats-client] Received message from publisher subj=transfer-frames guid=c83741a957d99ebb57bbd8
payload-nats  | [1] 2020/12/16 07:54:18.625172 [TRC] STREAM: [Client:decryption-nats-client] Delivering msg to subid=1, subject=transfer-frames, seq=69
payload-nats  | [1] 2020/12/16 07:54:18.654733 [TRC] STREAM: [Client:reception-nats-client] Acking Publisher subj=transfer-frames guid=c83741a957d99ebb57bbd8
payload-nats  | [1] 2020/12/16 07:54:20.440777 [TRC] STREAM: [Client:reception-nats-client] Received message from publisher subj=transfer-frames guid=4f0ea83057f49d7573c0fb
payload-nats  | [1] 2020/12/16 07:54:20.483498 [TRC] STREAM: [Client:decryption-nats-client] Delivering msg to subid=1, subject=transfer-frames, seq=70
payload-nats  | [1] 2020/12/16 07:54:20.513651 [TRC] STREAM: [Client:reception-nats-client] Acking Publisher subj=transfer-frames guid=4f0ea83057f49d7573c0fb
payload-nats  | [1] 2020/12/16 07:54:22.453016 [TRC] STREAM: [Client:reception-nats-client] Received message from publisher subj=transfer-frames guid=8fb174b663913f17d381d3
payload-nats  | [1] 2020/12/16 07:54:22.491614 [TRC] STREAM: [Client:decryption-nats-client] Delivering msg to subid=1, subject=transfer-frames, seq=71
payload-nats  | [1] 2020/12/16 07:54:22.521963 [TRC] STREAM: [Client:reception-nats-client] Acking Publisher subj=transfer-frames guid=8fb174b663913f17d381d3
payload-nats  | [1] 2020/12/16 07:54:22.808035 [DBG] STREAM: [Client:decryption-nats-client] Skipping redelivery to subid=1 due to missed client heartbeat
payload-nats  | [1] 2020/12/16 07:54:24.602429 [TRC] STREAM: [Client:reception-nats-client] Received message from publisher subj=transfer-frames guid=af2c36c67f2d8ff875879b
payload-nats  | [1] 2020/12/16 07:54:24.641358 [TRC] STREAM: [Client:decryption-nats-client] Delivering msg to subid=1, subject=transfer-frames, seq=72
payload-nats  | [1] 2020/12/16 07:54:24.680914 [TRC] STREAM: [Client:reception-nats-client] Acking Publisher subj=transfer-frames guid=af2c36c67f2d8ff875879b
payload-nats  | [1] 2020/12/16 07:54:26.648654 [TRC] STREAM: [Client:reception-nats-client] Received message from publisher subj=transfer-frames guid=c343299f7b8ecdf7c7aa07
payload-nats  | [1] 2020/12/16 07:54:26.691151 [TRC] STREAM: [Client:decryption-nats-client] Delivering msg to subid=1, subject=transfer-frames, seq=73
payload-nats  | [1] 2020/12/16 07:54:26.731492 [TRC] STREAM: [Client:reception-nats-client] Acking Publisher subj=transfer-frames guid=c343299f7b8ecdf7c7aa07
payload-nats  | [1] 2020/12/16 07:54:28.494142 [TRC] STREAM: [Client:reception-nats-client] Received message from publisher subj=transfer-frames guid=9028cc807f757ed168eb4
payload-nats  | [1] 2020/12/16 07:54:28.549392 [TRC] STREAM: [Client:decryption-nats-client] Delivering msg to subid=1, subject=transfer-frames, seq=74
payload-nats  | [1] 2020/12/16 07:54:28.581991 [TRC] STREAM: [Client:reception-nats-client] Acking Publisher subj=transfer-frames guid=9028cc807f757ed168eb4
payload-nats  | [1] 2020/12/16 07:54:30.474371 [TRC] STREAM: [Client:reception-nats-client] Received message from publisher subj=transfer-frames guid=1756c0e97fc9d8e6d1f4bd
payload-nats  | [1] 2020/12/16 07:54:30.515847 [TRC] STREAM: [Client:decryption-nats-client] Delivering msg to subid=1, subject=transfer-frames, seq=75
payload-nats  | [1] 2020/12/16 07:54:30.540395 [TRC] STREAM: [Client:reception-nats-client] Acking Publisher subj=transfer-frames guid=1756c0e97fc9d8e6d1f4bd
payload-nats  | [1] 2020/12/16 07:54:32.588546 [TRC] STREAM: [Client:reception-nats-client] Received message from publisher subj=transfer-frames guid=60f96f3b67784a1f73e72f
payload-nats  | [1] 2020/12/16 07:54:32.665808 [TRC] STREAM: [Client:decryption-nats-client] Delivering msg to subid=1, subject=transfer-frames, seq=76
payload-nats  | [1] 2020/12/16 07:54:32.724059 [TRC] STREAM: [Client:reception-nats-client] Acking Publisher subj=transfer-frames guid=60f96f3b67784a1f73e72f
payload-nats  | [1] 2020/12/16 07:54:32.808277 [DBG] STREAM: [Client:decryption-nats-client] Skipping redelivery to subid=1 due to missed client heartbeat
payload-nats  | [1] 2020/12/16 07:54:34.537266 [TRC] STREAM: [Client:reception-nats-client] Received message from publisher subj=transfer-frames guid=7ba066311e19e179a91089
payload-nats  | [1] 2020/12/16 07:54:34.590347 [TRC] STREAM: [Client:decryption-nats-client] Delivering msg to subid=1, subject=transfer-frames, seq=77
payload-nats  | [1] 2020/12/16 07:54:34.617071 [TRC] STREAM: [Client:reception-nats-client] Acking Publisher subj=transfer-frames guid=7ba066311e19e179a91089
payload-nats  | [1] 2020/12/16 07:54:36.479536 [TRC] STREAM: [Client:reception-nats-client] Received message from publisher subj=transfer-frames guid=eec13c2d05aae1637504b1
payload-nats  | [1] 2020/12/16 07:54:36.531909 [TRC] STREAM: [Client:decryption-nats-client] Delivering msg to subid=1, subject=transfer-frames, seq=78
payload-nats  | [1] 2020/12/16 07:54:36.571598 [TRC] STREAM: [Client:reception-nats-client] Acking Publisher subj=transfer-frames guid=eec13c2d05aae1637504b1
payload-nats  | [1] 2020/12/16 07:54:38.527431 [TRC] STREAM: [Client:reception-nats-client] Received message from publisher subj=transfer-frames guid=a58f321451d1fab3ce9a83
payload-nats  | [1] 2020/12/16 07:54:38.573499 [TRC] STREAM: [Client:decryption-nats-client] Delivering msg to subid=1, subject=transfer-frames, seq=79
payload-nats  | [1] 2020/12/16 07:54:38.613244 [TRC] STREAM: [Client:reception-nats-client] Acking Publisher subj=transfer-frames guid=a58f321451d1fab3ce9a83
payload-nats  | [1] 2020/12/16 07:54:40.670739 [TRC] STREAM: [Client:reception-nats-client] Received message from publisher subj=transfer-frames guid=33641e691e76275343ada3
payload-nats  | [1] 2020/12/16 07:54:40.714742 [TRC] STREAM: [Client:decryption-nats-client] Delivering msg to subid=1, subject=transfer-frames, seq=80
payload-nats  | [1] 2020/12/16 07:54:40.754354 [TRC] STREAM: [Client:reception-nats-client] Acking Publisher subj=transfer-frames guid=33641e691e76275343ada3
payload-nats  | [1] 2020/12/16 07:54:42.501172 [TRC] STREAM: [Client:reception-nats-client] Received message from publisher subj=transfer-frames guid=83c1c4342d0bf034ff0b9
payload-nats  | [1] 2020/12/16 07:54:42.539651 [TRC] STREAM: [Client:decryption-nats-client] Delivering msg to subid=1, subject=transfer-frames, seq=81
payload-nats  | [1] 2020/12/16 07:54:42.579348 [TRC] STREAM: [Client:reception-nats-client] Acking Publisher subj=transfer-frames guid=83c1c4342d0bf034ff0b9
payload-nats  | [1] 2020/12/16 07:54:42.808556 [DBG] STREAM: [Client:decryption-nats-client] Skipping redelivery to subid=1 due to missed client heartbeat
payload-nats  | [1] 2020/12/16 07:54:44.497203 [TRC] STREAM: [Client:reception-nats-client] Received message from publisher subj=transfer-frames guid=8ed3ca6a688a3db7edfde1
payload-nats  | [1] 2020/12/16 07:54:44.539523 [TRC] STREAM: [Client:decryption-nats-client] Delivering msg to subid=1, subject=transfer-frames, seq=82
payload-nats  | [1] 2020/12/16 07:54:44.568905 [TRC] STREAM: [Client:reception-nats-client] Acking Publisher subj=transfer-frames guid=8ed3ca6a688a3db7edfde1
payload-nats  | [1] 2020/12/16 07:54:46.507344 [TRC] STREAM: [Client:reception-nats-client] Received message from publisher subj=transfer-frames guid=d8582695c4166b0e605b97
payload-nats  | [1] 2020/12/16 07:54:46.547623 [TRC] STREAM: [Client:decryption-nats-client] Delivering msg to subid=1, subject=transfer-frames, seq=83
payload-nats  | [1] 2020/12/16 07:54:46.577388 [TRC] STREAM: [Client:reception-nats-client] Acking Publisher subj=transfer-frames guid=d8582695c4166b0e605b97
payload-nats  | [1] 2020/12/16 07:54:48.478923 [TRC] STREAM: [Client:reception-nats-client] Received message from publisher subj=transfer-frames guid=f0eeee4f43b7921735bafd
payload-nats  | [1] 2020/12/16 07:54:48.522442 [TRC] STREAM: [Client:decryption-nats-client] Delivering msg to subid=1, subject=transfer-frames, seq=84
payload-nats  | [1] 2020/12/16 07:54:48.561124 [TRC] STREAM: [Client:reception-nats-client] Acking Publisher subj=transfer-frames guid=f0eeee4f43b7921735bafd
payload-nats  | [1] 2020/12/16 07:54:50.604388 [TRC] STREAM: [Client:reception-nats-client] Received message from publisher subj=transfer-frames guid=6cffe00e271d763dfa2a35
payload-nats  | [1] 2020/12/16 07:54:50.647111 [TRC] STREAM: [Client:decryption-nats-client] Delivering msg to subid=1, subject=transfer-frames, seq=85
payload-nats  | [1] 2020/12/16 07:54:50.687445 [TRC] STREAM: [Client:reception-nats-client] Acking Publisher subj=transfer-frames guid=6cffe00e271d763dfa2a35
payload-nats  | [1] 2020/12/16 07:54:52.533081 [TRC] STREAM: [Client:reception-nats-client] Received message from publisher subj=transfer-frames guid=41c06dc7c5a85553632a3a
payload-nats  | [1] 2020/12/16 07:54:52.588706 [TRC] STREAM: [Client:decryption-nats-client] Delivering msg to subid=1, subject=transfer-frames, seq=86
payload-nats  | [1] 2020/12/16 07:54:52.620959 [TRC] STREAM: [Client:reception-nats-client] Acking Publisher subj=transfer-frames guid=41c06dc7c5a85553632a3a
payload-nats  | [1] 2020/12/16 07:54:52.808627 [DBG] STREAM: [Client:decryption-nats-client] Skipping redelivery to subid=1 due to missed client heartbeat
payload-nats  | [1] 2020/12/16 07:54:54.486800 [TRC] STREAM: [Client:reception-nats-client] Received message from publisher subj=transfer-frames guid=c9cd93415f18544b3dec1e
payload-nats  | [1] 2020/12/16 07:54:54.596800 [TRC] STREAM: [Client:decryption-nats-client] Delivering msg to subid=1, subject=transfer-frames, seq=87
payload-nats  | [1] 2020/12/16 07:54:54.630345 [TRC] STREAM: [Client:reception-nats-client] Acking Publisher subj=transfer-frames guid=c9cd93415f18544b3dec1e
payload-nats  | [1] 2020/12/16 07:54:56.644473 [TRC] STREAM: [Client:reception-nats-client] Received message from publisher subj=transfer-frames guid=42246cc5a653fa5816e9f8
payload-nats  | [1] 2020/12/16 07:54:56.704851 [TRC] STREAM: [Client:decryption-nats-client] Delivering msg to subid=1, subject=transfer-frames, seq=88
payload-nats  | [1] 2020/12/16 07:54:56.747584 [TRC] STREAM: [Client:reception-nats-client] Acking Publisher subj=transfer-frames guid=42246cc5a653fa5816e9f8
payload-nats  | [1] 2020/12/16 07:54:58.487835 [TRC] STREAM: [Client:reception-nats-client] Received message from publisher subj=transfer-frames guid=94b822fe1e74701a60aa9a
payload-nats  | [1] 2020/12/16 07:54:58.554769 [TRC] STREAM: [Client:decryption-nats-client] Delivering msg to subid=1, subject=transfer-frames, seq=89
payload-nats  | [1] 2020/12/16 07:54:58.584709 [TRC] STREAM: [Client:reception-nats-client] Acking Publisher subj=transfer-frames guid=94b822fe1e74701a60aa9a
payload-nats  | [1] 2020/12/16 07:54:58.651727 [DBG] 127.0.0.1:60290 - cid:2 - Client Ping Timer
payload-nats  | [1] 2020/12/16 07:54:58.651772 [DBG] 127.0.0.1:60290 - cid:2 - Delaying PING due to client activity 0s ago
payload-nats  | [1] 2020/12/16 07:54:58.673179 [DBG] 127.0.0.1:60288 - cid:1 - Client Ping Timer
payload-nats  | [1] 2020/12/16 07:54:58.673215 [DBG] 127.0.0.1:60288 - cid:1 - Delaying PING due to client activity 0s ago
payload-nats  | [1] 2020/12/16 07:54:58.771548 [DBG] 127.0.0.1:60292 - cid:3 - Client Ping Timer
payload-nats  | [1] 2020/12/16 07:54:58.771617 [DBG] 127.0.0.1:60292 - cid:3 - Delaying PING due to remote ping 2s ago
payload-nats  | [1] 2020/12/16 07:55:00.825547 [TRC] STREAM: [Client:reception-nats-client] Received message from publisher subj=transfer-frames guid=3b24b15a645961a80ffa84
payload-nats  | [1] 2020/12/16 07:55:00.887868 [TRC] STREAM: [Client:decryption-nats-client] Delivering msg to subid=1, subject=transfer-frames, seq=90
payload-nats  | [1] 2020/12/16 07:55:00.919374 [TRC] STREAM: [Client:reception-nats-client] Acking Publisher subj=transfer-frames guid=3b24b15a645961a80ffa84
payload-nats  | [1] 2020/12/16 07:55:02.479738 [TRC] STREAM: [Client:reception-nats-client] Received message from publisher subj=transfer-frames guid=ec8a36929bc6d2a54c9577
payload-nats  | [1] 2020/12/16 07:55:02.537827 [TRC] STREAM: [Client:decryption-nats-client] Delivering msg to subid=1, subject=transfer-frames, seq=91
payload-nats  | [1] 2020/12/16 07:55:02.569667 [TRC] STREAM: [Client:reception-nats-client] Acking Publisher subj=transfer-frames guid=ec8a36929bc6d2a54c9577
payload-nats  | [1] 2020/12/16 07:55:02.809063 [DBG] STREAM: [Client:decryption-nats-client] Skipping redelivery to subid=1 due to missed client heartbeat
payload-nats  | [1] 2020/12/16 07:55:04.481426 [TRC] STREAM: [Client:reception-nats-client] Received message from publisher subj=transfer-frames guid=383204036227d3bcf555b9
payload-nats  | [1] 2020/12/16 07:55:04.570898 [TRC] STREAM: [Client:decryption-nats-client] Delivering msg to subid=1, subject=transfer-frames, seq=92
payload-nats  | [1] 2020/12/16 07:55:04.603823 [TRC] STREAM: [Client:reception-nats-client] Acking Publisher subj=transfer-frames guid=383204036227d3bcf555b9
payload-nats  | [1] 2020/12/16 07:55:06.576856 [TRC] STREAM: [Client:reception-nats-client] Received message from publisher subj=transfer-frames guid=e14a7a4a1ae2aa1d2cd264
payload-nats  | [1] 2020/12/16 07:55:06.620605 [TRC] STREAM: [Client:decryption-nats-client] Delivering msg to subid=1, subject=transfer-frames, seq=93
payload-nats  | [1] 2020/12/16 07:55:06.646272 [TRC] STREAM: [Client:reception-nats-client] Acking Publisher subj=transfer-frames guid=e14a7a4a1ae2aa1d2cd264
payload-nats  | [1] 2020/12/16 07:55:08.526099 [TRC] STREAM: [Client:reception-nats-client] Received message from publisher subj=transfer-frames guid=2c1d9ed22cbe252dd56693
payload-nats  | [1] 2020/12/16 07:55:08.579173 [TRC] STREAM: [Client:decryption-nats-client] Delivering msg to subid=1, subject=transfer-frames, seq=94
payload-nats  | [1] 2020/12/16 07:55:08.608569 [TRC] STREAM: [Client:reception-nats-client] Acking Publisher subj=transfer-frames guid=2c1d9ed22cbe252dd56693
payload-nats  | [1] 2020/12/16 07:55:10.537388 [TRC] STREAM: [Client:reception-nats-client] Received message from publisher subj=transfer-frames guid=d25e5685836902b3836800
payload-nats  | [1] 2020/12/16 07:55:10.578670 [TRC] STREAM: [Client:decryption-nats-client] Delivering msg to subid=1, subject=transfer-frames, seq=95
payload-nats  | [1] 2020/12/16 07:55:10.625387 [TRC] STREAM: [Client:reception-nats-client] Acking Publisher subj=transfer-frames guid=d25e5685836902b3836800
payload-nats  | [1] 2020/12/16 07:55:12.519825 [TRC] STREAM: [Client:reception-nats-client] Received message from publisher subj=transfer-frames guid=798cb1784dd7f958fcab4a
payload-nats  | [1] 2020/12/16 07:55:12.570274 [TRC] STREAM: [Client:decryption-nats-client] Delivering msg to subid=1, subject=transfer-frames, seq=96
payload-nats  | [1] 2020/12/16 07:55:12.601761 [TRC] STREAM: [Client:reception-nats-client] Acking Publisher subj=transfer-frames guid=798cb1784dd7f958fcab4a
payload-nats  | [1] 2020/12/16 07:55:12.809589 [DBG] STREAM: [Client:decryption-nats-client] Skipping redelivery to subid=1 due to missed client heartbeat
payload-nats  | [1] 2020/12/16 07:55:14.485028 [TRC] STREAM: [Client:reception-nats-client] Received message from publisher subj=transfer-frames guid=f8528534ac0512b55012e6
payload-nats  | [1] 2020/12/16 07:55:14.519950 [TRC] STREAM: [Client:decryption-nats-client] Delivering msg to subid=1, subject=transfer-frames, seq=97
payload-nats  | [1] 2020/12/16 07:55:14.568311 [TRC] STREAM: [Client:reception-nats-client] Acking Publisher subj=transfer-frames guid=f8528534ac0512b55012e6
payload-nats  | [1] 2020/12/16 07:55:16.618316 [TRC] STREAM: [Client:reception-nats-client] Received message from publisher subj=transfer-frames guid=4cd50d601232920a0e836e
payload-nats  | [1] 2020/12/16 07:55:16.653137 [TRC] STREAM: [Client:decryption-nats-client] Delivering msg to subid=1, subject=transfer-frames, seq=98
payload-nats  | [1] 2020/12/16 07:55:16.701241 [TRC] STREAM: [Client:reception-nats-client] Acking Publisher subj=transfer-frames guid=4cd50d601232920a0e836e
payload-nats  | [1] 2020/12/16 07:55:18.561065 [TRC] STREAM: [Client:reception-nats-client] Received message from publisher subj=transfer-frames guid=62f676956ad73876a21828
payload-nats  | [1] 2020/12/16 07:55:18.611263 [TRC] STREAM: [Client:decryption-nats-client] Delivering msg to subid=1, subject=transfer-frames, seq=99
payload-nats  | [1] 2020/12/16 07:55:18.660669 [TRC] STREAM: [Client:reception-nats-client] Acking Publisher subj=transfer-frames guid=62f676956ad73876a21828
payload-nats  | [1] 2020/12/16 07:55:20.711790 [TRC] STREAM: [Client:reception-nats-client] Received message from publisher subj=transfer-frames guid=4b0b16fa9b965b6e6f97b5
payload-nats  | [1] 2020/12/16 07:55:20.753676 [TRC] STREAM: [Client:decryption-nats-client] Delivering msg to subid=1, subject=transfer-frames, seq=100
payload-nats  | [1] 2020/12/16 07:55:20.795750 [TRC] STREAM: [Client:reception-nats-client] Acking Publisher subj=transfer-frames guid=4b0b16fa9b965b6e6f97b5
payload-nats  | [1] 2020/12/16 07:55:22.555289 [TRC] STREAM: [Client:reception-nats-client] Received message from publisher subj=transfer-frames guid=76ad40f653ec7c86a58efb
payload-nats  | [1] 2020/12/16 07:55:22.611241 [TRC] STREAM: [Client:decryption-nats-client] Delivering msg to subid=1, subject=transfer-frames, seq=101
payload-nats  | [1] 2020/12/16 07:55:22.637250 [TRC] STREAM: [Client:reception-nats-client] Acking Publisher subj=transfer-frames guid=76ad40f653ec7c86a58efb
payload-nats  | [1] 2020/12/16 07:55:22.809903 [DBG] STREAM: [Client:decryption-nats-client] Skipping redelivery to subid=1 due to missed client heartbeat
payload-nats  | [1] 2020/12/16 07:55:24.807957 [TRC] STREAM: [Client:reception-nats-client] Received message from publisher subj=transfer-frames guid=57814c876b8b3f32f870ca
payload-nats  | [1] 2020/12/16 07:55:24.877580 [TRC] STREAM: [Client:decryption-nats-client] Delivering msg to subid=1, subject=transfer-frames, seq=102
payload-nats  | [1] 2020/12/16 07:55:24.917383 [TRC] STREAM: [Client:reception-nats-client] Acking Publisher subj=transfer-frames guid=57814c876b8b3f32f870ca
payload-nats  | [1] 2020/12/16 07:55:25.855164 [DBG] 172.18.0.4:60104 - cid:4 - Client Ping Timer
payload-nats  | [1] 2020/12/16 07:55:25.855255 [DBG] 172.18.0.4:60104 - cid:4 - Delaying PING due to client activity 1s ago
payload-nats  | [1] 2020/12/16 07:55:26.752639 [TRC] STREAM: [Client:reception-nats-client] Received message from publisher subj=transfer-frames guid=4729f849952bb91f272e5
payload-nats  | [1] 2020/12/16 07:55:26.794080 [TRC] STREAM: [Client:decryption-nats-client] Delivering msg to subid=1, subject=transfer-frames, seq=103
payload-nats  | [1] 2020/12/16 07:55:26.823040 [TRC] STREAM: [Client:reception-nats-client] Acking Publisher subj=transfer-frames guid=4729f849952bb91f272e5
payload-nats  | [1] 2020/12/16 07:55:28.596201 [TRC] STREAM: [Client:reception-nats-client] Received message from publisher subj=transfer-frames guid=df8da31780cecf0b6c4659
payload-nats  | [1] 2020/12/16 07:55:28.635474 [TRC] STREAM: [Client:decryption-nats-client] Delivering msg to subid=1, subject=transfer-frames, seq=104
payload-nats  | [1] 2020/12/16 07:55:28.675400 [TRC] STREAM: [Client:reception-nats-client] Acking Publisher subj=transfer-frames guid=df8da31780cecf0b6c4659
payload-nats  | [1] 2020/12/16 07:55:30.566945 [TRC] STREAM: [Client:reception-nats-client] Received message from publisher subj=transfer-frames guid=36c6c52412317bfc87ccf1
payload-nats  | [1] 2020/12/16 07:55:30.610195 [TRC] STREAM: [Client:decryption-nats-client] Delivering msg to subid=1, subject=transfer-frames, seq=105
payload-nats  | [1] 2020/12/16 07:55:30.658051 [TRC] STREAM: [Client:reception-nats-client] Acking Publisher subj=transfer-frames guid=36c6c52412317bfc87ccf1
payload-nats  | [1] 2020/12/16 07:55:32.588489 [TRC] STREAM: [Client:reception-nats-client] Received message from publisher subj=transfer-frames guid=f644d0aaf62b4dabb5fa77
payload-nats  | [1] 2020/12/16 07:55:32.635239 [TRC] STREAM: [Client:decryption-nats-client] Delivering msg to subid=1, subject=transfer-frames, seq=106
payload-nats  | [1] 2020/12/16 07:55:32.685000 [TRC] STREAM: [Client:reception-nats-client] Acking Publisher subj=transfer-frames guid=f644d0aaf62b4dabb5fa77
payload-nats  | [1] 2020/12/16 07:55:32.810381 [DBG] STREAM: [Client:decryption-nats-client] Skipping redelivery to subid=1 due to missed client heartbeat
payload-nats  | [1] 2020/12/16 07:55:32.834381 [DBG] 172.18.0.5:58202 - cid:5 - Client Ping Timer
payload-nats  | [1] 2020/12/16 07:55:32.834434 [DBG] 172.18.0.5:58202 - cid:5 - Delaying PING due to client activity 0s ago
payload-nats  | [1] 2020/12/16 07:55:34.535016 [TRC] STREAM: [Client:reception-nats-client] Received message from publisher subj=transfer-frames guid=4204ce8187adb16f4e364a
payload-nats  | [1] 2020/12/16 07:55:34.593064 [TRC] STREAM: [Client:decryption-nats-client] Delivering msg to subid=1, subject=transfer-frames, seq=107
payload-nats  | [1] 2020/12/16 07:55:34.618032 [TRC] STREAM: [Client:reception-nats-client] Acking Publisher subj=transfer-frames guid=4204ce8187adb16f4e364a
payload-nats  | [1] 2020/12/16 07:55:36.548329 [TRC] STREAM: [Client:reception-nats-client] Received message from publisher subj=transfer-frames guid=574059185cb24f9e890815
payload-nats  | [1] 2020/12/16 07:55:36.592893 [TRC] STREAM: [Client:decryption-nats-client] Delivering msg to subid=1, subject=transfer-frames, seq=108
payload-nats  | [1] 2020/12/16 07:55:36.632629 [TRC] STREAM: [Client:reception-nats-client] Acking Publisher subj=transfer-frames guid=574059185cb24f9e890815
payload-nats  | [1] 2020/12/16 07:55:38.630175 [TRC] STREAM: [Client:reception-nats-client] Received message from publisher subj=transfer-frames guid=84d8931637580c6c1eb13e
payload-nats  | [1] 2020/12/16 07:55:38.684227 [TRC] STREAM: [Client:decryption-nats-client] Delivering msg to subid=1, subject=transfer-frames, seq=109
payload-nats  | [1] 2020/12/16 07:55:38.710869 [TRC] STREAM: [Client:reception-nats-client] Acking Publisher subj=transfer-frames guid=84d8931637580c6c1eb13e
payload-nats  | [1] 2020/12/16 07:55:40.575521 [TRC] STREAM: [Client:reception-nats-client] Received message from publisher subj=transfer-frames guid=eb0c69afcee71f673e7732
payload-nats  | [1] 2020/12/16 07:55:40.634238 [TRC] STREAM: [Client:decryption-nats-client] Delivering msg to subid=1, subject=transfer-frames, seq=110
payload-nats  | [1] 2020/12/16 07:55:40.662232 [TRC] STREAM: [Client:reception-nats-client] Acking Publisher subj=transfer-frames guid=eb0c69afcee71f673e7732
payload-nats  | [1] 2020/12/16 07:55:42.621719 [TRC] STREAM: [Client:reception-nats-client] Received message from publisher subj=transfer-frames guid=e7099cd38d8bbdf4a771e
payload-nats  | [1] 2020/12/16 07:55:42.675912 [TRC] STREAM: [Client:decryption-nats-client] Delivering msg to subid=1, subject=transfer-frames, seq=111
payload-nats  | [1] 2020/12/16 07:55:42.704041 [TRC] STREAM: [Client:reception-nats-client] Acking Publisher subj=transfer-frames guid=e7099cd38d8bbdf4a771e
payload-nats  | [1] 2020/12/16 07:55:42.810626 [DBG] STREAM: [Client:decryption-nats-client] Skipping redelivery to subid=1 due to missed client heartbeat
payload-nats  | [1] 2020/12/16 07:55:44.670630 [TRC] STREAM: [Client:reception-nats-client] Received message from publisher subj=transfer-frames guid=88a1efcffe16a3045ce3a1
payload-nats  | [1] 2020/12/16 07:55:44.750517 [TRC] STREAM: [Client:decryption-nats-client] Delivering msg to subid=1, subject=transfer-frames, seq=112
payload-nats  | [1] 2020/12/16 07:55:44.796099 [TRC] STREAM: [Client:reception-nats-client] Acking Publisher subj=transfer-frames guid=88a1efcffe16a3045ce3a1
payload-nats  | [1] 2020/12/16 07:55:46.918684 [TRC] STREAM: [Client:reception-nats-client] Received message from publisher subj=transfer-frames guid=7e1621e5073cb42b252beb
payload-nats  | [1] 2020/12/16 07:55:46.958560 [TRC] STREAM: [Client:decryption-nats-client] Delivering msg to subid=1, subject=transfer-frames, seq=113
payload-nats  | [1] 2020/12/16 07:55:46.988016 [TRC] STREAM: [Client:reception-nats-client] Acking Publisher subj=transfer-frames guid=7e1621e5073cb42b252beb
payload-nats  | [1] 2020/12/16 07:55:52.810762 [DBG] STREAM: [Client:decryption-nats-client] Skipping redelivery to subid=1 due to missed client heartbeat
payload-nats  | [1] 2020/12/16 07:56:02.811357 [DBG] STREAM: [Client:decryption-nats-client] Skipping redelivery to subid=1 due to missed client heartbeat
payload-nats  | [1] 2020/12/16 07:56:12.811844 [DBG] STREAM: [Client:decryption-nats-client] Skipping redelivery to subid=1 due to missed client heartbeat
payload-nats  | [1] 2020/12/16 07:56:22.812222 [DBG] STREAM: [Client:decryption-nats-client] Skipping redelivery to subid=1 due to missed client heartbeat
payload-nats  | [1] 2020/12/16 07:56:32.812744 [DBG] STREAM: [Client:decryption-nats-client] Skipping redelivery to subid=1 due to missed client heartbeat
payload-nats  | [1] 2020/12/16 07:56:42.812998 [DBG] STREAM: [Client:decryption-nats-client] Skipping redelivery to subid=1 due to missed client heartbeat
payload-nats  | [1] 2020/12/16 07:56:52.813580 [DBG] STREAM: [Client:decryption-nats-client] Skipping redelivery to subid=1 due to missed client heartbeat

I read up a few other issues which deal with skipping redelivery due to missed heartbeat and the idea is that the streaming server has lost connection, but the underlying NATS connection is intact, hence the clientID is no longer recognised. Please correct me if I'm wrong.

I think my messages are not being delivered at all. Or atleast, the subscription is not picking them up, because my callback_handler does not get executed (I have included a few log statements in the callback, which do not get printed even once). I can't seem to find any mistake in the code that would lead to this.

After a while, the subscription disconnected altogether due to Heartbeat timeout.

I'm not at liberty to share my entire code, but I have included certain snippets which might help.

  1. docker compose file
  nats:
    image: nats-streaming:alpine
    container_name: payload-nats
    restart: always
    env_file: ./.env.dev
    command:
      - "--addr"
      - "0.0.0.0"
      - "-SDV"
      - "--cluster_id"
      - nats-streaming
      - "--debug=true"
      - "--dir"
      - /data/msg
      - "--http_port"
      - "8222"
      - "--port"
      - "4222"
      - "--store"
      - file
      - "--stan_debug=true"
    expose:
      - "4222"
    ports:
      - 4222:4222
      - 8222:8222
    networks:
      - app-network
  1. Subscription code
	async def error_callback_handler(self, exception):
			logger.info(str(exception))

	async def callback_handler(self, msg):
			logger.info("Received a message on subscription (seq: {}): {} | Count of frames in transit {}".format(msg.sequence, msg.data, self._TRANSIT_FRAME_COUNT))

	async def setup_NATS(self, loop, event, mutex):
		self.nc = NATS()
		self.sc = STAN()
		self._event = event
		self._mutex = mutex
		logger.info("Connecting to NATS service")
		await self.nc.connect(servers="nats://nats:4222", io_loop=loop)
		await self.sc.connect(self.clusterid, self.clientid, nats=self.nc)
		logger.info("Connected to NATS successfully")
		self.subscription = await self.sc.subscribe(subject=self.topic, cb=self.callback_handler, error_cb=self.error_callback_handler)
		await asyncio.sleep(2)
		logger.info("Subscribed to topic transfer-frames successfully {}".format(self.subscription))

I have tried with manual acks as well, and I get the same traceback. Any help/suggestion from the community is welcome. Thanks!

nswarup14 avatar Dec 16 '20 08:12 nswarup14