ADR graph for node is missing

[stefanobimbo]

I use 0.6.7 Some of my activated devices (ABP) in the ADR section don't show the RX and RX quality graphs. Instead of the graphs, there are two "Table has no column" labels. I don't know why this happens: ADR is ON in the device, ADR mode is Auto-adjust in device profile, uplink frames are correctly received. It seems that ADR is not working in this condition.

[gotthardp]

Either no data uplink frame arrived yet, or there is some error (indicated in the log) that prevents the data from displaying.

[stefanobimbo]

I don't see errors in log, just some uplink missed. A grep'ed log for C0000001, that is my device address, follows

lora@ubuntu-dev:/var/log/lorawan-server$ grep C0000001 debug.log 2019-09-04 00:26:15.731 [debug] <0.331.0>@lorawan_db_guard:trim_rxframes:478 Expired rxframes from [<<"00001A59">>,<<"10820113">>,<<"A1000101">>,<<"C0000001">>] 2019-09-04 00:59:16.963 [info] <0.349.0> [[email protected]:46313] SENT: PUBLISH(Q0, R0, D0, TopicName=ClasseC/out/C0000001, PacketId=undefined, Payload=<<"{"codr":"4/5","data":"10002828004900ED6001016400000700","datetime":"2019-09-03T22:59:16Z","datr":"SF7BW125","devaddr":"C0000001","fcnt":31,"freq":868.5,"lsnr":5.8,"port":1,"rssi":-99}">>) 2019-09-04 01:26:15.730 [debug] <0.331.0>@lorawan_db_guard:trim_rxframes:478 Expired rxframes from [<<"00001A59">>,<<"10820113">>,<<"A1000101">>,<<"C0000001">>] 2019-09-04 01:59:34.191 [info] <0.349.0> [[email protected]:46313] SENT: PUBLISH(Q0, R0, D0, TopicName=ClasseC/out/C0000001, PacketId=undefined, Payload=<<"{"codr":"4/5","data":"10002D2D004F00EE6001026400000700","datetime":"2019-09-03T23:59:33Z","datr":"SF7BW125","devaddr":"C0000001","fcnt":32,"freq":867.7,"lsnr":7.2,"port":1,"rssi":-99}">>) 2019-09-04 02:26:15.731 [debug] <0.331.0>@lorawan_db_guard:trim_rxframes:478 Expired rxframes from [<<"00001A59">>,<<"10820113">>,<<"A1000101">>,<<"C0000001">>] 2019-09-04 02:59:51.277 [info] <0.349.0> [[email protected]:46313] SENT: PUBLISH(Q0, R0, D0, TopicName=ClasseC/out/C0000001, PacketId=undefined, Payload=<<"{"codr":"4/5","data":"10003232004900EF6001036400000700","datetime":"2019-09-04T00:59:51Z","datr":"SF7BW125","devaddr":"C0000001","fcnt":33,"freq":867.9,"lsnr":7.8,"port":1,"rssi":-97}">>) 2019-09-04 03:26:15.730 [debug] <0.331.0>@lorawan_db_guard:trim_rxframes:478 Expired rxframes from [<<"00001A59">>,<<"10820113">>,<<"A1000101">>,<<"C0000001">>] 2019-09-04 04:00:08.118 [info] <0.349.0> [[email protected]:46313] SENT: PUBLISH(Q0, R0, D0, TopicName=ClasseC/out/C0000001, PacketId=undefined, Payload=<<"{"codr":"4/5","data":"10003737005500F06001046400000700","datetime":"2019-09-04T02:00:07Z","datr":"SF7BW125","devaddr":"C0000001","fcnt":34,"freq":867.5,"lsnr":5.8,"port":1,"rssi":-96}">>) 2019-09-04 04:26:15.732 [debug] <0.331.0>@lorawan_db_guard:trim_rxframes:478 Expired rxframes from [<<"00001A59">>,<<"10820113">>,<<"A1000101">>,<<"C0000001">>] 2019-09-04 04:59:25.116 [info] <0.349.0> [[email protected]:46313] SENT: PUBLISH(Q0, R0, D0, TopicName=ClasseC/out/C0000001, PacketId=undefined, Payload=<<"{"codr":"4/5","data":"10003C3C004400F16001056400000700","datetime":"2019-09-04T02:59:24Z","datr":"SF7BW125","devaddr":"C0000001","fcnt":35,"freq":867.7,"lsnr":6.5,"port":1,"rssi":-97}">>) 2019-09-04 05:26:15.730 [debug] <0.331.0>@lorawan_db_guard:trim_rxframes:478 Expired rxframes from [<<"00001A59">>,<<"10820113">>,<<"A100000A">>,<<"A1000101">>,<<"C0000001">>] 2019-09-04 05:59:41.391 [info] <0.349.0> [[email protected]:46313] SENT: PUBLISH(Q0, R0, D0, TopicName=ClasseC/out/C0000001, PacketId=undefined, Payload=<<"{"codr":"4/5","data":"10004141004900F26001066400000700","datetime":"2019-09-04T03:59:41Z","datr":"SF7BW125","devaddr":"C0000001","fcnt":36,"freq":867.1,"lsnr":7.2,"port":1,"rssi":-95}">>) 2019-09-04 06:26:15.732 [debug] <0.331.0>@lorawan_db_guard:trim_rxframes:478 Expired rxframes from [<<"00001A59">>,<<"10820113">>,<<"A1000001">>,<<"A1000007">>,<<"A1000009">>,<<"A100000A">>,<<"A100000D">>,<<"A100000E">>,<<"A1000011">>,<<"A1000019">>,<<"A100001A">>,<<"A100001B">>,<<"A100001C">>,<<"A100001D">>,<<"A1000101">>,<<"C0000001">>] 2019-09-04 06:59:57.169 [info] <0.349.0> [[email protected]:46313] SENT: PUBLISH(Q0, R0, D0, TopicName=ClasseC/out/C0000001, PacketId=undefined, Payload=<<"{"codr":"4/5","data":"10004646004F00F36001076400000700","datetime":"2019-09-04T04:59:56Z","datr":"SF7BW125","devaddr":"C0000001","fcnt":37,"freq":867.3,"lsnr":8.2,"port":1,"rssi":-96}">>) 2019-09-04 07:26:15.741 [debug] <0.331.0>@lorawan_db_guard:trim_rxframes:478 Expired rxframes from [<<"00001A59">>,<<"10820113">>,<<"A1000001">>,<<"A1000005">>,<<"A1000006">>,<<"A1000007">>,<<"A1000009">>,<<"A100000A">>,<<"A100000C">>,<<"A100000D">>,<<"A100000E">>,<<"A100000F">>,<<"A1000011">>,<<"A1000016">>,<<"A1000019">>,<<"A100001A">>,<<"A100001B">>,<<"A100001C">>,<<"A100001D">>,<<"A1000101">>,<<"C0000001">>] 2019-09-04 07:59:14.087 [info] <0.349.0> [[email protected]:46313] SENT: PUBLISH(Q0, R0, D0, TopicName=ClasseC/out/C0000001, PacketId=undefined, Payload=<<"{"codr":"4/5","data":"10004B4B004F00F46001086400000700","datetime":"2019-09-04T05:59:13Z","datr":"SF7BW125","devaddr":"C0000001","fcnt":38,"freq":868.1,"lsnr":8.0,"port":1,"rssi":-101}">>) 2019-09-04 08:26:15.732 [debug] <0.331.0>@lorawan_db_guard:trim_rxframes:478 Expired rxframes from [<<"00001A59">>,<<"10820113">>,<<"A1000001">>,<<"A1000009">>,<<"A100000A">>,<<"A100000C">>,<<"A1000011">>,<<"A1000012">>,<<"A1000013">>,<<"A100001D">>,<<"A1000101">>,<<"C0000001">>] 2019-09-04 08:59:31.541 [info] <0.349.0> [[email protected]:46313] SENT: PUBLISH(Q0, R0, D0, TopicName=ClasseC/out/C0000001, PacketId=undefined, Payload=<<"{"codr":"4/5","data":"10005050004900F56001096400000700","datetime":"2019-09-04T06:59:31Z","datr":"SF7BW125","devaddr":"C0000001","fcnt":39,"freq":868.3,"lsnr":7.8,"port":1,"rssi":-100}">>) 2019-09-04 09:26:15.733 [debug] <0.331.0>@lorawan_db_guard:trim_rxframes:478 Expired rxframes from [<<"00001A59">>,<<"10820113">>,<<"A1000001">>,<<"A1000002">>,<<"A1000009">>,<<"A100000A">>,<<"A100000C">>,<<"A1000011">>,<<"A1000013">>,<<"A1000015">>,<<"A1000016">>,<<"A100001E">>,<<"A1000101">>,<<"C0000001">>] 2019-09-04 09:59:49.392 [info] <0.349.0> [[email protected]:46313] SENT: PUBLISH(Q0, R0, D0, TopicName=ClasseC/out/C0000001, PacketId=undefined, Payload=<<"{"codr":"4/5","data":"10005555005500F660010A6400000700","datetime":"2019-09-04T07:59:49Z","datr":"SF7BW125","devaddr":"C0000001","fcnt":40,"freq":867.7,"lsnr":6.8,"port":1,"rssi":-96}">>) 2019-09-04 10:26:15.736 [debug] <0.331.0>@lorawan_db_guard:trim_rxframes:478 Expired rxframes from [<<"00001A59">>,<<"10820113">>,<<"A1000002">>,<<"A1000009">>,<<"A100000A">>,<<"A100000F">>,<<"A1000011">>,<<"A1000013">>,<<"A1000015">>,<<"A1000016">>,<<"A100001E">>,<<"A1000101">>,<<"C0000001">>] 2019-09-04 11:00:08.026 [debug] <0.12751.0>@lorawan_mac_commands:calculate_adr:351 Power C0000001: average rssi -104, power 2 -> down by -2 2019-09-04 11:00:08.026 [debug] <0.12751.0>@lorawan_mac_commands:build_fopts:77 C0000001 <- [{link_adr_req,5,1,255,0,0}] 2019-09-04 11:00:08.027 [info] <0.349.0> [[email protected]:46313] SENT: PUBLISH(Q0, R0, D0, TopicName=ClasseC/out/C0000001, PacketId=undefined, Payload=<<"{"codr":"4/5","data":"10005A5A004900F760010B6400000700","datetime":"2019-09-04T09:00:07Z","datr":"SF7BW125","devaddr":"C0000001","fcnt":41,"freq":867.9,"lsnr":2.8,"port":1,"rssi":-106}">>) 2019-09-04 11:26:15.736 [debug] <0.331.0>@lorawan_db_guard:trim_rxframes:478 Expired rxframes from [<<"00001A59">>,<<"10820113">>,<<"A1000002">>,<<"A1000006">>,<<"A1000009">>,<<"A100000A">>,<<"A100000C">>,<<"A100000E">>,<<"A100000F">>,<<"A1000011">>,<<"A1000013">>,<<"A1000015">>,<<"A1000016">>,<<"A100001E">>,<<"A1000101">>,<<"C0000001">>] 2019-09-04 11:59:25.361 [warning] <0.13520.0> node C0000001 {uplinks_missed,1} 2019-09-04 11:59:25.562 [debug] <0.13520.0>@lorawan_mac_commands:handle_fopts:16 C0000001 -> [{link_adr_ans,1,1,1}] 2019-09-04 11:59:25.562 [debug] <0.13520.0>@lorawan_mac_commands:handle_adr:167 LinkADRReq C0000001 succeeded 2019-09-04 11:59:25.564 [info] <0.349.0> [[email protected]:46313] SENT: PUBLISH(Q0, R0, D0, TopicName=ClasseC/out/C0000001, PacketId=undefined, Payload=<<"{"codr":"4/5","data":"10005F5F004F00F860010C6400000700","datetime":"2019-09-04T09:59:25Z","datr":"SF7BW125","devaddr":"C0000001","fcnt":43,"freq":867.5,"lsnr":-1.8,"port":1,"rssi":-105}">>) 2019-09-04 12:26:15.739 [debug] <0.331.0>@lorawan_db_guard:trim_rxframes:478 Expired rxframes from [<<"00001A59">>,<<"10820113">>,<<"A1000001">>,<<"A1000002">>,<<"A1000009">>,<<"A100000A">>,<<"A100000E">>,<<"A100000F">>,<<"A1000011">>,<<"A1000013">>,<<"A1000015">>,<<"A1000016">>,<<"A100001E">>,<<"A1000101">>,<<"C0000001">>] 2019-09-04 12:59:42.847 [info] <0.349.0> [[email protected]:46313] SENT: PUBLISH(Q0, R0, D0, TopicName=ClasseC/out/C0000001, PacketId=undefined, Payload=<<"{"codr":"4/5","data":"10006464005A00F960010D6400000700","datetime":"2019-09-04T10:59:42Z","datr":"SF7BW125","devaddr":"C0000001","fcnt":44,"freq":867.5,"lsnr":6.0,"port":1,"rssi":-96}">>) 2019-09-04 13:26:15.737 [debug] <0.331.0>@lorawan_db_guard:trim_rxframes:478 Expired rxframes from [<<"00001A59">>,<<"10820113">>,<<"A1000001">>,<<"A1000002">>,<<"A1000009">>,<<"A100000A">>,<<"A100000E">>,<<"A100000F">>,<<"A1000011">>,<<"A1000013">>,<<"A1000015">>,<<"A1000016">>,<<"A100001E">>,<<"A1000101">>,<<"C0000001">>] 2019-09-04 13:59:59.243 [info] <0.349.0> [[email protected]:46313] SENT: PUBLISH(Q0, R0, D0, TopicName=ClasseC/out/C0000001, PacketId=undefined, Payload=<<"{"codr":"4/5","data":"10005F5F004F00FA60010E6400000700","datetime":"2019-09-04T11:59:59Z","datr":"SF7BW125","devaddr":"C0000001","fcnt":45,"freq":868.5,"lsnr":6.2,"port":1,"rssi":-97}">>) 2019-09-04 14:26:15.735 [debug] <0.331.0>@lorawan_db_guard:trim_rxframes:478 Expired rxframes from [<<"00001A59">>,<<"10820113">>,<<"A1000001">>,<<"A1000002">>,<<"A1000009">>,<<"A100000A">>,<<"A100000E">>,<<"A1000011">>,<<"A1000013">>,<<"A1000015">>,<<"A1000016">>,<<"A100001E">>,<<"A1000101">>,<<"C0000001">>] 2019-09-04 14:47:43.776 [debug] <0.15604.0>@lorawan_handler:retransmit:236 C0000001 retransmitting 2 2019-09-04 14:47:46.936 [warning] <0.15605.0> node C0000001 {uplinks_missed,1} 2019-09-04 14:47:47.137 [info] <0.349.0> [[email protected]:46313] SENT: PUBLISH(Q0, R0, D0, TopicName=ClasseC/out/C0000001, PacketId=undefined, Payload=<<"{"codr":"4/5","data":"E425000000","datetime":"2019-09-04T12:47:46Z","datr":"SF7BW125","devaddr":"C0000001","fcnt":47,"freq":868.5,"lsnr":5.5,"port":1,"rssi":-98}">>) 2019-09-04 14:47:56.847 [info] <0.349.0> [[email protected]:46313] SENT: PUBLISH(Q0, R0, D0, TopicName=ClasseC/out/C0000001, PacketId=undefined, Payload=<<"{"codr":"4/5","data":"10005F5F005500FB60010E6400000700","datetime":"2019-09-04T12:47:56Z","datr":"SF7BW125","devaddr":"C0000001","fcnt":48,"freq":868.3,"lsnr":7.2,"port":1,"rssi":-100}">>) 2019-09-04 14:49:31.400 [info] <0.349.0> [[email protected]:46313] SENT: PUBLISH(Q0, R0, D0, TopicName=ClasseC/out/C0000001, PacketId=undefined, Payload=<<"{"codr":"4/5","data":"C01500002D002D2D0000000200010200000100000400","datetime":"2019-09-04T12:49:31Z","datr":"SF7BW125","devaddr":"C0000001","fcnt":49,"freq":867.9,"lsnr":7.2,"port":1,"rssi":-97}">>) 2019-09-04 14:51:07.942 [info] <0.349.0> [[email protected]:46313] SENT: PUBLISH(Q0, R0, D0, TopicName=ClasseC/out/C0000001, PacketId=undefined, Payload=<<"{"codr":"4/5","data":"11005FD5045A00B906720A0000000000","datetime":"2019-09-04T12:51:07Z","datr":"SF7BW125","devaddr":"C0000001","fcnt":50,"freq":867.9,"lsnr":7.2,"port":1,"rssi":-99}">>) 2019-09-04 14:52:48.478 [info] <0.349.0> [[email protected]:46313] SENT: PUBLISH(Q0, R0, D0, TopicName=ClasseC/out/C0000001, PacketId=undefined, Payload=<<"{"codr":"4/5","data":"A11309040E3503031FC904","datetime":"2019-09-04T12:52:48Z","datr":"SF7BW125","devaddr":"C0000001","fcnt":51,"freq":867.3,"lsnr":6.5,"port":1,"rssi":-96}">>) 2019-09-04 14:54:37.087 [info] <0.349.0> [[email protected]:46313] SENT: PUBLISH(Q0, R0, D0, TopicName=ClasseC/out/C0000001, PacketId=undefined, Payload=<<"{"codr":"4/5","data":"10005F5F005A000000010E6400000700","datetime":"2019-09-04T12:54:36Z","datr":"SF7BW125","devaddr":"C0000001","fcnt":52,"freq":867.5,"lsnr":5.2,"port":1,"rssi":-99}">>) 2019-09-04 14:56:40.756 [info] <0.349.0> [[email protected]:46313] SENT: PUBLISH(Q0, R0, D0, TopicName=ClasseC/out/C0000001, PacketId=undefined, Payload=<<"{"codr":"4/5","data":"C0150000310031310000000200010200000100000400","datetime":"2019-09-04T12:56:40Z","datr":"SF7BW125","devaddr":"C0000001","fcnt":53,"freq":868.5,"lsnr":5.0,"port":1,"rssi":-100}">>) 2019-09-04 14:59:14.520 [info] <0.349.0> [[email protected]:46313] SENT: PUBLISH(Q0, R0, D0, TopicName=ClasseC/out/C0000001, PacketId=undefined, Payload=<<"{"codr":"4/5","data":"10005A5A004F000000010F6400000700","datetime":"2019-09-04T12:59:14Z","datr":"SF7BW125","devaddr":"C0000001","fcnt":54,"freq":867.3,"lsnr":4.5,"port":1,"rssi":-101}">>) 2019-09-04 15:02:54.774 [info] <0.349.0> [[email protected]:46313] SENT: PUBLISH(Q0, R0, D0, TopicName=ClasseC/out/C0000001, PacketId=undefined, Payload=<<"{"codr":"4/5","data":"11005AD5045500B906720A0000000000","datetime":"2019-09-04T13:02:54Z","datr":"SF7BW125","devaddr":"C0000001","fcnt":55,"freq":868.3,"lsnr":5.5,"port":1,"rssi":-99}">>) 2019-09-04 15:08:43.770 [info] <0.349.0> [[email protected]:46313] SENT: PUBLISH(Q0, R0, D0, TopicName=ClasseC/out/C0000001, PacketId=undefined, Payload=<<"{"codr":"4/5","data":"A11309040F0907031FD904","datetime":"2019-09-04T13:08:43Z","datr":"SF7BW125","devaddr":"C0000001","fcnt":56,"freq":868.5,"lsnr":6.5,"port":1,"rssi":-95}">>) 2019-09-04 15:18:50.233 [info] <0.349.0> [[email protected]:46313] SENT: PUBLISH(Q0, R0, D0, TopicName=ClasseC/out/C0000001, PacketId=undefined, Payload=<<"{"codr":"4/5","data":"10005A5A0055000000010F6400000700","datetime":"2019-09-04T13:18:50Z","datr":"SF7BW125","devaddr":"C0000001","fcnt":57,"freq":867.7,"lsnr":5.5,"port":1,"rssi":-101}">>) 2019-09-04 15:26:15.739 [debug] <0.331.0>@lorawan_db_guard:trim_rxframes:478 Expired rxframes from [<<"00001A59">>,<<"10820113">>,<<"A1000002">>,<<"A1000009">>,<<"A100000A">>,<<"A100000E">>,<<"A1000011">>,<<"A1000013">>,<<"A1000015">>,<<"A1000016">>,<<"A100001E">>,<<"A1000101">>,<<"C0000001">>] 2019-09-04 15:37:31.857 [info] <0.349.0> [[email protected]:46313] SENT: PUBLISH(Q0, R0, D0, TopicName=ClasseC/out/C0000001, PacketId=undefined, Payload=<<"{"codr":"4/5","data":"C0150000360036360000000200010200000100000400","datetime":"2019-09-04T13:37:31Z","datr":"SF7BW125","devaddr":"C0000001","fcnt":58,"freq":868.1,"lsnr":7.0,"port":1,"rssi":-100}">>) 2019-09-04 15:59:29.948 [info] <0.349.0> [[email protected]:46313] SENT: PUBLISH(Q0, R0, D0, TopicName=ClasseC/out/C0000001, PacketId=undefined, Payload=<<"{"codr":"4/5","data":"10005555004F00010001106400000700","datetime":"2019-09-04T13:59:29Z","datr":"SF7BW125","devaddr":"C0000001","fcnt":59,"freq":867.3,"lsnr":2.2,"port":1,"rssi":-106}">>) 2019-09-04 16:26:15.736 [debug] <0.331.0>@lorawan_db_guard:trim_rxframes:478 Expired rxframes from [<<"00001A59">>,<<"10820113">>,<<"A1000001">>,<<"A1000002">>,<<"A1000009">>,<<"A100000A">>,<<"A100000E">>,<<"A1000011">>,<<"A1000013">>,<<"A1000015">>,<<"A1000016">>,<<"A1000019">>,<<"A100001E">>,<<"A1000101">>,<<"C0000001">>]

I don't see errors in event web page, but the uplink missed:

But I still have errors in ADR page and no graphs, altough the ADR is working well (see debug.log above at 11:00:08.026 ADR calculation and response at 11:59:25.562) [this makes me think ADR has its required list of frames to compute average power and RSSI values, or, at least, it has consumed received frames]

A strange thing for me is that I don't see frames for C0000001 in frames web page. They seem to be deleted by trim_rxframes (some times I see one or two frames, but they disappear after a trim). It is strange because in sys.config I have configured (default value) % amount of rxframes retained for each device/node {retained_rxframes, 50}, so I should see the last 50 frames for C0000001

Last hint: I added an undocumented param to sys.config to have ADR calculation faster (debugging...) {frames_before_adr, 5} I don't think it can be the cause, but who may know? ;-)

[altishchenko]

frames_before_adr has no input in the problem, but can you show the full debug.log, without grep for a particular node?

[stefanobimbo]

I don't have log of 4th sept due to log rotation. But I have log of 5th sept. Situation didn't changed: no ADR graph and no frames in frames web page even if I see frames in the log. debug.log

[stefanobimbo]

Sorry, I closed the issue for a mistake.

[altishchenko]

Sorry for the delay, but I can't see anything wrong in the debug.log. So, edit the profile for your device and set the FCnt Check field to 'disabled' first, see if it helps. If not, post the screenshot of your device's Handler(application) and, may be, connector as well. Does this device show up on the Dashboard timeline when it transmits? And if does, what color it is?

[stefanobimbo]

I set FCnt Check to disabled and restared my device yesterday at 17:45. My device, after a startup phase when sends tenths of frames, typically sends 2 frames per hour, at every change of hour. I saw the first 2 frames in frames web page, but after the trim for expired rx frames was run, they disappeared. The same happens cyclically: I see 2 frames, but they are deleted after expired rx frame runs. ADR graphs are never showed.

Frames in dashboard (C0000001) are present:

Attached debug log for yesterday ... debug_20190912.log

and today debug_20190913.log

As you asked for my connector and handler configuration I checked them with a little bit more carefulness. The only "strange" thing (if it is so) is that I use the same connector/handler couple for devices with different profiles. Can it be a problem?

Device profile:

Handler:

Connector:

[altishchenko]

Multiple profiles for a single Application/Handler are perfectly ok, this is how it is designed. What worries me most, is the trim function behaviour. I'll go through your logs more carefully with trim in mind. Can I have a look at your sys.config? Can you describe your setup in more detail? Like: OS, OTP version, docker if any, downloaded binary or built from source, etc

[stefanobimbo]

Setup:

OTP version:

No Docker, real Ubuntu server:

Lorawan server: downloaded binary 0.6.4 then upgraded to 0.6.6 (binary) and then upgraded again to 0.6.7 (binary)

sys.config: sys.config.txt

[altishchenko]

Ok, about this binary upgrade - how did you transfer the database? dbexport/dbimport or some other way?

[stefanobimbo]

Never done a dbexport/dbimport. I just stopped the running lorawan-server, installed with dpkg and then started a new version. I never deleted Mnesia db related files. Sometimes, after re-start, I had to reconfigure some, due to new tables/fields.

[altishchenko]

Oops, there is your problem then... There were some changes in the database structures. Some internal tables, not accessible from admin UI, were also changed inbetween the upgrades and trim behaviour can be the result of the inconsistency in rxframes table. Try this:

1. export the database
2. stop the server
3. Save your mnesia database somewhere and delete it from the server directory
4. start the server
5. import the database

Upon startup the server will create fresh mnesia setup and import the data correctly. Beware of the users.json file - either delete it from the exported files or, if you have any extra users defined, edit the file adding the field "pass" with clear text password to each one of them. Passwords are not exported, only the default user admin/admin gets created automatically. Importing extra users without "pass":"password" fields will result in no access for them.