thingsboard
[HELP] Modbus Poll period vs Report period and "Time stretching"
Describe the issue I am reading several of the same devices form several gateways an did a test with different poll vs submission periods, it looks like the timestamps are saved in internal memory of the gateway, but -somehow- server time is applied when the data is submitted to the server.
The gateways all read similar devices, but some gateways may have more devices on their modbus networks ( 7 inverters for GW 1, 6 inverters for GW 2 ,4 for GW3, 3 for GW 4, 5 for GW5 )
I am not sure if this is a quirk of the gateway, server or widgets :(
I have 5 Gateways set up GW1 : poll period : 15 seconds, submission period 30 seconds GW2 : poll period : 15 seconds, submission period 30 seconds GW3 : poll period : 30 seconds, submission period 30 seconds GW4 : poll period : 12 seconds, submission period 30 seconds GW5 : poll period : 10 seconds, submission period 30 seconds ( GW 5's RS485 to TCP converter went offline so its just sending the same value )
When GW 1,2,4 and 5 submit data it looks like the data is "stretched", when GW3 sends it s data the data is 'correct'
Configuration (Attach your configuration file)
GW config is similar fo rall gateways :
{ "thingsboard": { "host": "<My server>", "port": 1883, "remoteShell": true, "remoteConfiguration": true, "latencyDebugMode": false, "statistics": { "enable": true, "enableCustom": true, "statsSendPeriodInSeconds": 60, "customStatsSendPeriodInSeconds": 3600 }, "deviceFiltering": { "enable": false, "filterFile": "list.json" }, "maxPayloadSizeBytes": 8196, "minPackSendDelayMS": 500, "minPackSizeToSend": 500, "checkConnectorsConfigurationInSeconds": 60, "handleDeviceRenaming": true, "security": { "type": "accessToken", "accessToken": "<Access token>" }, "qos": 1, "reportStrategy": { "type": "ON_CHANGE" }, "checkingDeviceActivity": { "checkDeviceInactivity": false, "inactivityTimeoutSeconds": 300, "inactivityCheckPeriodSeconds": 10 }, "rateLimits": "DEFAULT_TELEMETRY_RATE_LIMIT", "dpRateLimits": "DEFAULT_TELEMETRY_DP_RATE_LIMIT", "messagesRateLimits": "DEFAULT_MESSAGES_RATE_LIMIT", "deviceMessagesRateLimits": "DEFAULT_MESSAGES_RATE_LIMIT", "deviceRateLimits": "DEFAULT_TELEMETRY_RATE_LIMIT", "deviceDpRateLimits": "DEFAULT_TELEMETRY_DP_RATE_LIMIT", "ts": 1763408892992 }, "storage": { "type": "memory", "read_records_count": 100, "max_records_count": 100000, "data_folder_path": "./data/", "max_file_count": 10, "max_read_records_count": 10, "max_records_per_file": 10000, "data_file_path": "./data/", "messages_ttl_check_in_hours": 1, "messages_ttl_in_days": 7, "ts": 1763408892992 }, "grpc": { "enabled": false, "serverPort": 9595, "keepAliveTimeMs": 10001, "keepAliveTimeoutMs": 5000, "keepAlivePermitWithoutCalls": true, "maxPingsWithoutData": 0, "minTimeBetweenPingsMs": 10000, "minPingIntervalWithoutDataMs": 5000 }, "connectors": [ { "type": "modbus", "name": "Modbus Inverters", "configuration": "modbusInverters.json" }, { "type": "modbus", "name": "Modbus Sensors", "configuration": "modbusSensors.json" }, { "type": "modbus", "name": "Modbus AC Meters", "configuration": "modbusAcMeters.json" }, { "type": "modbus", "name": "Modbus DC Meters", "configuration": "modbusDcMeters.json" } ] }
Connector name : There are 4 connectors being used because i found the device sometimes does not submit data if there is a problematic device in the list , not sure if this would affect the queuing/ memory storage on the gateway )
I used the memory logs to cross check data read on GW3 with the telemetry as seen on the server
[Modbus Connector]
{ "name": "Modbus AC Meters", "id": "<Device ID>", "master": { "slaves": [ { "host": "192.168.1.30", "port": 502, "method": "socket", "unitId": 71, "deviceName": "NRES-MET-001", "deviceType": "RE Meter", "timeout": 3, "byteOrder": "BIG", "wordOrder": "BIG", "retries": true, "retryOnEmpty": true, "retryOnInvalid": true, "pollPeriod": 30000, // < this varies between gateways > "connectAttemptTimeMs": 1000, "connectAttemptCount": 5, "waitAfterFailedAttemptsMs": 30000, "reportStrategy": { "type": "ON_REPORT_PERIOD", "reportPeriod": 30000 }, "type": "tcp", "attributes": [ { "tag": "Scaling.Current.Enum", "type": "16int", "address": 137, "objectsCount": 1, "functionCode": 3 }, { "tag": "Scaling.Voltage.Enum", "type": "16int", "address": 138, "objectsCount": 1, "functionCode": 3 }, { "tag": "Scaling.Power.Enum", "type": "16int", "address": 139, "objectsCount": 1, "functionCode": 3 }, { "tag": "Scaling.Energy.Enum", "type": "16int", "address": 140, "objectsCount": 1, "functionCode": 3 } ], "timeseries": [ { "tag": "Scaling.Current.Enum", "type": "16int", "address": 137, "objectsCount": 1, "functionCode": 3 }, { "tag": "Scaling.Voltage.Enum", "type": "16int", "address": 138, "objectsCount": 1, "functionCode": 3 }, { "tag": "Scaling.Power.Enum", "type": "16int", "address": 139, "objectsCount": 1, "functionCode": 3 }, { "tag": "Scaling.Energy.Enum", "type": "16int", "address": 140, "objectsCount": 1, "functionCode": 3 }, { "tag": "Energy.Real.Net", "type": "32int", "address": 0, "objectsCount": 2, "functionCode": 3 }, { "tag": "Energy.Real.Import", "type": "32uint", "address": 2, "objectsCount": 2, "functionCode": 3 }, { "tag": "Energy.Real.Export", "type": "32uint", "address": 4, "objectsCount": 2, "functionCode": 3 }, { "tag": "Power.Real.Instantaneous", "type": "16int", "address": 20, "objectsCount": 1, "functionCode": 3, "multiplier": -100 }, { "tag": "Voltage.LL.Average", "type": "16uint", "address": 24, "objectsCount": 1, "functionCode": 3, "divider": 10 }, { "tag": "Voltage.LN.Average", "type": "16uint", "address": 25, "objectsCount": 1, "functionCode": 3, "divider": 10 }, { "tag": "Current.Average", "type": "16uint", "address": 26, "objectsCount": 1, "functionCode": 3 }, { "tag": "Grid.Frequency", "type": "16uint", "address": 27, "objectsCount": 1, "functionCode": 3, "divider": 100 }, { "tag": "Power.Real.PhaseA", "type": "16int", "address": 90, "objectsCount": 1, "functionCode": 3, "multiplier": -1000 }, { "tag": "Power.Real.PhaseB", "type": "16int", "address": 91, "objectsCount": 1, "functionCode": 3, "multiplier": -1000 }, { "tag": "Power.Real.PhaseC", "type": "16int", "address": 92, "objectsCount": 1, "functionCode": 3, "multiplier": -1000 }, { "tag": "Energy.Real.Import.Float", "type": "32float", "address": 258, "objectsCount": 2, "functionCode": 3 }, { "tag": "Energy.Real.Export.Float", "type": "32float", "address": 260, "objectsCount": 2, "functionCode": 3 }, { "tag": "Power.Real.Instantaneous.Float", "type": "32float", "address": 276, "objectsCount": 2, "functionCode": 3, "multiplier": -1000 }, { "tag": "Energy.Reactive.Import1", "type": "32uint", "address": 6, "objectsCount": 2, "functionCode": 3 }, { "tag": "Energy.Reactive.Import2", "type": "32uint", "address": 8, "objectsCount": 2, "functionCode": 3 }, { "tag": "Energy.Reactive.Export1", "type": "32uint", "address": 10, "objectsCount": 2, "functionCode": 3 }, { "tag": "Energy.Reactive.Export2", "type": "32uint", "address": 12, "objectsCount": 2, "functionCode": 3 }, { "tag": "Energy.Reactive.Net", "type": "32int", "address": 14, "objectsCount": 2, "functionCode": 3 }, { "tag": "Energy.Reactive.Import", "type": "32uint", "address": 16, "objectsCount": 2, "functionCode": 3 }, { "tag": "Energy.Reactive.Export", "type": "32uint", "address": 18, "objectsCount": 2, "functionCode": 3 }, { "tag": "Power.Reactive.Instantaneous", "type": "16int", "address": 21, "objectsCount": 1, "functionCode": 3, "multiplier": -100 }, { "tag": "Power.Apparent.Instantaneous", "type": "16uint", "address": 22, "objectsCount": 1, "functionCode": 3, "multiplier": -100 }, { "tag": "Power.Factor.Total", "type": "16int", "address": 23, "objectsCount": 1, "functionCode": 3, "multiplier": 0.0001 }, { "tag": "Power.Real.Demand.Present", "type": "16int", "address": 28, "objectsCount": 1, "functionCode": 3, "multiplier": -100 }, { "tag": "Power.Reactive.Demand.Present", "type": "16int", "address": 29, "objectsCount": 1, "functionCode": 3, "multiplier": -100 }, { "tag": "Power.Apparent.Demand.Present", "type": "16int", "address": 30, "objectsCount": 1, "functionCode": 3, "multiplier": -100 }, { "tag": "Power.Real.Demand.Max.Import", "type": "16int", "address": 31, "objectsCount": 1, "functionCode": 3, "multiplier": -100 }, { "tag": "Power.Reactive.Demand.Max.Import", "type": "16int", "address": 32, "objectsCount": 1, "functionCode": 3, "multiplier": -100 }, { "tag": "Power.Apparent.Demand.Max.Import", "type": "16int", "address": 33, "objectsCount": 1, "functionCode": 3, "multiplier": -100 }, { "tag": "Power.Real.Demand.Max.Export", "type": "16int", "address": 34, "objectsCount": 1, "functionCode": 3, "multiplier": -100 }, { "tag": "Power.Reactive.Demand.Max.Export", "type": "16int", "address": 35, "objectsCount": 1, "functionCode": 3, "multiplier": -100 }, { "tag": "Power.Apparent.Demand.Max.Export", "type": "16int", "address": 36, "objectsCount": 1, "functionCode": 3, "multiplier": -100 }, { "tag": "Pulse.Import.Real", "type": "32uint", "address": 38, "objectsCount": 2, "functionCode": 3 }, { "tag": "Pulse.Export.Real", "type": "32uint", "address": 40, "objectsCount": 2, "functionCode": 3 }, { "tag": "Energy.Real.PhaseA.Import", "type": "32uint", "address": 42, "objectsCount": 2, "functionCode": 3 }, { "tag": "Energy.Real.PhaseB.Import", "type": "32uint", "address": 44, "objectsCount": 2, "functionCode": 3 }, { "tag": "Energy.Real.PhaseC.Import", "type": "32uint", "address": 46, "objectsCount": 2, "functionCode": 3 }, { "tag": "Energy.Real.PhaseA.Export", "type": "32uint", "address": 48, "objectsCount": 2, "functionCode": 3 }, { "tag": "Energy.Real.PhaseB.Export", "type": "32uint", "address": 50, "objectsCount": 2, "functionCode": 3 }, { "tag": "Energy.Real.PhaseC.Export", "type": "32uint", "address": 52, "objectsCount": 2, "functionCode": 3 }, { "tag": "Energy.Apparent.PhaseA.Import", "type": "32uint", "address": 78, "objectsCount": 2, "functionCode": 3 }, { "tag": "Energy.Apparent.PhaseB.Import", "type": "32uint", "address": 80, "objectsCount": 2, "functionCode": 3 }, { "tag": "Energy.Apparent.PhaseC.Import", "type": "32uint", "address": 82, "objectsCount": 2, "functionCode": 3 }, { "tag": "Energy.Apparent.PhaseA.Export", "type": "32uint", "address": 84, "objectsCount": 2, "functionCode": 3 }, { "tag": "Energy.Apparent.PhaseB.Export", "type": "32uint", "address": 86, "objectsCount": 2, "functionCode": 3 }, { "tag": "Energy.Apparent.PhaseC.Export", "type": "32uint", "address": 88, "objectsCount": 2, "functionCode": 3 }, { "tag": "Power.Reactive.PhaseA", "type": "16int", "address": 93, "objectsCount": 1, "functionCode": 3, "multiplier": -1000 }, { "tag": "Power.Reactive.PhaseB", "type": "16int", "address": 94, "objectsCount": 1, "functionCode": 3, "multiplier": -1000 }, { "tag": "Power.Reactive.PhaseC", "type": "16int", "address": 95, "objectsCount": 1, "functionCode": 3, "multiplier": -1000 }, { "tag": "Power.Apparent.PhaseA", "type": "16uint", "address": 96, "objectsCount": 1, "functionCode": 3, "multiplier": -1000 }, { "tag": "Power.Apparent.PhaseB", "type": "16uint", "address": 97, "objectsCount": 1, "functionCode": 3, "multiplier": -1000 }, { "tag": "Power.Apparent.PhaseC", "type": "16uint", "address": 98, "objectsCount": 1, "functionCode": 3, "multiplier": -1000 }, { "tag": "Power.Factor.PhaseA", "type": "16int", "address": 99, "objectsCount": 1, "functionCode": 3, "multiplier": 0.0001 }, { "tag": "Power.Factor.PhaseB", "type": "16int", "address": 100, "objectsCount": 1, "functionCode": 3, "multiplier": 0.0001 }, { "tag": "Power.Factor.PhaseC", "type": "16int", "address": 101, "objectsCount": 1, "functionCode": 3, "multiplier": 0.0001 }, { "tag": "Voltage.LL.AB", "type": "16uint", "address": 102, "objectsCount": 1, "functionCode": 3, "divider": 10 }, { "tag": "Voltage.LL.BC", "type": "16uint", "address": 103, "objectsCount": 1, "functionCode": 3, "divider": 10 }, { "tag": "Voltage.LL.AC", "type": "16uint", "address": 104, "objectsCount": 1, "functionCode": 3, "divider": 10 }, { "tag": "Voltage.LN.A", "type": "16uint", "address": 105, "objectsCount": 1, "functionCode": 3, "divider": 10 }, { "tag": "Voltage.LN.B", "type": "16uint", "address": 106, "objectsCount": 1, "functionCode": 3, "divider": 10 }, { "tag": "Voltage.LN.C", "type": "16uint", "address": 107, "objectsCount": 1, "functionCode": 3, "divider": 10 }, { "tag": "Current.PhaseA", "type": "16uint", "address": 108, "objectsCount": 1, "functionCode": 3 }, { "tag": "Current.PhaseB", "type": "16uint", "address": 109, "objectsCount": 1, "functionCode": 3 }, { "tag": "Current.PhaseC", "type": "16uint", "address": 110, "objectsCount": 1, "functionCode": 3 } ], "attributeUpdates": [], "rpc": [] }
Error traceback (If it was raised): No Error
Versions (please complete the following information):
- OS:Ubuntu on AWS
- Thingsboard PE 4.2
- Python version on Gateway: 3.13.5
- Gateway version: 3.7.8 and 3.7.9 ( I see the same behavior on both versions when i change the ration between Polling and Submission intervals)
Additional context Does the Gateway add timestamps to the data ( in the memory queue ) when it Polls ( so timestamp is Polling time ) and gets data from the device, or when it submits data to the server ( so data timestamp is submission timestamp )
Based on the reporting strategy docs, polling at 15S intervals, but reporting at 30S should only send the latest value, I wonder if the "laggy" data could be a backup in transmitting, or rate limiting ?
IS there a way to see the "message backlog" / queue size in the gateway ?
I'm not sure if this is a TB server / queue issue or a rule chain ignoring the timestamp in the message OR the gateway somehow not including the timestamp not message it sends to the Server ( I see the timestamp in the logs when it saves to memory so i think this might be unlikely )
Log from the Gateway for 2x submissions when the polling period is 15S and submission is 30S: it submits every 30 seconds and only seems to have the latest datapoint ( will need to create a modbus device which adds a local TS to its registers so a TS can be a telemetry point so we can verify this, but pretty sure it is the case ): 15S poll 30s submit.txt
We also found, when I rebooted the gateway, the memory queue is cleared ( its memory queue, not File or database ) and it immediately sends a "current" data point ( we saw a jump" in teh telemetry value , a value which should l have been low at the time went from high to low ( i think it was PV power which should have been low -at night- was high - daytime- and after the reboot it was the 'real' current value), so maybe rate limiting or slow sending, or race condition / blocking condition between all my modbus connectors ?
Follow up
We removed a bunch of register reads ( going from ~68 or so, down to 7 ) and the data now lines up ( or is delivered on time ).. will keep poking and trying to figure out why it behaves like this
I have set the GW to use File Storage and can see the file steadily growing ( is it a FIFO which should get emptier or more like a circular buffer and 'position' indicates the next data to send allowing the file/memory grow to max size and it will b e cleared at some point ? )
cat state {"file": "data_1763564021490.txt", "position": 1338} {"file": "data_1763564021490.txt", "position": 1339} about 2 hours later : {"file": "data_1763564021490.txt", "position": 3048}
the data_<>.txt also seems to be growing. ls -l data* -rw-r--r-- 1 thingsboard_gateway thingsboard_gateway 2139675 Nov 19 09:34 data_1763564021490.txt
If I 'tail -f data_1763564021490.txt | base64 -d ' i can see the system adding data to the memory file / queue ?
Can confirm this is happening to us too, with ThingsBoard Cloud and thingsboard-gateway software 3.7.9 and Modbus connectors.
Pretty frustrating bug since the data we get with the timestamps is incorrect.
