[HELP] Unable to configure rate limits in version 3.4.5

[tomhaddad]

Describe the issue Since upgrading to version 3.4.5, the newly introduced rate limiting is preventing normal functionality of the gateway. Without setting any extra configuration we are hitting the 8 messages per second limit constantly.

I have tried setting the TB_GW_RATE_LIMITS environment variable, and added the following to tb_gateway.json:

"rateLimits": "1000:1;10000:60;60000:3600;",

Neither of these seem to change the default rate limit, as I am seeing these logs:

|DEBUG| - [tb_device_mqtt.py] - tb_device_mqtt - check_limit_reached - 198 - Rate limit exceeded for 1 second
|DEBUG| - [tb_device_mqtt.py] - tb_device_mqtt - check_limit_reached - 199 - Queue size: 8

This is a problem, as it is preventing most messages from being processed by the gateway.

Configuration (Attach your configuration file)

{
  "thingsboard": {
    "host": "thingsboard",
    "port": 8883,
    "remoteShell": false,
    "remoteConfiguration": true,
    "statistics": {
      "enable": true,
      "statsSendPeriodInSeconds": 3600
    },
    "deviceFiltering": {
      "enable": false,
      "filterFile": "list.json"
    },
    "maxPayloadSizeBytes": 1024,
    "minPackSendDelayMS": 200,
    "minPackSizeToSend": 500,
    "checkConnectorsConfigurationInSeconds": 60,
    "handleDeviceRenaming": true,
    "security": {
      "type": "accessToken",
      "accessToken": "access_token"
    },
    "qos": 1,
    "rateLimits": "1000:1;10000:60;60000:3600;",
    "checkingDeviceActivity": {
      "checkDeviceInactivity": false,
      "inactivityTimeoutSeconds": 120,
      "inactivityCheckPeriodSeconds": 10
    }
  },
  "storage": {
    "type": "memory",
    "read_records_count": 100,
    "max_records_count": 100000
  },
  "grpc": {
    "enabled": false,
    "serverPort": 9595,
    "keepaliveTimeMs": 10000,
    "keepaliveTimeoutMs": 5000,
    "keepalivePermitWithoutCalls": true,
    "maxPingsWithoutData": 0,
    "minTimeBetweenPingsMs": 10000,
    "minPingIntervalWithoutDataMs": 5000,
    "keepAliveTimeMs": 10000,
    "keepAliveTimeoutMs": 5000
  },
  "connectors": [
    {
      "name": "MQTT Broker Connector",
      "type": "mqtt",
      "configuration": "mqtt.json"
    }
  ]
}

Versions (please complete the following information):

• Docker
• Thingsboard IoT Gateway version 3.4.5

[github-actions[bot]]

Hi @tomhaddad.
Thank you for your interest in ThingsBoard IoT Gateway.
Your issue was registered, please wait for response from engineer.

IOTGW-180

[imbeacon]

Hi @tomhaddad,

Unfortunately, you are right and rate limits now - limits the data sending, it is required by ThingsBoard cloud servers. For now we have the manual rate limits configuration, but in future releases of ThingsBoard and the gateway - we are going to add automatic getting of rate limits from the server. Sorry for the temporary inconvenience.

[tomhaddad]

Hi @imbeacon,

Thanks for your reply. Just to be clear, is it expected behaviour that I am unable to change the rate limit configuration, either using the environment variable TB_GW_RATE_LIMITS or setting thingsboard.rateLimits in tb_gateway.json?

[imbeacon]

We will check point, related to TB_GW_RATE_LIMITS, it is not expected behaviour, that you cannot change them.

[yfPanne]

Is there any update on this issue? The gateway rate limit is a big problem for me！！

[yfPanne]

I have found two bugs about rate limit: thingsboard-python-client-sdk #59 #1408

[imbeacon]

This bug really relates to ThingsBoard Client Python SDK, it is solved in the master branch in that repository and will be available in the next release of the SDK.