thingsboard-gateway icon indicating copy to clipboard operation
thingsboard-gateway copied to clipboard
verified publisher icon thingsboard

[HELP] Thingsboard Gateway MQTT connector is not reconnecting until the Gateway container is restarted

Open akseerali opened this issue 2 years ago • 3 comments

Describe the issue I am sending the data from MQTT broker to TB Edge via TB Gateway. At gateway, I am using MQTT 8883 TLS based connection to join the broker. The gateway is working fine and I am also able to receive the data at Edge Devices; however, when I am restarting the remote MQTT broker, the gateway is throwing an error and it can no longer subscribe as a MQTT client until the container is restarted. I have also tried to use Thingsboard Gateway as a DEB package on Ubuntu, but it has the same issue.

Below are the attached configurations for Thingsboard Gateway, MQTT connector, and Docker Compose. tb_gateway.yaml configuration:

thingsboard:
  host: 10.5.0.11
  port: 1883
  remoteShell: false
  remoteConfiguration: false
  statistics:
    enable: true
    statsSendPeriodInSeconds: 3600
  maxPayloadSizeBytes: 1024
  minPackSendDelayMS: 100
  checkConnectorsConfigurationInSeconds: 60
  handleDeviceRenaming: false
  checkingDeviceActivity:
    checkDeviceInactivity: true
    inactivityTimeoutSeconds: 120
    inactivityCheckPeriodSeconds: 10
  security:
    accessToken: #access token
  qos: 1
storage:
#  type: memory
#  read_records_count: 100
#  max_records_count: 100000
#  type: file  
#  data_folder_path: /home/iiotedge/.tb-gateway/data
#  max_file_count: 10
#  max_read_records_count: 10
#  max_records_per_file: 1000
  type: sqlite
  data_file_path: /home/iiotedge/.tb-gateway/data/data.db
  messages_ttl_check_in_hours: 1
  messages_ttl_in_days: 7
grpc:
  enabled: false
  serverPort: 9595
  keepaliveTimeMs: 10000
  keepaliveTimeoutMs: 5000
  keepalivePermitWithoutCalls: true
  maxPingsWithoutData: 0
  minTimeBetweenPingsMs: 10000
  minPingIntervalWithoutDataMs: 5000
connectors:
   - name: Paint Booth connector
     type: mqtt
     configuration: mqtt-connector.json

**mqtt connector configuration: **

{
   "broker":{
      "name":"MQTT broker",
      "host":"192.168.1.100",
      "port":8883,
      "clientId":"mqttconnector",
      "maxMessageNumberPerWorker":10,
      "maxNumberOfWorkers":100,
      "security":{
         "type":"tls",
         "username":"mqttconnector",
         "password":"**mqttconnector**"
      }
   },
   "mapping":[
      {
         "topicFilter":"machine1/point2/#",
         "converter":{
            "type":"json",
            "deviceNameJsonExpression":"Device2",
            "deviceTypeJsonExpression":"Demo_Profile",
            "ts":"${ts}",
            "timeout":60000,
            "attributes":[],
            "timeseries":[
               {
                  "type":"double",
                  "key":"temperature",
                  "value":"${temperature}"
               },
               {
                  "type":"double",
                  "key":"humidity",
                  "value":"${humidity}"
               }
            ]
         }
      }
   ],
   "connectRequests":[],
   "disconnectRequests":[],
   "attributeRequests":[],
   "attributeUpdates":[],
   "serverSideRpc":[]
}

**Docker compose configuration: **

version: '3'
services:
  mytbedge:
    restart: always
    image: "thingsboard/tb-edge-pe:3.4.3EDGEPE"
    container_name: tb-edge
    ports:
      - "8080:8080"
      - "1883:1883"
      - "5683-5688:5683-5688/udp"
    environment:
      SPRING_DATASOURCE_URL: jdbc:postgresql://postgres:5432/tb-edge
      EDGE_LICENSE_INSTANCE_DATA_FILE: /data/instance-edge-license.data
      CLOUD_ROUTING_KEY: INSERT EDGE ROUTING KEY # e.g. 19ea7ee8-5e6d-e642-4f32-05440a529015
      CLOUD_ROUTING_SECRET:  INSERT EDGE SECRET KEY # e.g. bztvkvfqsye7omv9uxlp
      CLOUD_RPC_HOST: thingsboard.cloud
      CLOUD_RPC_SSL_ENABLED: 'true' # set it to 'true' if you are connecting edge to thingsboard.cloud
    volumes:
      - ~/.mytb-edge-data:/data
      - ~/.mytb-edge-logs:/var/log/tb-edge
    networks:
      mynetwork:
        ipv4_address: 10.5.0.11
  postgres:
    restart: always
    image: "postgres:12"
    container_name: postgres-db
    ports:
      - "5432"
    environment:
      POSTGRES_DB: tb-edge
      POSTGRES_PASSWORD: postgres    
    volumes:
      - ~/.mytb-edge-data/db:/var/lib/postgresql/data
    networks:
      mynetwork:
        ipv4_address: 10.5.0.12
  pgadmin:
    image: dpage/pgadmin4
    container_name: pgadmin4
    restart: "no"
    ports:
      - "5050:80"
    environment:
      PGADMIN_DEFAULT_EMAIL: pgadmin4
      PGADMIN_DEFAULT_PASSWORD: 115599
    volumes:
      - ~/.mytb-edge-data:/db
    networks:
      mynetwork:
        ipv4_address: 10.5.0.13
  mytbgw:
    image: "thingsboard/tb-gateway"
    container_name: tb-gw
    restart: always
    ports:
      - "5000:5000"
    volumes:
      - ~/.tb-gateway/logs:/thingsboard_gateway/logs
      - ~/.tb-gateway/extensions:/thingsboard_gateway/extensions
      - ~/.tb-gateway/config:/thingsboard_gateway/config
      - ~/.tb-gateway/data:/home/iiotedge/.tb-gateway/data
    networks:
      mynetwork:
        ipv4_address: 10.5.0.14
networks:
  mynetwork:
    driver: bridge
    ipam:
      config:
        - subnet: 10.5.0.0/16
          gateway: 10.5.0.1

Error logs:

tb-gw  | ""2023-03-13 15:26:00" - |INFO| - [sqlite_event_storage.py] - sqlite_event_storage - put - 76 - Sending data to storage"
tb-gw  | ""2023-03-13 15:26:00" - |INFO| - [sqlite_event_storage.py] - sqlite_event_storage - put - 76 - Sending data to storage"
tb-gw  | ""2023-03-13 15:26:00" - |INFO| - [sqlite_event_storage.py] - sqlite_event_storage - put - 76 - Sending data to storage"
tb-gw  | ""2023-03-13 15:26:15" - |INFO| - [sqlite_event_storage.py] - sqlite_event_storage - put - 76 - Sending data to storage"
tb-gw  | ""2023-03-13 15:26:15" - |INFO| - [sqlite_event_storage.py] - sqlite_event_storage - put - 76 - Sending data to storage"
tb-gw  | ""2023-03-13 15:26:15" - |INFO| - [sqlite_event_storage.py] - sqlite_event_storage - put - 76 - Sending data to storage"
tb-gw  | ""2023-03-13 15:26:30" - |INFO| - [sqlite_event_storage.py] - sqlite_event_storage - put - 76 - Sending data to storage"
tb-gw  | ""2023-03-13 15:26:30" - |INFO| - [sqlite_event_storage.py] - sqlite_event_storage - put - 76 - Sending data to storage"
tb-gw  | ""2023-03-13 15:26:30" - |INFO| - [sqlite_event_storage.py] - sqlite_event_storage - put - 76 - Sending data to storage"
tb-gw  | ""2023-03-13 15:26:45" - |INFO| - [sqlite_event_storage.py] - sqlite_event_storage - put - 76 - Sending data to storage"
tb-gw  | ""2023-03-13 15:26:45" - |INFO| - [sqlite_event_storage.py] - sqlite_event_storage - put - 76 - Sending data to storage"
tb-gw  | ""2023-03-13 15:26:45" - |INFO| - [sqlite_event_storage.py] - sqlite_event_storage - put - 76 - Sending data to storage"
tb-gw  | Exception in thread Paint Booth connector:
tb-gw  | Traceback (most recent call last):
tb-gw  |   File "/usr/local/lib/python3.10/threading.py", line 1016, in _bootstrap_inner
tb-gw  |     self.run()
tb-gw  |   File "/thingsboard_gateway/connectors/mqtt/mqtt_connector.py", line 272, in run
tb-gw  |     self.__connect()
tb-gw  |   File "/thingsboard_gateway/connectors/mqtt/mqtt_connector.py", line 279, in __connect
tb-gw  |     self._client.connect(self.__broker['host'],
tb-gw  |   File "/root/.local/lib/python3.10/site-packages/paho/mqtt/client.py", line 914, in connect
tb-gw  |     return self.reconnect()
tb-gw  |   File "/root/.local/lib/python3.10/site-packages/paho/mqtt/client.py", line 1073, in reconnect
tb-gw  |     sock.do_handshake()
tb-gw  |   File "/usr/local/lib/python3.10/ssl.py", line 1342, in do_handshake
tb-gw  |     self._sslobj.do_handshake()
tb-gw  | ssl.SSLEOFError: EOF occurred in violation of protocol (_ssl.c:997)
tb-gw  | Exception in thread Thread-7 (_thread_main):
tb-gw  | Traceback (most recent call last):
tb-gw  |   File "/usr/local/lib/python3.10/threading.py", line 1016, in _bootstrap_inner
tb-gw  |     self.run()
tb-gw  |   File "/usr/local/lib/python3.10/threading.py", line 953, in run
tb-gw  |     self._target(*self._args, **self._kwargs)
tb-gw  |   File "/root/.local/lib/python3.10/site-packages/paho/mqtt/client.py", line 3591, in _thread_main
tb-gw  |     self.loop_forever(retry_first_connection=True)
tb-gw  |   File "/root/.local/lib/python3.10/site-packages/paho/mqtt/client.py", line 1756, in loop_forever
tb-gw  |     rc = self._loop(timeout)
tb-gw  |   File "/root/.local/lib/python3.10/site-packages/paho/mqtt/client.py", line 1164, in _loop
tb-gw  |     rc = self.loop_read()
tb-gw  |   File "/root/.local/lib/python3.10/site-packages/paho/mqtt/client.py", line 1556, in loop_read
tb-gw  |     rc = self._packet_read()
tb-gw  |   File "/root/.local/lib/python3.10/site-packages/paho/mqtt/client.py", line 2439, in _packet_read
tb-gw  |     rc = self._packet_handle()
tb-gw  |   File "/root/.local/lib/python3.10/site-packages/paho/mqtt/client.py", line 3039, in _packet_handle
tb-gw  |     return self._handle_connack()
tb-gw  |   File "/root/.local/lib/python3.10/site-packages/paho/mqtt/client.py", line 3135, in _handle_connack
tb-gw  |     on_connect(self, self._userdata,
tb-gw  |   File "/thingsboard_gateway/connectors/mqtt/mqtt_connector.py", line 389, in _on_connect
tb-gw  |     if result_code in result_codes:
tb-gw  | TypeError: unhashable type: 'ReasonCodes'

Versions (please complete the following information):

  • OS: [Ubuntu 22.04]
  • Thingsboard IoT Gateway version [latest]
  • Thingsboard Edge PE version [3.4.3EDGEPE]
  • Docker Engine version [23.0.1]
  • Docker Compose version [v2.16.0]

akseerali avatar Mar 13 '23 15:03 akseerali

Hi @akseerali, thanks for your interest in ThingsBoard IoT Gateway! We are investigating your problem.

samson0v avatar Mar 21 '23 19:03 samson0v

Hello @samson0v we are experiencing the same error. We are using a thingsboard gateway with a MQTT connection to a remote broken, and when the broker disconnect we see same logs as previous post:

if result_code in result_codes: tb-gw | TypeError: unhashable type: 'ReasonCodes'

The solution is to restart the IoT Gateway docker container to restablish the connection.

Any news ?

mdameda avatar Sep 04 '23 13:09 mdameda

I have the same problem for the OPCUA connector and we use the same solution (which is inefficient) . I'm using commit b684e46cc23469865ed2b45c984cbd67455d24b9 (12 July 2024) Do you have any news about this?

cos-gl avatar Jul 29 '24 09:07 cos-gl