[HELP] how to implement datapoint discovery protocol for a customized connector?

[jpiwek]

Describe the issue Hi,

I would like to create a new connector which is similar to Modbus RTU over serial but with a different workflow to set up data reading. I developed this driver in Python few year ago and would migrate to support TB Connector structure.

The protocol I plan to implement supports a command to read a data point description. I would like to implement kind of data point discovery function the user can invoke when configuring my connector. The workflow would be as follows:

1. Connector config - configure serial /dev/ttyUSB0 and device address ID (e.g. 0x06)
2. scan data points by pressing a button in the web gui - retrieves description (offset, multiplier, logical address, unit,..) of each data point
3. after read out, data points are supposed to be depicted as spreadsheet
4. user can select which data point has to be read out continously (by checking a checkbox)

Connector name (If you need help with some connector/converter): Customized Connector

Versions (please complete the following information):

OS: RaspiOS Bookworm Linux 6.6.62+rpt-rpi-2712
Thingsboard IoT Gateway version [3.7.5] with Docker

[imbeacon]

Hi @jpiwek,

Thank you for your interest in ThingsBoard IoT Gateway, you can find the example of the custom connector in our official documentation - https://thingsboard.io/docs/iot-gateway/custom/serial-connector/

Described functionality like control from UI or anything like this - you can implement as RPC requests from ThingsBoard.

[jpiwek]

Hi @imbeacon,

I looked through the example few days ago. Thanks for that again. What I would like to achieve is to add the workflow steps 1 - 4 when adding connector device.

At the moment I am not sure on how realize the feature set technically with TB (see exmaple picture below) and I would appreciate some technical TB details:

• how develop the customized view for scanning devices on the bus. user fills out required fields (port, baudrate,....=
• user presses "Scan" button -> RPC call is invoked to execute my Python discovery function
• how to reflect scanned devices and datapoints in the table structure?
• where to apply and to store the final customer selection for data reading/writings atrributes/time series/attribute update RPC request? For Modbus data points of interest each single datapoint has to be added manually. with my protocol I read out the complete data point description (Datentyp | Datenlaenge | Adresse | Datenrichtung | Bezeichnung | Einheit | Offset | Gain | Format | Min. Wert | Max. Wert | Wert/Zustand) up front and the user just needs to select which data point is supposed to be attribute/time series/rpc write request

[imbeacon]

@jpiwek,

It is a complex task that requires changes in UI and the gateway itself. I can answer about the gateway part - you can use RPC to connectors to route it to the required connector, to do this - you need to send RPC to the gateway device, with the following format of the method: <connector_type>_<method_name>' - RPC to the gateway with this mathod name format will be passed to all active connectors with this type (connectors server_side_rpc_handler` methods will be called, you can handle them inside as you want.

After scanning, in server_side_rpc_handler - you can create a response, probably array with dicts that contains different fields, like type, datatype, unit, etc.

After creating the response - you returns it from server_side_rpc_handler . The gateway will send it as a response for received RPC and you will have this information on TB.

Then you need to parse it on UI side, display to user, give him ability to choose required fields, etc.

Then, you can form configuration and save it as a configuration for the connector.

[jpiwek]

@imbeacon That sounds like a plan. I pressume that it would make sense to start with the Serial Connector Example and then create kind of mock-up first. I would use the Serial Connector Example to create a basic mock-up example for the use-case I have mind. Maybe you can give some more detailed hint what to look for.

Few questions on your feedback:

1. send scan RPC request to the gateway device: <connector_type>_<method_name> -> "serial-connector_scan" would then invoke my serial-connectors' Python scan function, right? -where to create this RPC request and the method so that it gets executed by gateway devices (is it the serverSideRpc object in serial-connector custom_serial.json LINK?)

"serverSideRpc": [
{
"method": "setValue", // -> need to be changed to scan 
"type": "int",
"withResponse": true,
"responseType": "string",
"responseTimeoutSec": 5
...
}]

2. I assume you mean the server_side_rpc_handler callback in the Serial connector class implementation?

3. After creating the response - you returns it from server_side_rpc_handler

If would create a Mock-Response for the scan function which returns an simple JSON object after sleeping 5 seconds....

I will cover these topics first.

[imbeacon]

@jpiwek Please find answers below:

1. send scan RPC request to the gateway device: <connector_type>_<method_name> -> "serial-connector_scan" would then invoke my serial-connectors' Python scan function, right? -where to create this RPC request and the method so that it gets executed by gateway devices (is it the serverSideRpc object in serial-connector custom_serial.json LINK?)

Not actually, RPC with method serial_scan will be passed to server_side_rpc_handler method of serial connector, then you can call anything you need.

2. I assume you mean the server_side_rpc_handler callback in the Serial connector class implementation?

Yes

3. After creating the response - you returns it from server_side_rpc_handler If would create a Mock-Response for the scan function which returns an simple JSON object after sleeping 5 seconds....

5 seconds delay may provoke timeout error on ThingsBoard, I suggest not to add any additional delay or increase a timeout in widget configuration.

[jpiwek]

@jpiwek Please find answers below:

1. send scan RPC request to the gateway device: <connector_type>_<method_name> -> "serial-connector_scan" would then invoke my serial-connectors' Python scan function, right? -where to create this RPC request and the method so that it gets executed by gateway devices (is it the serverSideRpc object in serial-connector custom_serial.json LINK?)

Not actually, RPC with method serial_scan will be passed to server_side_rpc_handler method of serial connector, then you can call anything you need.

@imbeacon : I there an example which I can re-use to develop my mock-up example?

[imbeacon]

Hi @jpiwek,

You can check ble_connector.py, for example method 'SCAN'.

[jpiwek]

@imbeacon : By the way, what is the most efficient way to develop, debug a customized connector with TB Gateway? Currently my TB Gateway is running with Docker.

[imbeacon]

We use IDE(VSCode or Pycharm) and run the gateway for sources for development, but you can run as you want. Anyway, unfortunately, at the moment - there is no way to add a custom type for connector on UI side, so it may be more convenient to develop the connector, without using remote configuration feature. Then, when we will add ability to add custom connector types on UI - you will be able to add your connector to remote configuration. Or you can change the sources of the gateways page and use it instead of the official one.

[jpiwek]

@imbeacon : I use VS Code as well. I bound my Docker Container "thingsboard_gateway" folder into my host folder strucuture.

Anyway, unfortunately, at the moment - there is no way to add a custom type for connector on UI side, so it may be more convenient to develop the connector, without using remote configuration feature Does it mean that my connector is not available when using "add connector" and that it 's not listed under "Connectors"? (see picture)

Which mean I have to add the corresponding custom connector config into tb_gateway.json file and then debug by my application using debug console?

[jpiwek]

Just tried the Customized Serial Connector example with TB Gateway 3.7.5 and got this error:

2025-06-26 14:27:22.176 - |INFO| - [tb_gateway_service.py] - tb_gateway_service - init - 154 - Gateway starting... 2025-06-26 14:27:22.951 - |INFO| - [tb_gateway_service.py] - tb_gateway_service - init - 168 - ThingsBoard IoT gateway version: 3.7.5 2025-06-26 14:27:22.958 - |INFO| - [tb_loader.py] - tb_loader - import_module - 77 - No module named 'serial' 2025-06-26 14:27:22.961 - |WARNING| - [tb_gateway_service.py] - tb_gateway_service - _load_connectors - 883 - Connector implementation not found for Serial Connector 2025-06-26 14:27:22.961 - |ERROR| - [tb_gateway_service.py] - tb_gateway_service - _load_connectors - 886 - The following error occurred during importing connector class: No module named 'serial' NoneType: None

My config/tb_gateway.json

"connectors": [
    {
      "type": "serial",
      "name": "Serial Connector",
      "configuration": "custom_serial.json",
      "class": "SerialConnector"
    }
  ]
....

My config/custom_serial.json

{
  "name": "Custom serial connector",
  "logLevel": "DEBUG",
  "uplinkQueueSize": 100000,
  "devices": [
    {
      "name": "SerialDevice1",
      "type": "default",
      "port": "/dev/ttysUSB0",
      "baudrate": 9600,
      "converter": "SerialUplinkConverter",
      "downlink_converter": "SerialDownlinkConverter",
....

My extension/serial folder

root@9359736749b7:/# ls -la /thingsboard_gateway/extensions/serial
total 52
drwxr-xr-x  3 1000 1000  4096 Jun 26 14:26 .
drwxrwxr-x 21 1000 1000  4096 Jun 26 14:25 ..
-rw-rw-r--  1 1000 1000   616 Jun 26 14:25 __init__.py
drwxr-xr-x  2 1000 1000  4096 Jun 26 14:27 __pycache__
-rw-rw-r--  1 1000 1000 21074 Jun 26 14:25 serial_connector.py
-rw-rw-r--  1 1000 1000  2237 Jun 26 14:25 serial_downlink_converter.py
-rw-rw-r--  1 1000 1000  7732 Jun 26 14:25 serial_uplink_converter.py 

My Serial ttyUSB0 mapped into the Docker Container:

root@9359736749b7:/# ls -la /dev/ttyUSB0 
crw-rw---- 1 root dialout 188, 0 Jun 26 14:27 /dev/ttyUSB0

[imbeacon]

Hi @jpiwek,

Which mean I have to add the corresponding custom connector config into tb_gateway.json file and then debug by my application using debug console? Unfortunately, yes.

According to the error with serial connector - it looks like you don't have installed pyserial module in your environment. just run pip install pyserial or add to the very beginning of the connector the following code:

from thingsboard_gateway.tb_utility.tb_utility import TBUtility

try:
    import pyserial
except (ImportError, ModuleNotFoundError):
    print("PySerial library not found")
    TBUtility.install_package("pyserial")
    import pyserial

[jpiwek]

It seems to work now. What I did:

1. added a CUSTOM Connector (see PIC2)
2. changed the connector config in thingsboard_gateway.json

"connectors": [
    {
      "type": "serial",
      "name": "My Serial Connector",
      "configuration": "mySerialConnector.json",
      "class": "SerialConnector"
    }
  ]

4. copied the custom_serial.json config into the CONNECTOR Config textbox (see PIC2)
5. saved config (see PIC2)
6. send data (need to check the right data protocol format of the serial connector example) from ttyUSB1 to TB-Gateway SerialDevice1 on ttyUSB0 (see PIC1) -----------------------------------------Pic1------------------------------------------------------------

------------------------------------------Pic2-----------------------------------------------------------

[imbeacon]

Hi @jpiwek,

I closes this issue, because it looks finalised for me. Feel free to open the new one if you discover some issue, or you also may open a discussion, if you want to discuss some points, etc. - https://github.com/thingsboard/thingsboard-gateway/discussions

[jpiwek]

@imbeacon: thanks for your guideance! I will be porting my project to run as a connector and as I go along, questions will pop up.