Running into GatewayResultCode.GW_RES_SINK_OUT_OF_MEMORY when sending a lot of requests

[vespakoen]

Describe the bug

I am trying to "spam" MQTT with a lot of messages to send into the network, to see what the limits are. I am running mosquitto, wp-sink-service, wp-transport-service and wp-dbus-service on a Raspberry Pi 400, with a NINA B3 USB Dongle.

I am using the latest wirepas_mqtt_library for python with something like:

        res = client.send_message(
            gw_id=parameters.gw_id,
            sink_id=parameters.sink_id,
            dest=destination,
            src_ep=parameters.src_ep,
            dst_ep=parameters.dst_ep,
            payload=bytes.fromhex("8A")
        )
        if res != GatewayResultCode.GW_RES_OK:
            print("Sending data synchronously failed: res=%s" % res)
        else:
            print("Sending data synchronously succeeded")

I have a script that spams a FastAPI endpoint which calls the code above:

#!/usr/bin/env python

import requests
from concurrent import futures
from itertools import repeat

wpparameters_list = repeat({
   "gw_id": "my_wp_gateway",
   "sink_id": "sink0",
   "dest": 1,
   "src_ep": 1,
   "dst_ep": 1
}, 500)

def make_request(wpparameters):
  r = requests.post('http://localhost:3000/execute', timeout=4000, verify=False, json=wpparameters)
  print(r.status_code)

with futures.ThreadPoolExecutor(max_workers=10) as executor:
    res = executor.map(make_request, wpparameters_list)

The res output from the backend is:

Sending data synchronously failed: res=GatewayResultCode.GW_RES_SINK_OUT_OF_MEMORY

On the Raspberry Pi logs I find a bunch of:

wp-sink-service-1       | 2023-06-23 17:30:25,100 | [ERROR] wpc:Cannot send data. Dualmcu error code: 4
wp-sink-service-1       | 2023-06-23 17:30:25,100 | [ERROR] Data:Cannot send data: 21
wp-transport-service-1  | 2023-06-23 17:30:25,102 | [ERROR] wirepas_gateway@sink_manager.py:86:Cannot send message err=21

I have been looking through the c-mesh-api code but cannot find from where the out of memory error is being returned. And I am also wondering if I should be putting a queue in between my backend and MQTT so that messages are not coming in too fast for the sink to handle them? I am basically looking for a way to make the sink send out as many messages as possible ;)

To Reproduce Steps to reproduce the behavior:

1. Start gateway services
2. Send a whole lot of messages
3. See error

Expected behavior No errors? ;)

Platform details:

• OS: *e.g. Docker on Raspberry Pi 400, Docker macOS 13.0 *
• Gateway host: rpi
• Python version: 3.9?
• Docker engine: 23.0.1
• MQTT Broker vendor: mosquitto
• MQTT Broker version: 2.0.15