Stream server component (serial-over-wifi)

[oxan]

Describe the problem you have/What new integration you would like A server that allows bidirectional communication with a serial port over WiFi. Now, I know this has been requested and rejected before (#213, #415, #619, #638), but I disagree with the given reasoning (and I have a patch :)):

• Re #638: Plenty of HA integrations communicate over a serial port, and can be used with such a server to control remote devices. E.g. acer_projector, alarmdecoder, arduino, aurora_abb_powerone, dsmr, edl21, elv, enocean, etc.
• Re #415: I've taken a quick look at some HA integrations using serial ports, and pretty much all of them just send some messages back and forth over the serial port. Nothing more fancy such as control sequences or hardware flow control. So I think 95% of the use cases can be captured with a quite simple implementation, and if it's made extensible the rest can use a custom component.
• Re #213: I agree that its frontend and backend should be separated - but luckily ESPHome already has a Stream abstraction implemented by the UART interface. Thus, the backend is already there, and we only need a frontend to expose a Stream over TCP.
• One other major benefit of integrating a serial-over-wifi bridge is that it'd allow a single ESP to be used as a serial bridge and for sensors. For example, I have an ESP with OpenTherm GW next to my central heating installation, but I also want to hookup some sensors to monitor power consumption.

Please describe your use case for this integration and alternatives you've tried: See above. I want to hookup both a serial port and some other sensors (or switches, etc) to a single ESP integrated in Home Assistant.

Alternatives are various different ESP firmwares (ESPEasy, ESPLink); but neither of those allows me to easily integrate a sensor in HA.

External component For now I've implemented this as a external component, which is available here. It creates a TCP server listening on port 6638, and relays all data between the connected clients and the serial port. It doesn't support any control sequences, telnet options or RFC 2217, just raw data (I don't think the fancy stuff is actually useful, very few devices require it and they aren't actually wired on the common ESP modules).

However, I think it'd be nice if this was available in ESPHome by default. I can create a PR to rework my custom component to integrate it in ESPHome itself (including configuration etc.) if you agree to accept this functionality.

[glmnet]

I like this, if it's useful for you then it should be useful for other people.

The thing being so generic leads to lot of issues, but this might be considered for advanced users, or something like that, that's why Otto recommended this being a custom_component / cookbook thing, so anybody could modify it quickly.

For example your implementation allows for several clients connecting simultaneously to the bridge, this might not be what other people needs, then there will come the "option" to allow one connection and stick to it even if other connection arrives or another "option" to discard the active connection and the last request will win.

Initially I'd like it to be implemented as a cookbook thing, it is really useful because it explains how stuff works and allows you to start from tested code, see e.g. the Arduino Port Expander

Then there is the new custom_components folder method to implement ESPHome components, that's equally good for Cookbooks, and it allows for quickly addition to ESPHome codebase as it will be a matter of copying the files to the repository.

There were a few messages exchanged with Otto, Brandon and other users, Otto is trying to make ESPHome more like HA where many contributors are easily allowed to add their own stuff and "take responsibility" of it, but nothing come out just yet.

Anyway thanks for sharing the code!

[Hedda]

This request might be related to #688 = Request for "ESPHome support on Sonoff ZBBridge as a remote Zigbee adapter for Home Assistant's ZHA component via its zigpy and bellows libraries".

As I understand that ##688 would depend on some kind of stream server for Serial or UART over Wi-Fi.

Though while related I would not say that #660 replaced #688 and their ultimate goals are different.

[zuidwijk]

I would LOVE (yes capitals) this feature to be added to ESPHome. I'm building many different automation hardware for Home Assistant based on ESPHome, and this is something I really mis. Adding a second ESP only to use esp-link next to ESPHome is such a waste...

[schnizki]

I can confirm this custom component works with the new Sonoff ZBBridge and the ZHA intergation. This is my config in ESPHome:

esphome:
  name: zigbee
  platform: ESP8266
  board: esp01_1m
  
  includes:
    - stream_server.h
    - stream_server.cpp

wifi:
  ssid: "HASSos"
  password: "***"

  ap:
    ssid: "Zigbee Fallback Hotspot"
    password: "***"
captive_portal:

logger:
 baud_rate: 0


api:
  password: "***"

ota:
  password: "***"

uart:
  id: uart_bus
  tx_pin: GPIO1
  rx_pin: GPIO3
  baud_rate: 115200
  
custom_component:
  - lambda: |-
      auto stream_server = new StreamServerComponent(id(uart_bus));
      return {stream_server}; 

ZHA dosent resolve .local adresses so just use the IP.

It would be awesome if this could be added to ESPHome.

[zuidwijk]

I can confirm this custom component works with the new Sonoff ZBBridge and the ZHA intergation.

This is my config in ESPHome:

esphome:

  name: zigbee

  platform: ESP8266

  board: esp01_1m

  

  includes:

    - stream_server.h

    - stream_server.cpp



wifi:

  ssid: "HASSos"

  password: "***"



  ap:

    ssid: "Zigbee Fallback Hotspot"

    password: "***"

captive_portal:



logger:

 baud_rate: 0





api:

  password: "***"



ota:

  password: "***"



uart:

  id: uart_bus

  tx_pin: GPIO1

  rx_pin: GPIO3

  baud_rate: 115200

  

custom_component:

  - lambda: |-

      auto stream_server = new StreamServerComponent(id(uart_bus));

      return {stream_server}; 

ZHA dosent resolve .local adresses so just use the IP.

It would be awesome if this could be added to ESPHome.

Nice! Is that the same as this one?

https://gist.github.com/phha/651a2eeb30c6d6ca0937b3394e2bd9e6

[oxan]

Nice! Is that the same as this one?

https://gist.github.com/phha/651a2eeb30c6d6ca0937b3394e2bd9e6

Yes, that's exactly the same code, someone forked it but made no changes for some reason.

[zuidwijk]

Nice! Is that the same as this one? https://gist.github.com/phha/651a2eeb30c6d6ca0937b3394e2bd9e6

Yes, that's exactly the same code, someone forked it but made no changes for some reason.

Ah ok! I didn't saw the link in your first message, so I Googled it and found this one... found yours now :)

[zuidwijk]

Confirmed :)

esphome:
  name: p1test
  platform: ESP8266
  board: d1_mini

  includes:
    - stream_server.h
    - stream_server.cpp
    
wifi:
  ssid: "***"
  password: "***"

  # Enable fallback hotspot (captive portal) in case wifi connection fails
  ap:
    ssid: "P1Test Fallback Hotspot"
    password: "***"

captive_portal:

# Enable logging
logger:

# Enable Home Assistant API
api:

ota:

uart:
  id: uart_bus
  tx_pin: GPIO1
  rx_pin: GPIO3
  baud_rate: 115200
  
custom_component:
  - lambda: |-
      auto stream_server = new StreamServerComponent(id(uart_bus));
      return {stream_server};

Works like a charm!

[Harald-P]

I like this too, but i have beginner problems with compiling it: In file included from src/stream_server.cpp:17:0: src/stream_server.h:25:25: fatal error: ESPAsyncTCP.h: No such file or directory

So i searched ESPAsyncTCP.h but i found on github only a version for ESP 8266 (https://github.com/me-no-dev/ESPAsyncTCP or i must use an other one?) and for ESP32 i found only AsyncTCP.h (https://github.com/me-no-dev/AsyncTCP/tree/master/src).

Can you give me considerations how can i let compile this custom_component for ESP32? Thanks in advance...

[oxan]

@Harald-P Try adding this:

esphome:
   libraries:
   - [email protected]

(note that this version is for ESPHome 1.14.5, newer/older versions might need a different version).

Usually this library is already included by the api or ota component.

[Harald-P]

hmmm, the version is 1.14.5 and i use api and ota ...

i tryed adding your consideration, but i recive now:

Compiling /data/cellar/.pioenvs/cellar/src/stream_server.cpp.o
In file included from src/main.cpp:27:0:
src/stream_server.h:25:25: fatal error: ESPAsyncTCP.h: No such file or directory

*********************************************************************
* Looking for ESPAsyncTCP.h dependency? Check our library registry!
*
* CLI  > platformio lib search "header:ESPAsyncTCP.h"
* Web  > https://platformio.org/lib/search?query=header:ESPAsyncTCP.h
*
*********************************************************************

compilation terminated.
*** [/data/cellar/.pioenvs/cellar/src/main.cpp.o] Error 1
In file included from src/stream_server.cpp:17:0:
src/stream_server.h:25:25: fatal error: ESPAsyncTCP.h: No such file or directory

*********************************************************************
* Looking for ESPAsyncTCP.h dependency? Check our library registry!
*
* CLI  > platformio lib search "header:ESPAsyncTCP.h"
* Web  > https://platformio.org/lib/search?query=header:ESPAsyncTCP.h
*
*********************************************************************

compilation terminated.
*** [/data/cellar/.pioenvs/cellar/src/stream_server.cpp.o] Error 1

i use hass.io and compile it over the esphome-dashboard in home assistant on a Raspberry pi...

[oxan]

@Harald-P Oh, I didn't see that you were on ESP-32. I haven't tried it with an ESP-32 personally. Can you try replacing ESPAsyncTCP.h with AsyncTCP.h in both files?

[Harald-P]

@oxan i have done the change in stream_server.h and i have deletet this 2 rows:

   libraries:
   - [email protected]

The compilation concludet than without error, but now my ESP32 its not more working... It is pingable but not available: WARNING Initial connection failed. The ESP might not be connected to WiFi yet (Error connecting to 192.168.0.125: [Errno 111] Connection refused). Re-Trying in 11 seconds INFO Connecting to cellar.local:6053 (192.168.0.125)

So i look tomorrow if i can install the old immage to bring the ESP32 back online...

[mjkl-gh]

@oxan i have done the change in stream_server.h and i have deletet this 2 rows:

   libraries:
   - [email protected]

The compilation concludet than without error, but now my ESP32 its not more working... It is pingable but not available: WARNING Initial connection failed. The ESP might not be connected to WiFi yet (Error connecting to 192.168.0.125: [Errno 111] Connection refused). Re-Trying in 11 seconds INFO Connecting to cellar.local:6053 (192.168.0.125)

So i look tomorrow if i can install the old immage to bring the ESP32 back online...

I'm also on esp32 (m5stickc) did you get any further with this?

edit:

tried it myself still crashes on esp32. Hook it up to serial to see 9 boot attempts:

[22:23:28][I][logger:166]: Log initialized
[22:23:28][C][ota:366]: There have been 7 suspected unsuccessful boot attempts.
[22:23:28][I][app:029]: Running through setup()...
[22:23:28][C][spi:022]: Setting up SPI bus...
[22:23:28][C][uart_esp32:072]: Setting up UART...
[22:23:28][C][streamserver:011]: Setting up stream server...
[22:23:29]/home/runner/work/esp32-arduino-lib-builder/esp32-arduino-lib-builder/esp-idf/components/freertos/queue.c:1442 (xQueueGenericReceive)- assert failed!
[22:23:29]abort() was called at PC 0x40089505 on core 1
[22:23:29]
[22:23:29]Backtrace: 0x4008d104:0x3ffb0df0 0x4008d335:0x3ffb0e10 0x40089505:0x3ffb0e30 0x4015451a:0x3ffb0e70 0x401547f6:0x3ffb0e90 0x40142e9c:0x3ffb0eb0 0x40142f05:0x3ffb0ed0 0x40144534:0x3ffb0ef0 0x40144620:0x3ffb0f20 0x400e764b:0x3ffb0f40 0x400e6d47:0x3ffb0f90 0x4016ccad:0x3ffb1000 0x4016cd69:0x3ffb1020 0x400dfad5:0x3ffb1040 0x400e6521:0x3ffb1090 0x400f4b8b:0x3ffb1fb0 0x40089819:0x3ffb1fd0
[22:23:29]
[22:23:29]Rebooting...
[22:23:29]ets Jun  8 2016 00:22:57
[22:23:29]
[22:23:29]rst:0xc (SW_CPU_RESET),boot:0x13 (SPI_FAST_FLASH_BOOT)
[22:23:29]configsip: 188777542, SPIWP:0xee
[22:23:29]clk_drv:0x00,q_drv:0x00,d_drv:0x00,cs0_drv:0x00,hd_drv:0x00,wp_drv:0x00
[22:23:29]mode:DIO, clock div:2
[22:23:29]load:0x3fff0018,len:4
[22:23:29]load:0x3fff001c,len:952
[22:23:29]load:0x40078000,len:6084
[22:23:29]load:0x40080000,len:7944
[22:23:29]entry 0x40080310

after which it proceeds to safe mode

[oxan]

@MaartenKleijwegt Can you try inserting this->server_.setNoDelay(false); before line 31 in stream_server.cpp?

EDIT: Or actually, try this first: add before line 36 of stream_server.h:

float get_setup_priority() const override { return esphome::setup_priority::AFTER_WIFI; }

[mjkl-gh]

@MaartenKleijwegt Can you try inserting this->server_.setNoDelay(false); before line 31 in stream_server.cpp?

EDIT: Or actually, try this first: add before line 36 of stream_server.h:

float get_setup_priority() const override { return esphome::setup_priority::AFTER_WIFI; }

Thanks for the reply will try. was already looking into how esphome calls the app.setup loop. Figured it was setting up immediately after uart before wifi was setup. this #https://github.com/espressif/arduino-esp32/issues/3467 hinted I was trying to setup some wifi component before wifi was initialized

Could you point to where I can read up on esphome setup priority and such? haven't been able to find much in the last hour

EDIT; tested and doesn't crash anymore. Able to connect through web server telnet (putty) and ESPHome the fix was:

float get_setup_priority() const override { return esphome::setup_priority::AFTER_WIFI; }

didnt't try the other

[oxan]

Ok, great! I've updated the gist to incorporate this fix.

Could you point to where I can read up on esphome setup priority and such?

The code, I guess.

[mjkl-gh]

Would it be apreciated/an idea for me to open a pull request to update https://esphome.io/custom/custom_component.html with this example?

that's where I would expect it and actually searched for it

[oxan]

@MaartenKleijwegt the get_setup_priority() you mean? yeah, that'd make sense I think.

[duhow]

Works like a charm with ESP32 and CC2530 (zigbee2mqtt) ! I hope to see this officially integrated with ESPHome, as I'm using it for BLE detection as well :)

[yzg011]

thanks for your project，but when I use this for xiaomi zigbee button sometimes not working

[RoganDawes]

This is great, thank you! I can see that this will work perfectly for my Texecom Premier alarm, as a replacement for their COM-IP or COM-WiFi devices. All I will need to add is a level converter.

It would be great if it was possible to customise the listening port in the configuration, though. Or should I just update the source?

Also, how would I go about limiting this to a single concurrent connection? In theory, more than one concurrent connection should not happen, but in practice, it would probably cause havoc if it did, so it's probably worth while preventing it!

Turns out that is simple enough, just add the following lines into the onClient method:

        if(this->clients_.size() > 0) {
            ESP_LOGW(TAG, "Got a second connection, dropping it!");
            tcpClient->close();
            return;
        }

So, my ultimate objective is to create an "intelligent" device, not purely a simple serial passthrough to allow for the Texecom app to do its own configuration/analytics/etc. Ideally, this device will implement enough of the Texecom SIMPLE protocol to allow it to interact with the panel, activate/deactivate zones, activate/deactivate the alarm itself, and report zone activations/violations both when the alarm is not set, and when it is. All of this would be available via the API, or via MQTT (I still need to figure out how the API works!)

The plan is to have the component interacting with the UART until a new TCP connection comes in, at which point the component will cleanly disconnect from the panel, and allow the external program exclusive access. Once the TCP connection disconnects, the component will resume interaction with the panel, reporting back to HA or any other MQTT subscriber.

Does this sound reasonable? I guess I should move this to a new issue, as it is far beyond a simple TCP-UART server now!

[ioanfesteu]

I can confirm too that is working like charm with ESP8266+CC2530 and remote zigbee2mqtt. Thank you!

[Hedda]

Tip is also to check out Tube's Zigbee Gateways open-source hardware by @tube0013 which runs ESPHome firmware by default.

Based on WT32-ETH01 (Wireless-Tag) ESP32 board which has wired Ethernet that makes for a much more stable serial connection.

He designed two variants; one has a Silicon Labs EFR32 Series 2 module and one that has a Texas Instruments CC2652P module.

https://github.com/tube0013/tube_gateways

https://github.com/zigpy/zigpy/discussions/584

https://www.tubeszb.com/shop/coordinators/2

[oxan]

I've made my custom component available as a new external component for ESPHome v1.18+ under my GitHub account:

external_components:
  - source: github://oxan/esphome-stream-server

stream_server:

[Jpsy]

Hi all,
I am experimenting with @oxan's custom component and hitting a wall.

Setup is: CC2530 <-> ESP8266 + ESPHome + oxan component <-> WLAN <-> Zigbee2MQTT on HA 2021.11 on RPi

I have flashed the C2530 with Koenkk´s Z-Stack_Home_1.2/bin/default/CC2530_DEFAULT_20201127.zip.

When Zigbee2MQTT's Herdsman tries to connect, I can see that in the ESPHome log, but it finally fails after 20 seconds:

[11:42:19][D][streamserver:106]: New client connected from 192.168.178.180
[11:42:39][D][streamserver:055]: Client 192.168.178.180 disconnected

The corresponding Z2M log looks like this:

Zigbee2MQTT:info  2021-11-21 11:42:18: Starting zigbee-herdsman (0.13.164)
Zigbee2MQTT:error 2021-11-21 11:42:38: Error while starting zigbee-herdsman
Zigbee2MQTT:error 2021-11-21 11:42:38: Failed to start zigbee
Zigbee2MQTT:error 2021-11-21 11:42:38: Check https://www.zigbee2mqtt.io/information/FAQ.html#help-zigbee2mqtt-fails-to-start for possible solutions
Zigbee2MQTT:error 2021-11-21 11:42:38: Exiting...
Zigbee2MQTT:error 2021-11-21 11:42:39: Error: Failed to connect to the adapter (Error: SRSP - SYS - ping after 6000ms)

May I ask some questions here to help me debug this:

• I have verified that I am using the correct IP address and port. Is this the correct notation for the config of Z2M to connect to the stream component?:

serial:
  port: tcp://192.168.178.35:6638

• What should the stream component report in the ESPHome logs when there is communication between Z2M and ESPHome? (my logger level is VERBOSE).

• Is there some basic test to check whether a communication from HA to the CC2530 is possible? I.e. some shell commands that I could issue from HA?

• I get inconsistent information about the wiring between ESP8266 and CC2530. I tried two versions:

  • 3v3 -- VCC GND -- GND TX -- P02 RX -- P03
  • the above plus GND -- P20 GND -- P04 GND -- P05
    

  Are the 3 additional Ground connections in the second block needed?

• Any other ideas where I could start to dig in?

Many thanks! :D

[Hedda]

Z2M configuration side at least looks correct so guess is that the issue probably is in wiring between ESP8266 and CC2530:

https://www.zigbee2mqtt.io/guide/adapters/flashing/connecting_cc2530.html

https://www.zigbee2mqtt.io/advanced/zigbee/05_create_a_cc2530_router.html#result

https://www.zigbee2mqtt.io/advanced/remote-adapter/connect_to_a_remote_adapter.html#_2-b-configure-ser2net-4-0

I primarily use ZHA in HA and CC2652P myself (and not Zigbee2MQTT) so can not really help but can suggest posting here:

https://github.com/Koenkk/zigbee2mqtt/discussions

PS: Regardless, strongly recommend that buy CC2652P based module/adapter since CC2530 and CC2531 are really obsolete now (both very old hardware with limited resources and old "Z-Stack Home 1.2" firmware that is end-of-life by Texas Instruments).

https://github.com/Koenkk/Z-Stack-firmware/blob/master/coordinator/Z-Stack_3.x.0/bin/README.md

https://www.zigbee2mqtt.io/guide/adapters/#recommended

[Hedda]

• Is there some basic test to check whether a communication from HA to the CC2530 is possible? I.e. some shell commands that I could issue from HA?

FYI, could test basic socat connection to CC2530x with zigpy-znp (Python script) which CLI commands for some tools like backup:

https://github.com/zigpy/zigpy-znp/blob/dev/TOOLS.md

https://github.com/zigpy/zigpy-znp

https://pypi.org/project/zigpy-znp/

[sergeolkhovik]

Hi all, I am experimenting with @oxan's custom component and hitting a wall. ... Zigbee2MQTT:error 2021-11-21 11:42:38: Exiting... Zigbee2MQTT:error 2021-11-21 11:42:39: Error: Failed to connect to the adapter (Error: SRSP - SYS - ping after 6000ms)

That's really strange and interesting, as I'm experiencing the same error but via direct USB connection: https://github.com/Koenkk/zigbee2mqtt/issues/2997 I had to revert down to 1.18.3, and going to try that USB reset tool (not sure if I can run it inside the docker), and reflash latest 3.0 on my CC2530.

[RoganDawes]

You could also enable ESPHome serial logging, so that you can see what the dongle receives and responds. Alternatively, logging on the TCP side of the stream-server, to reduce the number of lines of log output.

I modified my stream_server.cpp with the following:

std::string StreamServerComponent::hexencode(const uint8_t *data, uint32_t off, uint32_t len) {
  char buf[20];
  std::string res;
  for (uint32_t i = off; i < off + 16; i++) {
    if (i + 1 <= off + len) {
      sprintf(buf, "%02X ", data[i]);
    } else {
      sprintf(buf, "   ");
    }
    res += buf;
  }
  res += "|";
  for (uint32_t i = off; i < off + 16; i++) {
    if (i + 1 <= off + len) {
      if (data[i]>0x1F && data[i]<0x7F) {
        sprintf(buf, "%c", data[i]);
      } else {
        sprintf(buf, ".");
      }
    } else {
      sprintf(buf, " ");
    }
    res += buf;
  }
  res += "|";
  return res;
}

and

void StreamServerComponent::read() {
    int len;
    while ((len = this->stream_->available()) > 0) {
        char buf[128];
        size_t read = this->stream_->readBytes(buf, min(len, 128));
        for (int i = 0; i < read; i += 0x10) {
          int l = std::min(static_cast<int>(read)-i, 0x10);
          ESP_LOGD(TAG, "(%d) Read : %s", this->port_, this->hexencode((uint8_t *)buf, i, l).c_str());
        }
        for (auto const& client : this->clients_)
            client->tcp_client->write(buf, read);
    }
}

void StreamServerComponent::write() {
    size_t len;
    while ((len = this->recv_buf_.size()) > 0) {
        this->stream_->write(this->recv_buf_.data(), len);
        for (int i = 0; i < len; i += 0x10) {
          int l = std::min(static_cast<int>(len)-i, 0x10);
          ESP_LOGD(TAG, "(%d) Write: %s", this->port_, this->hexencode((uint8_t *)this->recv_buf_.data(), i, l).c_str());
        }
        this->recv_buf_.erase(this->recv_buf_.begin(), this->recv_buf_.begin() + len);
    }
}

Just added logging lines.