Missing characters at speeds >38400Bd on target to PC/telnet connection

[ing-olf]

Hi guys, I've an ESP-Link (esp-link v2.2.3 - 2016-06-21) running as transparent link between a µController UART and a PC. My actual test setup is a STM32F103 on a Blue-Pill running Forth and a PC running telnet on Linux.

I noticed that the ESP-Link does not transport all characters sent from the µC. For example, if I want to see the word list in Forth (typing in 'words') a huge list of should be shown on my telnet console, but there are blocks missing.

This is also seen on the the ESP-LINK Microcontroller Console. I assume this is an issue on receiving bytes at the UART of the ESP-Link. So I reduced the serial speed from 115200 in steps until I got a complete transmission and landed at 38400Bd.

But I wanted to increase the speed by 4 (115200 -> 460800) not to decrease by 4 ;-) Is there any chance to get around this issue? Happy programming

Ingolf, DH0PE

[uzi18]

try with latest version : esp-link-v3.2.47.alpha.5-ga0d3bb3-dirty-Lucjan3.tgz.zip

[ing-olf]

Thanks for the hint - I just tried that version, but no success. It's still the same. If You transfer a big block the ESP-LINK stops the transfer somehow during transfer. Sometimes after a few bytes, sometimes after a big block but never complete. Ingolf, DH0PE

[uzi18]

esp buffer is limited, is it possible for you to find size of this big block? maybe also debug log after issue words command?

[ing-olf]

Ok, I did some trials at 115200Bd, most blocks are in the range 3400 bytes (2888 to 3680). I thought the esp is using a big enough ring buffer to receive stuff while sending the other half of the buffer ...

[uzi18]

in fact maybe here is a place to improve, need to check sources

[uzi18]

console log is limited to 1000bytes

[ing-olf]

Thanks for checking. Does this also influences the telnet connection? I measured the 3000-something bytes via telnet. I just let my micro send bytes continuously and receiving via telnet.

[uzi18]

No it is only console log, maybe I can try to increase it so you can try to check if console log will be better?

[ing-olf]

Hi, thanks for Your reply, telnet and console log have the same problem. Let me know, if and how I can contribute.

[vytautasmiel]

Dear Ing-Olf,

have you managed to fix it?

I'm having the same problem with ESP-01 and esp-link v3.2.47.alpha.16-gcf329e8 firmware. ESP-01 works as serial bridge (115200bps) between Arduino DUE and linux/telnet PC client. From time to time (unfortunately, I cannot detect any pattern), telnet data stream looses some data. Messages sent by Arduino are way shorter than 1000 bytes long.

[ing-olf]

Hi, no I did not solve the problem, I reduced the baudrate down to 38400. I don't have time and knowledge to debug the esp-link. I'm still looking for an alternative. I just need a transparent serial bridge...

[vytautasmiel]

Thanks for reply. Lowering the communication speed does not look attractive but I would consider it if I was absolutely sure it works. I have no idea how to replicate the problem, what makes testing complicated - it just happens occasionally @115200 bps. Have you ever observed this problem @38400 bps?

[ing-olf]

To make sure, we talk about the same issue: My system looses characters on the way from target to PC. I can reproduce this while sending data from target via the esp-link. It starts at about 3400 bytes (2888 to 3680). So I assume a buffer overrun. It's a pity, the esp-link is so nice to use but it seems no one sends enought data from target to PC ;-) Everybody is downloading code to the target ...

[vytautasmiel]

Yes, I am trying to use the same way - code of Arduino DUE outputs data to serial port which is connected to ESP01. Controller application which runs on linux server connects to ESP01 via telnet (port 23) and processes data received from Arduino DUE. Messages from Arduino are pretty short (up to 100bytes) but occasionally they come even shorter than supposed to :(

Anyway, thanks for sharing your experience.

[tve]

I don't have time and knowledge to debug the esp-link.

There is no need to "debug esp-link", it's simple math. At 115200 baud it takes 1/11520th of a second to transmit a character (8 data bits + start bit + stop bit). That means 11.52 characters per millisecond. TCP implementations have a great feature, which is to delay ACKs: it allows multiple ACKs to be combined and ACKs to be combined with data sent back at the application level. Various TCP stacks routinely delay ACKs by up to 500ms. Windows is particularly notorious (I last tested that with Win7). Well, 500ms * 11.52char/ms = 5760 bytes. And that doesn't count the fact that there need to be pre-allocated buffers for the next chars, to RX stuff, etc. When you consider that esp-link generally runs with 8-15KB free memory you realize that there is just no headroom. Any Wifi hiccup or lost packet or concurrent web page access, etc, causes an out-of-memory situation which results in lost characters.

it seems no one sends enough data from target to PC ;-) Everybody is downloading code to the target ...

Downloading code to the target has worse issues in a way: when programming an AVR there are hiccups of multiple hundred milliseconds when waiting for it to sync and additional smaller but significant hiccups when waiting for flash blocks to be programmed. The reason it works is end-to-end flow-control. The problem with what you're doing is lack of flow-control.

I'm still looking for an alternative.

Something very bare-bones on the esp8266 will have a bit more memory and may work better if it gets you over the main network hiccup threshold, so to speak. But there's no magic bullet, buffering requires memory. Using an esp32 could give you lots of buffer space and a better Wifi implementation.

[uzi18]

or we can think more about UDP protocol?

[yesyesuk]

or we can think more about UDP protocol?

That's what I was thinking at first. But then nothing is guaranteed and you may lose whole packets instead of individual characters.

[uzi18]

there is a patch from @leodesigner to merge all buffers in esp-link maybe we can test it with this issue?

[leodesigner]

Works reliable at 1.5 Mbit/s

https://github.com/leodesigner/esp-link/commit/f5b9a5633feab587a6b6c526557d3ac908030275

You can test my fork at: https://github.com/leodesigner/esp-link

[ing-olf]

@tve Thanks for the explanation ... I'm not so well enlighted in TCP etc. I'm just a hardware guy. @leodesigner I'll try it. It'll take me some time to put up an environemnt to compile, or miss I a ready built firmware?

[uzi18]

Will try to upload here prebuilt bins with this patch on top master but need to find a time. @leodesigner have own built images here: https://github.com/leodesigner/esp-link/tree/master/firmware

[ing-olf]

@leodesigner sorry, but I've no success sending continuous characters from the target to the PC via Telnet with baudrates > 38400bps. I always loose big blocks of bytes. So tve's explanation makes sense to me and I'll try an ESP32 in my next project as an "serial over wifi" connector. As a newcomer to this wifi software stuff I did not expect such a limitation - but learning curves are steep in the beginning ...