esp8266
ESP8266WifiServer timeout when requesting through external IP
I have a basic ESP8266WebServer that sends roughly 1000 characters worth of response via send(). Everything works fine when the traffic is over the local IPv4 address 192.168.xx.xx.
However, I have set up port forwarding over my external IP to be directed to the NodeMCU device, and when the request is made over the external IP 87.xx.xx.xx via a device connected to the local network, the server times out during send() operation. Upon closer inspection, it seems that ClientContext::availableForWrite() / tcp_sndbuf returns 0 in this case. Increasing the timeout to 10s didn't help. Seems the buffer is never available.
As an additional step, I set up a VPN on the request device to make the request device's IP as 105.xx.xx.xx and made the same request. This time ClientContext::availableForWrite() / tcp_sndbuf returned a non-zero value and the request was successful.
Any ideas why the tcp_sndbuf would not work when a request is made by a device connected on the same network but via external IP that is port forwarded to my NodeMCU?
You can try and play with the embedded Netdump library which will eventually show the differences between your port redirect and your vpn setup.
That sketch crashes intermittently with (decoded exception)
Exception 28: LoadProhibited: A load referenced a page mapped with an attribute that does not permit loads
PC: 0x4021ff24: tcp_output at core/tcp_out.c line 1390
EXCVADDR: 0x0000000c
Decoding stack results
0x402250e3: ip_chksum_pseudo at core/inet_chksum.c line 392
0x4021fe3e: tcp_output at core/tcp_out.c line 1621
0x4020ac81: ClientContext::_write_from_source(char const*, unsigned int) at /home/nish/.arduino15/packages/esp8266/hardware/esp8266/3.1.1/libraries/ESP8266WiFi/src/include/ClientContext.h line 474
0x4020d844: Stream::SendGenericPeekBuffer(Print*, int, int, unsigned int) at /home/nish/.arduino15/packages/esp8266/hardware/esp8266/3.1.1/cores/esp8266/Print.h line 63
0x4020d064: Print::println(int, int) at /home/nish/.arduino15/packages/esp8266/hardware/esp8266/3.1.1/cores/esp8266/Print.cpp line 238
0x40207ac4: esp8266webserver::ESP8266WebServerTemplate ::send(int, char const*, Stream*, unsigned int) at /home/nish/.arduino15/packages/esp8266/hardware/esp8266/3.1.1/cores/esp8266/Stream.h line 64
0x4020dacb: String::changeBuffer(unsigned int) at /home/nish/.arduino15/packages/esp8266/hardware/esp8266/3.1.1/cores/esp8266/WString.h line 328
0x40207be7: esp8266webserver::ESP8266WebServerTemplate ::send(int, char const*, String const&) at /home/nish/.arduino15/packages/esp8266/hardware/esp8266/3.1.1/cores/esp8266/WString.h line 115
0x4020db5f: String::reserve(unsigned int) at /home/nish/.arduino15/packages/esp8266/hardware/esp8266/3.1.1/cores/esp8266/WString.cpp line 192
0x4020de01: String::String(unsigned char, unsigned char) at /home/nish/.arduino15/packages/esp8266/hardware/esp8266/3.1.1/cores/esp8266/WString.cpp line 44
0x40207d5a: std::_Function_handler >::_M_invoke(const std::_Any_data &) at /home/nish/.arduino15/packages/esp8266/hardware/esp8266/3.1.1/cores/esp8266/WString.h line 115
0x40205e95: esp8266webserver::FunctionRequestHandler ::handle(esp8266webserver::ESP8266WebServerTemplate &, HTTPMethod, String const&) at /home/nish/.arduino15/packages/esp8266/hardware/esp8266/3.1.1/libraries/ESP8266WebServer/src/detail/RequestHandlersImpl.h line 62
0x40207c34: esp8266webserver::ESP8266WebServerTemplate ::_handleRequest() at /home/nish/.arduino15/packages/esp8266/tools/xtensa-lx106-elf-gcc/3.1.0-gcc10.3-e5f9fec/xtensa-lx106-elf/include/c++/10.3.0/bits/std_function.h line 498
0x4020b3a0: WiFiServer::accept() at /home/nish/.arduino15/packages/esp8266/hardware/esp8266/3.1.1/libraries/ESP8266WiFi/src/WiFiServer.cpp line 147
0x40208172: esp8266webserver::ESP8266WebServerTemplate ::handleClient() at /home/nish/.arduino15/packages/esp8266/hardware/esp8266/3.1.1/libraries/ESP8266WebServer/src/ESP8266WebServer-impl.h line 282
0x4020ee2c: loop_wrapper() at /home/nish/.arduino15/packages/esp8266/hardware/esp8266/3.1.1/cores/esp8266/core_esp8266_main.cpp line 258
raw:
11:37:16.130 -> 5780 0 out HTTP 192.168.178.27>192.168.178.76 80:44384 [PA] len: 114 seq: 6511, ack: 3693242410, wnd: 1696
11:37:16.130 -> H .P.`...o."h*P...$...
11:37:16.130 -> D HTTP/1.1 200 OK..Content-Type: text/html..Content-Length: 50..Connection: keep-a
11:37:16.163 -> D live..Ke 5798 0 in HTTP 192.168.178.76>192.168.178.27 44384:80 [A] len: 0 seq: 3693242410, ack: 6625, wnd: 64126
11:37:16.163 -> H .`.P."h*....P..~....
11:37:16.163 -> 5811 0 out HTTP 192.168.178.27>192.168.178.76 80:44384 [A] len: 0 seq: 6511, ack: 3693242410, wnd: 1696
11:37:16.163 -> H .P.`...o."h*P.......
11:37:16.163 -> 5824 0 out HTTP 192.168.178.27>192.168.178.76 80:44384 [PA] len: 114 seq: 6511, ack: 3693242410, wnd: 1696
11:37:16.196 -> H .P.`...o."h*P...$...
11:37:16.196 -> D HTTP/1.1 200 OK..Content-Type: text/html..Content-Length: 50..Connection: keep-a
11:37:16.196 -> D live..Keep-Alive: timeout=2000....
11:37:16.196 -> ep-Alive: timeout=2000....
11:37:16.196 ->
11:37:16.196 -> --------------- CUT HERE FOR EXCEPTION DECODER ---------------
11:37:16.196 ->
11:37:16.196 -> Exception (28):
11:37:16.229 -> epc1=0x4021ff24 epc2=0x00000000 epc3=0x00000000 excvaddr=0x0000000c depc=0x00000000
11:37:16.229 ->
11:37:16.229 -> >>>stack>>>
11:37:16.229 ->
11:37:16.229 -> ctx: cont
11:37:16.229 -> sp: 3ffffbc0 end: 3fffffd0 offset: 0190
11:37:16.229 -> 3ffffd50: 3ffeee84 00000000 00000000 402250e3
11:37:16.229 -> 3ffffd60: 00000001 00000072 3fff01e4 4021fe3e
11:37:16.229 -> 3ffffd70: 3ffeee84 3fff034c 3fff0164 00000000
11:37:16.229 -> 3ffffd80: 00000860 3fff0040 0000196f 00000072
11:37:16.262 -> 3ffffd90: 3ffffd00 00000001 3ffffe54 00000002
11:37:16.262 -> 3ffffda0: 00000001 00000000 3ffefe54 4020ac81
11:37:16.262 -> 3ffffdb0: 00000072 00000000 3ffeea1c 00000072
11:37:16.262 -> 3ffffdc0: 3ffee89c 00000007 3ffeea1c 00000072
11:37:16.262 -> 3ffffdd0: 3ffee89c 3ffeea1c 3ffffe30 4020d844
11:37:16.262 -> 3ffffde0: 04c4b400 5611d6d9 3ffeea00 4020d064
11:37:16.262 -> 3ffffdf0: 00000000 ffffffff 00000000 ffffffff
11:37:16.262 -> 3ffffe00: 00000000 ffffffff 00000000 3fff01e4
11:37:16.295 -> 3ffffe10: 3ffe87f0 00000000 3fffff08 000000c8
11:37:16.295 -> 3ffffe20: 3ffffe90 3ffee868 00000032 40207ac4
11:37:16.295 -> 3ffffe30: 4021233c 00000000 000003e8 00000043
11:37:16.295 -> 3ffffe40: 00000000 3fff01e4 00000072 3fff0101
11:37:16.295 -> 3ffffe50: 00000000 3fff01e4 0072007f 89000020
11:37:16.295 -> 3ffffe60: 3ffffeb4 00000040 3ffffeec 4020dacb
11:37:16.295 -> 3ffffe70: 3fff011c 00000009 3ffffeb4 3ffe8884
11:37:16.328 -> 3ffffe80: 3fff011c 000000c8 3ffee868 40207be7
11:37:16.328 -> 3ffffe90: 4021233c 00000000 000003e8 4020db5f
11:37:16.328 -> 3ffffea0: 00000000 3fff011c 00000032 4020de01
11:37:16.328 -> 3ffffeb0: 00000000 74786574 6d74682f 0900006c
11:37:16.328 -> 3ffffec0: 3ffe8847 3ffe8857 3ffffeec 3ffeeadc
11:37:16.328 -> 3ffffed0: 3fffdad0 00000001 3ffefa44 40207d5a
11:37:16.328 -> 3ffffee0: 00000000 002d002f 00000000 00000000
11:37:16.328 -> 3ffffef0: 0032003f 00000000 3fff011c 0032003f
11:37:16.361 -> 3fffff00: 81000000 00000000 00000000 3ffeeadc
11:37:16.361 -> 3fffff10: 3fffdad0 3ffee8c0 3ffefa44 40205e95
11:37:16.361 -> 3fffff20: 00000000 3ffee8c0 3ffee868 40207c34
11:37:16.361 -> 3fffff30: 3fffffa0 00000000 00000000 000e000f
11:37:16.361 -> 3fffff40: 00000000 3ffeea1c 3ffee87c 4020b3a0
11:37:16.361 -> 3fffff50: 00000000 0000168c 00000000 00000001
11:37:16.361 -> 3fffff60: 00000000 00000000 3ffee868 3ffeeadc
11:37:16.394 -> 3fffff70: 3fffdad0 3ffee89c 3ffee868 40208172
11:37:16.394 -> 3fffff80: 40212448 00000000 00001388 3ffeeadc
11:37:16.394 -> 3fffff90: 00000000 00000000 3ffefe54 00000000
11:37:16.394 -> 3fffffa0: 3fffdad0 00000000 3ffeeab0 3ffeeadc
11:37:16.394 -> 3fffffb0: 3fffdad0 00000000 3ffeeab0 4020ee2c
11:37:16.394 -> 3fffffc0: feefeffe feefeffe 3fffdab0 40100d79
11:37:16.394 -> <<<stack<<<
11:37:16.394 ->
11:37:16.394 -> --------------- CUT HERE FOR EXCEPTION DECODER ---------------
11:37:16.427 ->
11:37:16.427 -> ets Jan 8 2013,rst cause:2, boot mode:(3,6)
11:37:16.427 ->
11:37:16.427 -> load 0x4010f000, len 3424, room 16
11:37:16.427 -> tail 0
11:37:16.427 -> chksum 0x2e
11:37:16.427 -> load 0x3fff20b8, len 40, room 8
11:37:16.427 -> tail 0
11:37:16.427 -> chksum 0x2b
11:37:16.427 -> csum 0x2b
I tried to mimize the sketch to only contain:
webServer.on("/req", []() {
static int rq = 0;
String a = "<h1>You are connected, Number of requests = " + String(rq++) + "</h1>";
webServer.send(200, "text/html", a);
});
webServer.on("/reset", []() {
nd.reset();
tracefile.close();
tcpServer.close();
webServer.send(200, "text.html", "<h1>Netdump session reset</h1>");
});
// webServer.serveStatic("/", *filesystem, "/");
webServer.begin();
startSerial(SerialOption::HTTPChar);
Even through the crashes I was able to analyze the tcpdump on the client side and the nodemcu side:
It seems that over external IP the ACKs for the HTTP body are not reaching the NodeMCU, but over a VPN they do. I'm still confused so as why it is like this.
External IP (hangs on body receive)
13:14:06.077418 IP nodemcu.network.9980 > home-pc.local.48394: Flags [P.], seq 6511:6627, ack 2424900121, win 2060, length 116
13:14:06.077465 IP home-pc.local.48394 > nodemcu.network.9980: Flags [R], seq 2424900121, win 0, length 0
13:14:06.952399 IP home-pc.local.41472 > nodemcu.network.9980: Flags [S], seq 110092386, win 64240, options [mss 1460,sackOK,TS val 3588784746 ecr 0,nop,wscale 7], length 0
13:14:07.099363 IP nodemcu.network.9980 > home-pc.local.41472: Flags [S.], seq 6552, ack 110092387, win 2144, options [mss 536,nop,nop,sackOK], length 0
13:14:07.099388 IP home-pc.local.41472 > nodemcu.network.9980: Flags [.], ack 1, win 64240, length 0
13:14:07.099430 IP home-pc.local.41472 > nodemcu.network.9980: Flags [P.], seq 1:84, ack 1, win 64240, length 83
13:14:07.142672 IP nodemcu.network.9980 > home-pc.local.41472: Flags [.], ack 84, win 2061, length 0
13:14:07.155944 IP nodemcu.network.9980 > home-pc.local.41472: Flags [P.], seq 1:117, ack 84, win 2061, length 116
13:14:07.155954 IP home-pc.local.41472 > nodemcu.network.9980: Flags [.], ack 117, win 64124, length 0
13:14:07.217393 IP nodemcu.network.9980 > home-pc.local.41472: Flags [.], ack 84, win 2061, length 0
13:14:07.217407 IP home-pc.local.41472 > nodemcu.network.9980: Flags [.], ack 117, win 64124, length 0
13:15:07.639824 IP home-pc.local.41472 > nodemcu.network.9980: Flags [.], ack 117, win 64124, length 0
13:15:07.774334 IP nodemcu.network.9980 > home-pc.local.41472: Flags [R.], seq 117, ack 83, win 24584, length 0
External IP over VPN (good)
13:04:27.819146 IP nish-pc.53648 > nodemcu.network.9980: Flags [F.], seq 2573049806, ack 8445, win 64116, length 0
13:04:27.881189 IP nodemcu.network.9980 > nish-pc.53648: Flags [R.], seq 1, ack 1, win 24584, length 0
13:04:32.129136 IP nish-pc.59006 > nodemcu.network.9980: Flags [S], seq 33849241, win 64768, options [mss 1408,sackOK,TS val 2649173485 ecr 0,nop,wscale 7], length 0
13:04:32.283496 IP nodemcu.network.9980 > nish-pc.59006: Flags [S.], seq 6510, ack 33849242, win 2144, options [mss 536,nop,nop,sackOK], length 0
13:04:32.283524 IP nish-pc.59006 > nodemcu.network.9980: Flags [.], ack 1, win 64768, length 0
13:04:32.283565 IP nish-pc.59006 > nodemcu.network.9980: Flags [P.], seq 1:84, ack 1, win 64768, length 83
13:04:32.357765 IP nodemcu.network.9980 > nish-pc.59006: Flags [P.], seq 1:117, ack 84, win 2061, length 116
13:04:32.357795 IP nish-pc.59006 > nodemcu.network.9980: Flags [.], ack 117, win 64652, length 0
13:04:32.392284 IP nodemcu.network.9980 > nish-pc.59006: Flags [.], seq 117:653, ack 84, win 2061, length 536
13:04:32.392315 IP nish-pc.59006 > nodemcu.network.9980: Flags [.], ack 653, win 64116, length 0
13:04:32.483747 IP nodemcu.network.9980 > nish-pc.59006: Flags [.], seq 117:653, ack 84, win 2061, length 536
13:04:32.483764 IP nish-pc.59006 > nodemcu.network.9980: Flags [.], ack 653, win 64116, options [nop,nop,sack 1 {117:653}], length 0
13:04:32.555288 IP nodemcu.network.9980 > nish-pc.59006: Flags [P.], seq 653:1073, ack 84, win 2061, length 420
13:04:32.555308 IP nish-pc.59006 > nodemcu.network.9980: Flags [.], ack 1073, win 64116, length 0
13:05:34.203822 IP nish-pc.59006 > nodemcu.network.9980: Flags [.], ack 1073, win 64116, length 0
13:05:34.258016 IP nodemcu.network.9980 > nish-pc.59006: Flags [R.], seq 1073, ack 83, win 24584, length 0
Note that I have a RPi also on the same network behind the same kind of port-forwarding that works just fine with the external IP, so I doubt it's anything network related.
Maybe timeout related? Is the connection via the external IP rather slow, or at least relatively high latency? Does the ESP have a proper gateway configured?
I can confim similar problem. Connecting directly from lan works, but when try to access my esp using port forward / vpn it's not working. VPN client with browser -> VPN server on raspberry pi -> port redirect from raspberry to esp (raspberry and esp in the same subnet).
Page is loading very slowly, and eating whole ram of esp device
(to the deleted message and the above)
Might be useful to also record what network & firewall configuration used to pass the packets through the network. Also, how 'high bandwidth' version bares in comparison. iirc old lwip v1 had a very big problem when using mixed mss 536 vs. 1460 values, like in the log above
not sure if this will help you but building the firmware with andruino framework 4.2.1 solved the issue, the issue was present when using 2.6.3. lwip v1vs2 make no difference. In my case there was a problem with memory leak - the browser requested the site, and nothing happens, only few packets were send.. and in few minutes my esp devices runns out of memory and crushes.