Port mapping, IP assignment

[Kobbe1]

Hi there again,

Not sure if this is an issue, or feature request, the problem might of course also be in front of the screen (me) :)

So I use this project to extend wifi into my concrete garage by putting the esp with external antenna into one of the vent holes, so it gets internet from outside and transfers it though the steel concrete/vent hole. Works very well - I get at least a few Mbit of internet inside now.

But for some devices I require specific ports, for example if there is an arduino with OTA in the downstream network (192.168.4.x) I can see it from upstream in the Arduino IDE (192.168.178.x) but OTA fails. I guess this is because specific ports need to be opened/translated.

So I thought I use the portmap feature. So when I do a portmap against 192.168.4.2 and click apply the ESP reboots, and afterwards the device the portmap was for actually had 192.168.4.5. So it seems portmap is more or less not usable as the IP addresses are distributed randomly after a reboot of the esp (or more likely by connection order so kind of random) with no way to fix them to the MAC?

Anything I am doing wrong? Is there a way to automatically transfer all ports or just make the downstream network part of the upstream network, ie. the ESP would just be an router? I don't have much detailed knowledge about networks, more of an electronics guy, maybe it is a similar question as issue 73 of disabling the NAT functionality? Would the "automesh" functionality achieve this (but it seems to only exist for ESP8266 so far).

Best,

-T

[dchristl]

Hello @Kobbe1 ,

I'm not entirely sure if I understand everything correctly or what the open ports are needed for. Which ones are they? Also, I'm not entirely sure about your network topology. It seems like your upstream IDE is in a different network (192.168.178.xx) than the ESP router (192.168.4.xx). This means that port forwarding wouldn't work. The two networks are separate, so setting up manual routing and disabling the NAT feature will be the only way to make it work. Just as described in the linked issue (#73, look at this comment, 1st point if your router doesn't support it ). Unfortunately, I'm not much of a networking expert when it comes to routing.

So I thought I use the portmap feature. So when I do a portmap against 192.168.4.2 and click apply the ESP reboots, and afterwards the device the portmap was for actually had 192.168.4.5. So it seems portmap is more or less not usable as the IP addresses are distributed randomly after a reboot of the esp (or more likely by connection order so kind of random) with no way to fix them to the MAC?

Even though it probably won't solve your problem, there is still an open feature here that I want to implement. What's missing is the ability to assign static IP addresses to the clients. It's been on my to-do list for a while, but I haven't had the chance to work on it yet.

Kind regards, Danny

[Kobbe1]

Thanks Danny, the network ist just a normal home network with a fritz!box, I have no idea why the downstream network got an different ip range. I set it up and this is the way it was. I mean, isn't it the ESP which defines that? But it does not really allow to set that IP, does it, we can only set the class, so not sure why. The static IPs will fix some of the problem. As an example check here, ESPHome uses certain ports to allow OTA. link

Thanks a lot!

-T

[dchristl]

the network ist just a normal home network with a fritz!box, I have no idea why the downstream network got an different ip range.

This is correct and the project works this way. An additional network with its own address range is set up. By default, these two networks cannot communicate with each other easily. Therefore, an additional route needs to be configured, and NAT must be turned off. Like described in the linked issue. Since you have a FritzBox, and routing is not supported on this router (I have one myself), you need to set up a static route on the machine running the Arduino IDE; then it will work.

The static IPs will fix some of the problem. As an example check here, ESPHome uses certain ports to allow OTA. link

I believe the port is opened by the Arduino IDE machine, and the ESP32 just needs to reach that port. For OTA updates, no port forwarding is required on the Arduino. The ESP32 NAT router also supports OTA updates, and these are downloaded directly from the internet (using port 443 here), and no port forwarding is necessary. This would also work within the local network if the ESP32 and the OTA update server are in the same network.

[Kobbe1]

I did some more testing. Within the downstream network (ie. laptop part of 192.168.4.x) things work as expected. Note the -p indicating the port.

C:\Users\xxx\AppData\Local\Arduino15\packages\esp32\hardware\esp32\2.0.11/tools/espota.exe -r -i 192.168.4.33 -p 3232 --auth=esp_dc_charger -f C:\Users\xxx\AppData\Local\Temp\arduino_build_487020/DCDC_digipot_r05.ino.bin 
Sending invitation to 192.168.4.33 
Authenticating...OK
...

If I try this in the upstream network it fails. And looking at it further the issue is not really relating to the ports in the first place, as I can not actually ping 192.168.4.33 from my 192.168.178.xx network. I thought I was able to ping a downstream adress before, but maybe this never worked - and maybe what I am trying to do here is not actually intended to work - ie. direct IP connection from up to downstream device though the NAT? Not really a big deal if it isn't, right now I can still just walk over and connect a cable :)

It might also be different for the OTA initiated through the ESP (like this project here uses) instead from the IDE, but right now my project has no webpage, just uses Homeassistant. Well actually Homeassistant's protocol works fine across the NAT in both directions so ... Mhmm, I am just no network guy I guess :)

[Kobbe1]

I did some more testing, static routing and disabling the NAT seems to be an feasible option. But I have multiple devices connected to the extender network, it really looks like the assignment of IPs is random after a reboot. So it is hard to set the routes up if the target IP changes and I don't know who is who. Maybe one workaround is to set static IPs in those devices directly, right now they are just on auto DHCP ... Will see if that works.

[Kobbe1]

So more testing: if I do something like "route add 192.168.4.2 MASK 255.255.255.255 192.168.178.xxx" on a windows machine I can reach and OTA the device downstream. There are a few small issues with that:

• I first need to turn off NAT of course on the ESP32 so any other devices in the downstream network loses connection. I would guess if I could tell the other downstream devices manually a similar route command they would not lose access (internet) and I could keep NAT off permanently? But they are ESPs so it gets complicated I guess
• And as I said before it is a bit hard to figure out who is who, so I now wrote down the MAC of all devices, thenI can manually find the assigned IP address through the webinterface and set the route command accordingly. After reboot it is all IPs are messed up again :)

I think I can not tag @Stan-Tastic here from issue 73, I wonder if there is a simple solution that I am missing.

In any case the ESP NAT here already provides the most important feature for me (ie. extending the network into my well shielded garage and elsewhere) so do not consider this a bug report, instead more of a feedback and help for others ...

Thanks

[dchristl]

Thanks for testing, the only thing I can do here is to build a possibility to manage the static IPs, like I mentioned before. The assignment is currently not totally random, it depends which device is conencted first. This gets the 192.168.4.2, the next on 192.168.4.3, ... So, you could power on the devices in the needed order, if possible until the feature is implemented.

It might also be possible to use 2 ESPs (with 2 different networks). One with NAT enabled and one without. How often do you actually need the connection to the Arduino IDE? If it's only occasionally, then it would also be possible to keep one ESP in "normal mode" with NAT enabled all the time, and when firmware needs to be updated (meaning a connection to the IDE is needed), temporarily deploy a second ESP with NAT disabled. And power off the NAT-enabled ESP during this time.

[Kobbe1]

Yes, I think the static IPs will help for sure. I can work around things, it is not a big deal. It just somehow feels there should be an easier way for what I am trying, after all I would just want to extend the existing homework to my concrete garage without having to drill big holes and routing cables :)

Maybe one of those fancy Fritz mesh things would work - but my fritzbox is too old for that and also it would not fit in one of the vent holes of the garage - the ESP32 + antenna fits in perfectly and by that it can communicate to the outside existing wifi and also to the inside downstream network.

But as I said this is issue more of a nice-to-have, I would say 80% of my requirement is achived already, inet access works well inside ... Thanks again - feel free to close, as you said, nothing really to do here.

[Kobbe1]

Just a quick extension of above. So with the above static routing we can reach the devices in the downstream network by defining a static route. This works on windows and linux. Great.

Now I thought I try the same from the downstream network to the upstream. Did not work. But then for fun I enabled the NAT again - and guess what, it works anyway in this direction.

So in conclusion the devices from the downstream network can reach all IP adresses upstream with the NAT enabled, but to reach downstream devices from the upstream network we need to disable the NAT and add a static route as above. And both at the same time is not possible. I think this clearly shows I do not understand much about NATs :)

[Stan-Tastic]

Hi, thanks for tagging me :-)

For me the most important question is what router are you using upstream of your network? I understand that 192.168.178.0/24 is your home network, and ESP router is connected to this network (for sake of further discussion let's assume it has address 192.168.178.10), but what device is connecting 192.168.178.0/24 to the Internet? (anyway, I will assume it's address is 192.168.178.1 - this device is responsible for NAT for 192.168.178.0/24 network)

I have no way to test it now, but I believe that even when NAT is enabled, and you force routing on your machine to say that 192.168.4.2 is reachable via 192.168.178.10, the ping will be sent to proper machine - however, ping reply will come from 192.168.178.10, because NAT. In general, let's not do that ^^.

I don't quite understand why your 192.168.4.0/24 devices lose connectivity when NAT is disabled. My hunch is that on your upstream router (192.168.178.1) is configured so that it source addresses from network 192.168.178.0/24 are translated, thus ones from 192.168.4.0/24 will NOT be translated, thus your ISP will not know how to route those packets back (in fact, it will discard them, as 192.168.0.0/16 is private network range).

So if at all possible, your upstream device might have to be reconfigured so that it

1. Can translate packets from 192.168.4.0/24 network
2. Knows where to send packets for 192.168.4.0/24 (static route via 192.168.178.10)
3. (optionally) setup DHCP option so that other devices in 192.168.178.10 will know where to send packets destined for 192.168.4.0/24

Additional note: the clients connecting to ESP router might be able to request specific IP address from DHCP (the protocol allows for that, but not sure if it's implemented in this DHCP stack), or be configured with static IP address (ESPHome can do this for certain, not sure about other ESP software).

Let me know if I can offer any more help.

[Kobbe1]

Hi, thanks for the reply!! This is an interesting issue I think. And also perfect excuse to hide in the garage :)

For me the most important question is what router are you using upstream of your network? I understand that 192.168.178.0/24 is your home network, and ESP router is connected to this network (for sake of further discussion let's assume it has address 192.168.178.10), but what device is connecting 192.168.178.0/24 to the Internet? (anyway, I will assume it's address is 192.168.178.1 - this device is responsible for NAT for 192.168.178.0/24 network)

It is a Fritzbox router, no special setting apart from an PiHole used as DNS. It has an option to set a static route, so I think this can be used to make downstream devices reachable to all devices upstream (when NAS is off), like the route ADD command I did for jsut one machine. It has only very basic settings otherwise, and I don't dare to change it to openwrt as I think that will break the ISP connection, it is technically the ISP's device.

I have no way to test it now, but I believe that even when NAT is enabled, and you force routing on your machine to say that 192.168.4.2 is reachable via 192.168.178.10, the ping will be sent to proper machine - however, ping reply will come from 192.168.178.10, because NAT. In general, let's not do that ^^.

I had switched off the downstream device, and then I could not ping it. So I guess it was actually the device (well or the ESP only responds with ping if the device is connected). But that part (static route) works anyway so guess no need to worry about it. The downstream device was in DHCP client mode. I did not try to set a static IP from 192.168.178.x - maybe try that next if you recommend.

I don't quite understand why your 192.168.4.0/24 devices lose connectivity when NAT is disabled. My hunch is that on your upstream router (192.168.178.1) is configured so that it source addresses from network 192.168.178.0/24 are translated, thus ones from 192.168.4.0/24 will NOT be translated, thus your ISP will not know how to route those packets back (in fact, it will discard them, as 192.168.0.0/16 is private network range).

Tested again, on a different machine.

NAT off:

• No internet, - can't reach anything in 192.168.178.x, can't ping 8.8.8.8

NAT on:

• Exact opposite of the above

So if at all possible, your upstream device might have to be reconfigured so that it

1. Can translate packets from 192.168.4.0/24 network
2. Knows where to send packets for 192.168.4.0/24 (static route via 192.168.178.10)
3. (optionally) setup DHCP option so that other devices in 192.168.178.10 will know where to send packets destined for 192.168.4.0/24

The router does not expose any DHCP settings at all.

Additional note: the clients connecting to ESP router might be able to request specific IP address from DHCP (the protocol allows for that, but not sure if it's implemented in this DHCP stack), or be configured with static IP address (ESPHome can do this for certain, not sure about other ESP software).

So you are saying I might be able to request a static IP in the downstream network from the upstream DHCP server (ie. my downstream device then would also have 192.168.178.x IP). So then it becomes the same network (or is it subnet?) essentially (?)

Let me know if I can offer any more help.

Ha, well, as you can see I don't know much about network architecture, but I am willing to experiment for my own and the benefit of others. So I guess from above I could try to set a static IP from the 192.168.178.x pool? I have doubts this would work as I can't ping the DHCP server with NAT off. Anyway, open for ideas or experiments, the devices I have available in the downstream network: several windows machines, linux, ESP32 w/ ESPHome, and an ESP32 running code from this repo: https://github.com/dawidchyrzynski/arduino-home-assistant - so sort of an arduino ESPHome. A wifi router with openwrt that I could connect.

Cheers!

[heyitsyang]

This is an awesome project! Thanks for all your work. Any progress on static IPs?

I have a similar problem. I am using the nat_router to reach a OTA programmable battery powered IoT device that sleeps most of the time. I've played with the Port Mapping, but the changing client IPs make it impossible to use:

• My battery powered client device sleeps and wakes periodically, so I can't rely on it getting the same IP address each time
• I can give the client device a static IP address, but the nat_router needs to know not to assign that static address

There are a couple of possible workarounds:

1. Rather than implementing reserved addresses on the nat_router, perhaps there is a quick way to exclude an address (or range) from assignment. This may be a quick CLI way without needing all of the UI work.
2. Given the IP address assignment simply increments up, I can assign a static address on my client that is high in the IP range something like xxx.xxx.xxx.254.

Do you see any problem with workaround #2 ?

[dchristl]

Hello @Stan-Tastic ,

thank you for your nice words. Unfortunately, I currently lack the time to continue working on this. The main issue is that I currently don't have an approach or examples on how to integrate a custom DHCP server with esp-idf. However, this is a prerequisite for static IPs and/or address excludes.

At the moment, I don't see any issues with the second point, but I haven't tried it myself yet.

Kind Regards, Danny

[heyitsyang]

Thanks for your reply. I tried option 2 and it doesn't work. When I use a static address, the Port Mapping table apparently will not list an address it hasn't issued. The IP address entered as 192.168.0.254 shows in the table as 000.000.000.000 after it has been applied. Oh well, I'll keep playing with it a bit more & see if I can trick it into working.