Support for dynamic "ProxySourceAddress" depending on the backend

[jb-boin]

I am currently trying to set up a reverse proxy on Debian 12, the FTP server incoming connections are on a public interface while the backends are on two different VLANs, each intrerface has it's own IP on the server so it can connect directly to the backends not using a gateway.

If i don't set a ProxySourceAddress on the configuration, i can only connect to backends that are accessible from the public IP via the default gateway (which is not what i need), if i set the ProxySourceAddress on one of the internal network interface (or IP), it will then only work for the backends that are also on this network.

Is there a way for the proxy to "detect" the right source address by itself if there's a local route on the system to the backend IP address?

If not, is there a way to set a different ProxySourceAddress depending on the backend name or it's IP/network?

[Castaglia]

Hmm. Interesting situation. Currently, there isn't a way to configure a per-backend ProxySourceAddress -- but I can see how, in your case, it would be useful.

Would it be possible to get the output from ifconfig -a on your proxy host, and the backend IPs? I'd like to see what it might take to make the module "smarter" about selecting a source address, given the backend IP, based on the available interfaces. Thanks!

[jb-boin]

On the proxy (those are not the real IPs but i kept the same netmasks) :

ens32: 190.145.11.62/255.255.255.0  (/24)
ens34: 10.252.0.62/255.255.224.0  (/19)
ens35: 10.250.225.62/255.254.0.0  (/15)
lo: 127.0.0.1/255.0.0.0 (/8)

And the network routes on that system :

default via 190.145.11.1 dev ens32 onlink 
10.250.0.0/15 dev ens35 proto kernel scope link src 10.250.225.62 
10.252.0.0/19 dev ens34 proto kernel scope link src 10.252.0.62
190.145.11.0/24 dev ens32 proto kernel scope link src 190.145.11.62 

Some of the backends will be on the 10.252.0.0 network (for example 10.252.3.19/19) and some backends will be on the 10.250.225.0 network (for example 10.250.17.35/15).

If there is no other solution than determining the route using your own code, you could parse /proc/net/route to determine the right interface for a destination.

[Castaglia]

I'm researching this, and in how exactly the Linux kernel will choose the source address to use for a given TCP connection, when the source address is not explicitly provided (which is the most common case). And this in turn makes me wonder why the kernel, in your case, appears to not be choosing the correct (well, expected) source address/interface.

Would it possible to provide the ProxyLog logging, and perhaps trace logging, for the exact errors (and log messages) you see, in the cases where you do (and do not) use ProxySourceAddress? The exact error messages might provide more details/clues into what's going on here. Thanks!

(You can send this info to my personal email address, if you'd rather not post it here.)

[Castaglia]

Recording this here for my future reference: this particular issue hinges on the value of bind_addr here:

• https://github.com/Castaglia/proftpd-mod_proxy/blob/master/lib/proxy/conn.c#L752

Hopefully the provided logs/messages can help inform how and when to leave bind_addr as NULL. As I suspect that, in this particular use case, if bind_addr is set to NULL, then the Linux kernel would (hopefully) Do The Right Thing(tm).

[Castaglia]

Recording this here for my future reference: this particular issue hinges on the value of bind_addr here:

• https://github.com/Castaglia/proftpd-mod_proxy/blob/master/lib/proxy/conn.c#L752

Hopefully the provided logs/messages can help inform how and when to leave bind_addr as NULL. As I suspect that, in this particular use case, if bind_addr is set to NULL, then the Linux kernel would (hopefully) Do The Right Thing(tm).

As an experiment, here's a PR/branch you might try, which allows for this bind_addr value to be NULL, when ProxySourceAddress is not explicitly configured:

• https://github.com/Castaglia/proftpd-mod_proxy/pull/273

[jb-boin]

I'm researching this, and in how exactly the Linux kernel will choose the source address to use for a given TCP connection, when the source address is not explicitly provided (which is the most common case). And this in turn makes me wonder why the kernel, in your case, appears to not be choosing the correct (well, expected) source address/interface.

Would it possible to provide the ProxyLog logging, and perhaps trace logging, for the exact errors (and log messages) you see, in the cases where you do (and do not) use ProxySourceAddress? The exact error messages might provide more details/clues into what's going on here. Thanks!

(You can send this info to my personal email address, if you'd rather not post it here.)

I haven't had time to try your patched version yet but i got the logs you asked initially. If i connect to the public IP of ProFTPD (190.145.11.62/24) to try to access the backend 10.252.0.62/19 without setting "ProxySourceAddress" (or setting it to another interface than the right one for that backend), it does timeout connecting and i have this logged on the ProxyLog :

2024-04-24 02:20:39,743 mod_proxy/0.9.3[117588]: selected backend server 'ftp://dns.of.backend'
2024-04-24 02:20:44,946 mod_proxy/0.9.3[117588]: error obtaining local socket info on fd 16: Transport endpoint is not connected
2024-04-24 02:20:44,947 mod_proxy/0.9.3[117588]: ProxyRetryCount 5 reached with no successful connection, failing

And on the tracelog :

2024-04-24 02:19:53,676 [117586] <dns:7>: attempting to resolve 'dns.of.backend' to IPv4 address via DNS
2024-04-24 02:19:53,677 [117586] <dns:7>: resolved 'dns.of.backend' to IPv4 address 10.252.3.19
2024-04-24 02:19:53,678 [117586] <dns:5>: stashed IP address '10.252.3.19' for name 'dns.of.backend' in the netaddr IP cache
2024-04-24 02:19:53,678 [117586] <dns:5>: stashed IP address '10.252.3.19' for name '10.252.3.19' in the netaddr IP cache
2024-04-24 02:19:53,678 [117586] <dns:7>: attempting to resolve 'dns.of.backend' to IPv6 address via DNS
2024-04-24 02:19:53,679 [117586] <dns:1>: IPv6 getaddrinfo 'dns.of.backend' error: No address associated with hostname
2024-04-24 02:19:53,680 [117586] <timer:7>: added timer ID 1025 ('ProxyTimeoutConnect', for module 'proxy'), triggering in 6 seconds
2024-04-24 02:19:53,680 [117586] <binding:4>: bound address 190.145.11.62, port 33303 to socket fd 16
2024-04-24 02:19:58,070 [117575] <signal:5>: signals blocked
2024-04-24 02:19:58,271 [117575] <signal:5>: signals unblocked
2024-04-24 02:19:58,273 [117575] <signal:9>: handling SIGALRM (signal 14)
2024-04-24 02:19:58,273 [117575] <timer:4>: 5 seconds for timer ID 24075 ('Controls polling', for module 'ctrls') elapsed, invoking callback (0x56014d1f8290)
2024-04-24 02:19:58,274 [117575] <timer:6>: restarting timer ID 24075 ('Controls polling'), as per callback
2024-04-24 02:19:58,680 [117586] <signal:5>: signals blocked
2024-04-24 02:19:58,881 [117586] <signal:5>: signals unblocked
2024-04-24 02:19:58,882 [117586] <signal:9>: handling SIGALRM (signal 14)
2024-04-24 02:19:58,882 [117586] <timer:7>: removed timer ID 1025 ('ProxyTimeoutConnect', for module 'proxy')
2024-04-24 02:19:58,882 [117586] <inet:3>: getpeername(2) error on fd 16: Transport endpoint is not connected

So it's clearly binding on the wrong interface.

[Castaglia]

@jb-boin Thanks for the info! Based on the IP addresses, we can indeed see that the wrong interface is being used. The "Transport endpoint is not connected" is a somewhat arcane error code/message to use for this situation, but it is not wrong.

I've just refined my PR in order to address some regressions discovered by the regression tests, mostly related to handling cases where frontend address is IPv6 and backend address is IPv4 (or vice versa); I think the PR is ready to be tried in your situation.

[Castaglia]

@jb-boin Any chance you've been able to try out the patch in your environment?