dpinger monitoring wrong interface in multi-wan with duplicate gateway ips

[Teruell]

trafficstars

Important notices

Before you add a new report, we ask you kindly to acknowledge the following:

• [Y] I have read the contributing guide lines at https://github.com/opnsense/core/blob/master/CONTRIBUTING.md
• [Y] I am convinced that my issue is new after having checked both open and closed issues at https://github.com/opnsense/core/issues?q=is%3Aissue

Describe the bug

Multi-WAN set up with two system gateways which have the same gateway IP (same ISP). Manually configured Monitor IPs for each gateway as follows:

Name: WAN4_Gateway (active), Interface: WAN4_igb4 Priority: 100 (upstream), Gateway: 100.64.0.1, Monitor IP: 1.0.0.1 Name: WAN5_Gateway, Interface: WAN5_igb5 Priority: 101 (upstream), Gateway: 100.64.0.1, Monitor IP: 1.1.1.1

dpinger logs (ips sanitized):

send_interval 1000ms loss_interval 2000ms time_period 60000ms report_interval 0ms data_len 0 alert_interval 1000ms latency_alarm 500ms loss_alarm 20% dest_addr 1.0.0.1 bind_addr 100.68.<>.<> identifier "WAN4_Gateway " send_interval 1000ms loss_interval 2000ms time_period 60000ms report_interval 0ms data_len 0 alert_interval 1000ms latency_alarm 500ms loss_alarm 20% dest_addr 1.1.1.1 bind_addr 100.120.<>.<> identifier "WAN5_Gateway "

I am getting monitor values for each gateway, however, if I look at the routes:

route get default

route to: default destination: default gateway: 100.64.0.1 interface: igb4

route get 1.0.0.1

route to: one.one.one.one destination: one.one.one.one gateway: 100.64.0.1 interface: igb4

route get 1.1.1.1

route to: one.one.one.one destination: default gateway: 100.64.0.1 interface: igb4

It's clear that this is actually only monitoring via igb4.

Expected behavior

I would expect a route for 1.1.1.1 be added for gateway 100.64.0.1 via interface igb5

Describe alternatives you considered

I have manually added the route and all seems to work as expected, but something seems to reset the route table periodically so this only works for a few seconds.

Also maybe interesting, dpinger logs also show me:

WAN5_Gateway 1.1.1.1: sendto error: 65

Though I do seem to be getting values back in the interface, and trying dpinger manually seems to also work, so unsure why I'm getting the 'no route to host' error as above in the logs about every 30m or so.

Environment

OPNsense 22.1.7_1-amd64

[fichtner]

This does not work in FreeBSD since forever.

This also isn’t a dpinger issue, the route table will never use the second interface. You need two distinct gateway addresses.

Cheers, Franco

[Teruell]

I'm obviously not in control of the gateway IPs, am I just out of luck then? I doubt that I'm the first to encounter this (or.. am I?), do you know if there are any plans to resolve or if there are any mitigations I could do?

[fichtner]

It's been the same forever in FreeBSD, may people noted it over the years but it's not a huge target. With FreeBSD 13.0 there is "multipath" routing which can add multiple routes for the same destination (and even default routes) but we had to turn it off because it messes with Multi-WAN reliability as currently implemented.

Yes, out of luck with FreeBSD here.

Cheers, Franco

[OPNsense-bot]

This issue has been automatically timed-out (after 180 days of inactivity).

For more information about the policies for this repository, please read https://github.com/opnsense/core/blob/master/CONTRIBUTING.md for further details.

If someone wants to step up and work on this issue, just let us know, so we can reopen the issue and assign an owner to it.