VPN no traffic is getting thru

[kabaga]

Important notices

Before you add a new report, we ask you kindly to acknowledge the following:

• [ x ] I have read the contributing guide lines at https://github.com/opnsense/core/blob/master/CONTRIBUTING.md
• [ x ] I am convinced that my issue is new after having checked both open and closed issues at https://github.com/opnsense/core/issues?q=is%3Aissue

Describe the bug

It started with this issue https://forum.opnsense.org/index.php?topic=29878.0 that I posted in the forums. I got no reply. I upgraded to the current version 22.7.2-amd64 and now my site-to-site WireGuard would not go up at all. It will stay up if only I remove the peers. With peers, the interface will get destroyed.

[#] ifconfig wg create name wg1
[#] wg setconf wg1 /dev/stdin
[#] ifconfig wg1 inet 192.168.0.1/24 alias
[#] ifconfig wg1 mtu 1420
[#] ifconfig wg1 up
[#] route -q -n add -inet 192.168.0.2/32 -interface wg1
[#] route -q -n add -inet 10.1.0.0/16 -interface wg1
[#] ifconfig wg1 destroy

My backup VPN is IPSec, this one is up, and able to ping it from OPNsense, but I could not reach it from LAN. BGP is running between my site and remote. The "nestat -r" also shows that the remote LAN is reachable via ipsec1 interface, but I could not reach it. The firewall Live Logs from the main site show that it is getting thru via the ipsec1 interface. It is the same with the remote site.

Tip: to validate your setup was working with the previous version, use opnsense-revert (https://docs.opnsense.org/manual/opnsense_tools.html#opnsense-revert)

To Reproduce

Steps to reproduce the behavior: N/A

Expected behavior

The expected behavior is the site-to-site WireGuard will stay up and whe it goes down, the route-based IPSec will be the up and should be able to let traffic through between the sites.

Screenshots N/A

Relevant log files

N/A

Additional context

N/A

Environment

Software version used and hardware type if relevant, e.g.:

Main site: Supermicro board C2758 OPNsense 22.7.2-amd64 FreeBSD 13.1-RELEASE-p1 OpenSSL 1.1.1q 5 Jul 2022

Remote: Protectli FW4B The final version of the 22.1

[mimugmail]

This usually happens when you have overlapping routes. I'm wondering why you set allowed ip's in endpoint when you want to use BGP?

[kabaga]

This usually happens when you have overlapping routes. I'm wondering why you set allowed ip's in endpoint when you want to use BGP?

Can you elaborate more regarding the allowed IP in endpoints? I always have allowed IP and my main site's wg interface is 192.168.0.1/24 and my endpoints are 192.168.0.2/32, 192.168.0.3/32, etc. They are not overlapping IPs. Also, the LAN side on the remote sites do not overlap with the main site either - for instance, my main site is 10.0.0.0/16 and the remote sites are 10.1.0.0/16, 10.2.0.0/16, etc.

Also, about the IPSec situation, I have been down for the past several days now because my IPsec tunnel. I checked the firewall s' Live Logs, I can see the ping is allowed and the "out" interface is the ipsec1 on the main site. I see the Live Logs on the remote site that is getting allowed. Both sites' BGP is working so they know where to send the routes which is this case is ipsec1 interface. I am hitting the wall on where to look.

In addition, I noticed there are two IPsec interfaces when creating firewall rules. One is called "IPsec" and the other which I enabled in Interfaces / Assignments. When I used the interface from Interfaces/Assignments, the rules will hit the default Deny. When I used the "IPsec", it worked. What is the purpose of the IPsec interface and the one from Interfaces / Assignments?

[mimugmail]

Every assigned interface gets an own submenu in rules, no problem. Can you post a screenshot of main site of ipsec p1 and p2 and wireguard instance?

[kabaga]

I want to add. The wireguard interface went up, but not passing any traffic and the BGP peering is not established. The remote site somehow sends traffic to the main site via the wireguard interface even though the routing table is showing the main site is reachable via the ipsec1 interface. Why does OPNsense using the wrong interface to send traffic?

The BGP peering has been established for the IPsec. Here are the screenshots of my IPsec setting.

[mimugmail]

Screenshot of linked endpoint of wg1

[kabaga]

I changed to 0.0.0.0/0 since this seems to be an accepted entry, but if I entered an IP block or /32 the wg interface gets destroyed. I tried to leave it blank, but I could not save the config if it is blank.

[mimugmail]

It should be 192.168.0.2/32 only

[kabaga]

It should be 192.168.0.2/32 only

That kept the wg1 interface stay up and the wg1 is showing up on the BGP table together with ipsec1, but the LAN at the remote sites is unreachable. When I ping a server at the remote site from the main where I am at, I get From 10.0.11.1 icmp_seq=6 Destination Host Unreachable. I really don't want to NAT between my sites since I don't use overlapping IP blocks.

[mimugmail]

Then you need to check Firewall rules and packets at wg1, bgp routing table etc. This is something to troubleshoot in the Forums with the community

[kabaga]

Then you need to check Firewall rules and packets at wg1, bgp routing table etc. This is something to troubleshoot in the Forums with the community

Is it the AllowedIPs' purpose to allow the IPs and advertise them? If I removed the 10.1.0.0/16 from AllowedIPs, would wireguard drop that traffic despite firewall rules?

Another odd part is, that when the wireguard interface goes down at the remote site, the ipsec1 goes down with it.

[OPNsense-bot]

This issue has been automatically timed-out (after 180 days of inactivity).

For more information about the policies for this repository, please read https://github.com/opnsense/plugins/blob/master/CONTRIBUTING.md for further details.

If someone wants to step up and work on this issue, just let us know, so we can reopen the issue and assign an owner to it.