frr
frr copied to clipboard
FRRouting
VPNv4 Routes failing to import into VRF in 8.0
Hi folks - I have a use case where I want to use FRR as a sort of IPv4 to VPNv4 route reflector. That is - I'd like to take in VPNv4 routes and then convert them into IPv4 routes to send to a Peer in a VRF (a typical PE model). I can't seem to get the VPNv4 routes to import into the VRF - they all show as inaccessible which makes sense since they likely can't resolve their next hop as Im not running LDP or anything to be able to resolve the underlying LSP.
I know in JunOS you can tell the router to resolve next hops from the inet.0 table rather than inet.3 to help with this kind of scenario, Is a similar configuration possible in FRR? I've tried disable connected checks etc but the routes stay stuck as inaccessible and never import into the VRF.
Thanks!
Can you provide some output showing what you're referring to? It's not clear which address-family shows the next-hop is inaccessible, or what may be causing it.
Would also probably be useful to see the output of show ip nht and show ip nht vrf <vrf name>.
What version of FRR are you running?
@taspelund Many apologies for the delay, I got hit with a flu or something and was out of commission for a long time. In either case, I think I might have stumbled upon a bug here. It looks like this should just work but the problem seems to be that newly added VPNv4 routes are not being properly advertised to a VRF based IPv4 peer. I built a quick lab to demonstrate what I'm seeing...
Lab topology
VM1 (10.10.10.0) <------> (10.10.10.1) VM2 (10.10.10.2) <------> VM3(10.10.10.3)
The base premise of the lab is that VM2 is running FRR and acting as a PE. VM1 is acting an upstream VPNv4 peer and sending VM2 VPN routes. VM3 is a normal IPv4 peer is that is connected to VM2 inside of a VRF and has a normal IPv4 BGP peer configuration. Both VM1 and VM3 are running GOBGP. The configuration files for the three VMs are below....
VM1
global:
config:
as: 65000
router-id: 10.10.10.0
neighbors:
- config:
neighbor-address: 10.10.10.1
peer-as: 65000
timers:
config:
connect-retry: 1
hold-time: 3
keepalive-interval: 1
afi-safis:
- config:
afi-safi-name: l3vpn-ipv4-unicast
VM2
frr version 8.0
frr defaults traditional
hostname vm2
log syslog informational
no ip forwarding
no ipv6 forwarding
bgp no-rib
service integrated-vtysh-config
!
router bgp 65000
bgp router-id 10.10.10.1
no bgp ebgp-requires-policy
bgp disable-ebgp-connected-route-check
bgp bestpath compare-routerid
no bgp network import-check
neighbor 10.10.10.0 remote-as 65000
neighbor 10.10.10.0 update-source 10.10.10.1
!
address-family ipv4 vpn
neighbor 10.10.10.0 activate
neighbor 10.10.10.0 attribute-unchanged next-hop
exit-address-family
!
router bgp 65000 vrf vrf_1
no bgp ebgp-requires-policy
bgp disable-ebgp-connected-route-check
bgp bestpath compare-routerid
no bgp network import-check
neighbor 10.10.10.3 remote-as 65123
!
address-family ipv4 unicast
redistribute connected
rd vpn export 65000:1
rt vpn both 65000:1
export vpn
import vpn
exit-address-family
!
line vty
!
end
VM3
global:
config:
as: 65123
router-id: 10.10.10.3
neighbors:
- config:
neighbor-address: 10.10.10.2
peer-as: 65000
ebgp-multihop:
config:
enabled: true
multihop-ttl: 255
timers:
config:
connect-retry: 1
hold-time: 3
keepalive-interval: 1
If I start everything up - I immediately see that the route for the directly connected VRF interface is being sent to VM1 as a VPNv4 route...
root@vm1:~# ./gobgp global rib -a ipv4-l3vpn
Network Labels Next Hop AS_PATH Age Attrs
*> 65000:1:10.10.10.2/31 [3] 10.10.10.1 01:09:48 [{Origin: ?} {Med: 0} {LocalPref: 100} {Extcomms: [65000:1]}]
root@vm1:~#
So that's working. And if I advertise an additional prefix from VM3 that also shows up so it appears that the IPv4 to VPNv4 prefix translation is working as expected...
root@vm3:~# ./gobgp global rib add 172.16.16.0/24
root@vm3:~#
root@vm1:~# ./gobgp global rib -a ipv4-l3vpn
Network Labels Next Hop AS_PATH Age Attrs
*> 65000:1:10.10.10.2/31 [3] 10.10.10.1 01:10:44 [{Origin: ?} {Med: 0} {LocalPref: 100} {Extcomms: [65000:1]}]
*> 65000:1:172.16.16.0/24 [3] 10.10.10.3 65123 00:00:11 [{Origin: ?} {LocalPref: 100} {Extcomms: [65000:1]}]
root@vm1:~#
However - If I generate a VPNv4 prefix - its is not being translated to the IPv4 peer correctly despite having a matching RT....
root@vm1:~# ./gobgp global rib add -a vpnv4 10.0.0.0/24 label 10 rd 65000:1 rt 65000:1
root@vm1:~#
The route shows up on the FRR instance - but it is marked as not valid....
vm2# show ip bgp vrf vrf_1 ipv4
BGP table version is 6, local router ID is 10.10.10.2, vrf id 5
Default local pref 100, local AS 65000
Status codes: s suppressed, d damped, h history, * valid, > best, = multipath,
i internal, r RIB-failure, S Stale, R Removed
Nexthop codes: @NNN nexthop's vrf id, < announce-nh-self
Origin codes: i - IGP, e - EGP, ? - incomplete
RPKI validation codes: V valid, I invalid, N Not found
Network Next Hop Metric LocPrf Weight Path
10.0.0.0/24 10.10.10.0@0< 100 0 ?
*> 10.10.10.2/31 0.0.0.0 0 32768 ?
*> 172.16.16.0/24 10.10.10.3 0 65123 ?
Displayed 3 routes and 3 total paths
vm2#
The actual prefix output...
vm2# show ip bgp vrf vrf_1 ipv4 10.0.0.0/24
BGP routing table entry for 10.0.0.0/24, version 0
Paths: (1 available, no best path)
Not advertised to any peer
Imported from 65000:1:10.0.0.0/24
Local
10.10.10.0 (inaccessible) from 0.0.0.0 (10.10.10.2) vrf default(0) announce-nh-self
Origin incomplete, localpref 100, invalid, sourced, local
Extended Community: RT:65000:1
Remote label: 10
Last update: Fri Aug 20 20:50:34 2021
vm2#
Now what's curious here is that if I reapply the RT configuration on the FRR node - the route becomes valid...
vm2# config t
vm2(config)# router bgp 65000 vrf vrf_1
vm2(config-router)# address-family ipv4 unicast
vm2(config-router-af)# rt vpn both 65000:1
vm2(config-router-af)#
vm2(config-router-af)# end
vm2# show ip bgp vrf vrf_1 ipv4
BGP table version is 7, local router ID is 10.10.10.2, vrf id 5
Default local pref 100, local AS 65000
Status codes: s suppressed, d damped, h history, * valid, > best, = multipath,
i internal, r RIB-failure, S Stale, R Removed
Nexthop codes: @NNN nexthop's vrf id, < announce-nh-self
Origin codes: i - IGP, e - EGP, ? - incomplete
RPKI validation codes: V valid, I invalid, N Not found
Network Next Hop Metric LocPrf Weight Path
*> 10.0.0.0/24 10.10.10.0@0< 100 0 ?
*> 10.10.10.2/31 0.0.0.0 0 32768 ?
*> 172.16.16.0/24 10.10.10.3 0 65123 ?
Displayed 3 routes and 3 total paths
vm2#
And is then received on VM3...
root@vm3:~# ./gobgp global rib
Network Next Hop AS_PATH Age Attrs
*> 10.0.0.0/24 10.10.10.2 65000 00:00:34 [{Origin: ?} {Extcomms: [65000:1]}]
*> 10.10.10.2/31 10.10.10.2 65000 01:11:40 [{Origin: ?} {Med: 0}]
*> 172.16.16.0/24 0.0.0.0 00:03:01 [{Origin: ?}]
root@vm3:~#
So it's sort of like the translation from VPNv4 routes is only happening when you apply the RT configuration? Or am I doing something wrong here?
I've confirmed that this is not an issue in the 7.5 code base. Possibly a regression in the 8.0 code train?
Hi @jonlangemak , it looks like that imported route target is badly generated. that would explain why route from gobgp is not imported. there should be a command giving you ert/irt for the vrf
@pguibert6WIND Thanks for the reply! Can you elaborate a little though? You believe that the RT coming from gobgp is malformed? Or that FRR is doing something to it on import that is malforming it?
@pguibert6WIND Thanks for the reply! Can you elaborate a little though? You believe that the RT coming from gobgp is malformed? Or that FRR is doing something to it on import that is malforming it?
frr. Just to be sure, when you modify on the fly l3vpn bgp config , there may be bad rt generation. I would request you to do two things:
- populate router-id at second bgp instance
- write config, then restart frr (because loading config from startup is safer on your case)
- do you observe same pb ?
Hello folks,
I'm facing almost the same problem, when the frr starts all VPNv4 routes aren't installed as "inaccessible". Config is saved and the issue is reproducible every time. But when I enter the same RT in the runtime - everything is fine. For instance, I restarted frr and then:
ubuntuguest# show bgp vrf testvrf 5013:100::/64
BGP routing table entry for 5013:100::/64, version 0
Paths: (2 available, no best path)
Not advertised to any peer
Imported from 121:1:5013:100::/64
65012 65013
2003:100::2 (inaccessible) from :: (3.3.3.3) vrf default(0) announce-nh-self
Origin incomplete, invalid, sourced, local
Extended Community: RT:121:1
Remote label: 51
Last update: Mon Oct 18 12:28:32 2021
Imported from 121:1:5013:100::/64
65013
5003:100::2 (inaccessible) from :: (3.3.3.3) vrf default(0) announce-nh-self
Origin incomplete, invalid, sourced, local
Extended Community: RT:121:1
Remote label: 47
Last update: Mon Oct 18 12:28:32 2021
ubuntuguest# conf t
ubuntuguest(config)# router bgp 65011 vrf testvrf
ubuntuguest(config-router)# address-family ipv6 unicast
ubuntuguest(config-router-af)# rt vpn both 121:1
ubuntuguest(config-router-af)# end
ubuntuguest# show bgp vrf testvrf 5013:100::/64
BGP routing table entry for 5013:100::/64, version 0
Paths: (2 available, no best path)
Not advertised to any peer
Imported from 121:1:5013:100::/64
65012 65013
2003:100::2 from :: (3.3.3.3) vrf default(0) announce-nh-self
Origin incomplete, valid, sourced, local
Extended Community: RT:121:1
Remote label: 51
Last update: Mon Oct 18 12:28:37 2021
Imported from 121:1:5013:100::/64
65013
5003:100::2 from :: (3.3.3.3) vrf default(0) announce-nh-self
Origin incomplete, valid, sourced, local
Extended Community: RT:121:1
Remote label: 47
Last update: Mon Oct 18 12:28:37 2021
ubuntuguest#
The configuration is pretty simple:
router bgp 65011 vrf testvrf
bgp router-id 3.3.3.3
!
address-family ipv6 unicast
redistribute kernel
redistribute connected
redistribute static
label vpn export 38
rd vpn export 121:1
rt vpn both 121:1
export vpn
import vpn
exit-address-family
Any thoughts?
I've noticed that here we have a couple of issue:
- The main thing why I get nexthops unaccessible - there is a check when leaking routes from vpn to vrf: bgp_isvalid_labeled_nexthop. But in my case I have a regular nexthops without any labels:
# show bgp nexthop
Current BGP nexthop cache:
2003:100::2 valid [IGP metric 0], #paths 9, peer 2003:100::2
if eth2
Last update: Wed Oct 20 11:27:13 2021
...
My main question: where it should be updated to become labeled?
- Another issue, is different logic inside vpn_leak_from_vrf_update (when I update vrf's RT for example). That's why I get all the routes with accessible nexthop in that case.
@zstas Anything I can do to help move this along from a testing perspective?
@jonlangemak
I hoped we will get some comments from FRR maintainers. There's almost no presence of life in frr-dev mailing list, so seems like it's the main place to ask question. I can prepare a patch, but I don't know what behaviour is expected. For my purposes, I prepared a simple patch with checking whether the nextop is BGP_NEXTHOP_LABELED_VALID or at least BGP_NEXTHOP_VALID.
diff --git a/bgpd/bgp_nht.c b/bgpd/bgp_nht.c
index 8f381e9ff..d6f665e95 100644
--- a/bgpd/bgp_nht.c
+++ b/bgpd/bgp_nht.c
@@ -67,7 +67,7 @@ static int bgp_isvalid_nexthop(struct bgp_nexthop_cache *bnc)
static int bgp_isvalid_labeled_nexthop(struct bgp_nexthop_cache *bnc)
{
return (bgp_zebra_num_connects() == 0
- || (bnc && CHECK_FLAG(bnc->flags, BGP_NEXTHOP_LABELED_VALID)
+ || (bnc && ((CHECK_FLAG(bnc->flags, BGP_NEXTHOP_LABELED_VALID)) || (CHECK_FLAG(bnc->flags, BGP_NEXTHOP_VALID)))
&& bnc->nexthop_num > 0));
}
every place we call isvalid_labeled_nexthop we have already checked that the Path info has labels. So I am a bit confused how we shouldn't be checking for a valid labels.
I understand that usually, we have labeled next-hop (igp+ldp or labeled unicast). But there's one case when we have only the regular next-hop - it's Inter-AS option B. In this case, we have an eBGP session with vpnv4/vpnv6, and the only label is the label of the VRF (or per-prefix in the VRF). So, it won't be a "labeled" next-hop in this case but rather a regular one. Please correct me if I'm wrong.
if we have a prefix with pi->extra->num_labels set then the prefix should have that label. In this case we should not set the pi->extra->num_labels for the case you just described. That would be the better fix imo
@zstas Forgot to mention if there's a patch fix available Im happy to test it locally to see if it resolves the issue. I'll be out on holiday here for a few days but when I get back I'd be happy to help with testing.
@jonlangemak Hi guys, sorry, was busy with holidays.
Could you please test it out with the following patch? [1]
1 - https://github.com/FRRouting/frr/pull/10319
@zstas I'd be happy to test this out. I was able to pull down your commit into a local branch on my test box but Im a little lost on how to do the FRR build. Which flags do I need to pass into configure etc? If you could provide an example of how to run the build I'll give it a test on my end.
Thanks!
My main question: where it should be updated to become labeled?
You should have LDP daemon or BGP LU configured with your EBGP peer. if not, you will not have a label.
* Another issue, is different logic inside vpn_leak_from_vrf_update (when I update vrf's RT for example). That's why I get all the routes with accessible nexthop in that case.
that looks like as if some callback did not refresh automatically BGP. By answering the first question, we may understand better what is wrong.
I tend do think that BGP nexthop tracking does not refresh because zebra nexthop tracking does not refresh When you get the problem, please type : `show IP route <your IP nexthop>' to check for any label available.
Just to comment - my use case is for FRR to act more as a VPNv4 to IPv4 route reflector. So I'd be doing something like asking FRR to advertise routes that it can't resolve through the MPLS table (or inet.3 in Juniper parlance). This is working in the config posted above - just wanted to reiterate the use case since we're talking about next hop resolution.
@zstas - Just to reiterate - Im happy to test if you can give me some info on doing the build. Thanks!
@jonlangemak you can give it a try with compiling my branch [1] with this instruction [2]. But maintainers said a clear "no" to this patch. let's wait until they decide how to proceed with this issue. if you post your use-case in the PR [3] - it might be helpful also.
[1] - https://github.com/zstas/frr [2] - http://docs.frrouting.org/projects/dev-guide/en/latest/building-frr-for-ubuntu2004.html [3] - https://github.com/FRRouting/frr/pull/10319
For what it is worth - I can confirm that 7.5.1 does not have the bug, and that those couple of changes in the small patch appear to work in git. Thank you to zstas for the work around and to everybody who commented on this thread and, of course, a huge thanks to those who work on this project.
This issue is stale because it has been open 180 days with no activity. Comment or remove the autoclose label in order to avoid having this issue closed.
![github-actions[bot] avatar](https://avatars.githubusercontent.com/in/15368?v=4 "Published by a pub.dev verified publisher"){kind=small}
This issue will be automatically closed in the specified period unless there is further activity.
![frrbot[bot] avatar](https://avatars.githubusercontent.com/in/136408?v=4 "Published by a pub.dev verified publisher"){kind=small}