FRRouting
FRR 10.3.1 - IsBGPsafeYet and RPKI filtering
Description
I suspect there is a problem with RPKI filtering. It could be that a router starts learning BGP prefixes before the RPKI filtering / cache is ready. Maybe after an FRR upgrade / restart.
This service checks that an ISP is filtering RPKI invalid prefixes: https://isbgpsafeyet.com
On one of 4 routers, it did not filter the rpki prefixes. All routers on same FRR version.
To resolve this, I did a 'clear bgp -cloudflare-ip-address-'
After clearing two BGP sessions, the problem went away - so I think the filtering config is correct, but somehow invalid prefixes made it into the routing table.
Maybe there needs to be an extra check that RPKI is ready / connected - before BGP sessions start connecting - or a delay which allows the router to get the rpki cache server connections up.
Version
FRRouting 10.3.1 (cpt-ter-rs1) on Linux(6.1.44-atomic).
Copyright 1996-2005 Kunihiro Ishiguro, et al.
configured with:
'--build=x86_64-linux-gnu' '--prefix=/usr' '--includedir=${prefix}/include' '--mandir=${prefix}/share/man' '--infodir=${prefix}/share/info' '--sysconfdir=/etc' '--localstatedir=/var' '--disable-option-checking' '--disable-silent-rules' '--libdir=${prefix}/lib/x86_64-linux-gnu' '--libexecdir=${prefix}/lib/x86_64-linux-gnu' '--disable-maintainer-mode' '--sbindir=/usr/lib/frr' '--with-vtysh-pager=/usr/bin/pager' '--libdir=/usr/lib/x86_64-linux-gnu/frr' '--with-moduledir=/usr/lib/x86_64-linux-gnu/frr/modules' '--disable-dependency-tracking' '--enable-rpki' '--disable-scripting' '--enable-pim6d' '--disable-grpc' '--with-libpam' '--enable-doc' '--enable-doc-html' '--enable-snmp' '--enable-fpm' '--disable-protobuf' '--disable-zeromq' '--enable-ospfapi' '--enable-bgp-vnc' '--enable-multipath=256' '--enable-pcre2posix' '--enable-user=frr' '--enable-group=frr' '--enable-vty-group=frrvty' '--enable-configfile-mask=0640' '--enable-logfile-mask=0640' 'build_alias=x86_64-linux-gnu' 'PYTHON=python3'
How to reproduce
On a router that has Cloudflare peering.
Check if you are learning their invalid prefixes:
show bgp ipv4 103.21.244.8 show bgp ipv6 2606:4700:7000::6715:f408
Expected behavior
Do not learn RPKI invalid prefixes - filter them out.
Actual behavior
Cloudflare's v4 and v6 invalid prefixes do get into the router.
Additional context
RPKI filtering part of the config:
route-map recv-from-napjhb-peer-v4 deny 2 match rpki invalid exit
show rpki cache-connection
Connected to group 1 rpki tcp cache vc1-cpt.inx.net.za 3323 pref 2 rpki tcp cache rpki1.ct1.nap.africa 3323 pref 1 (connected)
ping vc1-cpt.inx.net.za
PING vc1-cpt.inx.net.za(vc1-cpt.inx.net.za (2001:43f8:1f4:200::76)) 56 data bytes 64 bytes from vc1-cpt.inx.net.za (2001:43f8:1f4:200::76): icmp_seq=1 ttl=63 time=0.859 ms
ping rpki1.ct1.nap.africa
PING rpki1.ct1.nap.africa (196.60.70.2) 56(84) bytes of data. 64 bytes from rs0.ixp.capetown (196.60.70.2): icmp_seq=1 ttl=64 time=0.196 ms
Checklist
- [x] I have searched the open issues for this bug.
- [x] I have not included sensitive information in this report.
What is the configuration? Once you filter the routes at incoming direction, you lost the track because the routes do not exist in the RIB anymore. Clearing a session of course fixes that, because you receive the route again. To overcome this situation, soft-reconfiguration might help.
Hi, don't think your comment matches what we noticed.
Our config will filter all prefixes that are rpki invalid.
The filter failed and we ingested invalid prefixes.
The 'bgp clear' reset the bgp session, then the filter worked. No invalid routes after bgp session reset.
The issue was only on 1 of 4 routers with same FRR versions, so I think there is some edge case which makes the filter rules fail.
The 'bgp clear' reset the bgp session, then the filter worked.
Doing a soft clear isn't enough to "fix" the state, or do you really need a hard reset?
Hi,
I'll have to confirm the next time we spot the problem, but I do remember that earlier in the year I noticed the same issue, and then a clear 'soft in' did not fix it. Only 'hard' clear worked (hard reset).
If soft clear won't work, try route refresh for inbound too before doing a hard reset.
Ideally we need to be able to upgrade FRR, and know that the filters work from the 1st moment it's busy starting up.
I've made a note to check this on next upgrade / restart.
Ideally we need to be able to upgrade FRR, and know that the filters work from the 1st moment it's busy starting up.
Of course, contributions are welcome.
@ton31337 we discussed this in the tech meeting and one possible solution that we see is to have the rpki state be a stop on whether or not a peer can be formed. IF rpki is configured then don't let peering start until rpki comes up and establishes state.
Isn't this too strict? What happens if the RPKI cache server is connected and immediately disconnected after BGP is established?
I'd consider it the same as when nht goes down, right? We shut the peer down. Maybe it needs to be some sort of strict rpki mode. Isn't it the whole point of rpki though? To provide some sort of ability to prevent peers from sending us bad data. Why would we operate without it up?
We have multiple rpki servers configured - if one fails, it should in theory recover fairly quickly.
IMO, RPKI is sort of optional (yes, BFD is also optional, but it's more related to the connectivity).
If RPKI fails badly, can't talk to any of our RPKI servers, and all the BGP sessions are stuck waiting... we'd have to log in via out of band network and figure out what went wrong - not the end of the world.
Maybe under the rpki config section there could be a setting like 'rpki cache require 2'
which would halt the BGP setup process until 2 cache servers are connected and ready
update: looks like 'show rpki cache-connection' only connects to one cache at a time
BGP RPKI is such an interesting work. If it's not working how can we trust anything? Do we really want to default to trust when we cannot actually figure anything out? I hear what Donatas is saying here though, which is why I softened to a lesser setting where you can as the operator specify if rpki is down don't peer.
Adding an optional knob to say it's "strict" might be decent medicine preventing to establish BGP session until RPKI session is UP. If that's what we want, I can implement that.
@SwimGeek could you test this branch https://github.com/FRRouting/frr/pull/19103?
Hi, thank you for adding this feature. Unfortunately we don't have an easy way to test this. The cloudflare BGP session is on a production router. We can test when the next version is out.