frr icon indicating copy to clipboard operation
frr copied to clipboard

Applying route-maps end upd killing bgpd process

Open aalmenar opened this issue 2 years ago • 9 comments

Describe the bug

When applying a list of route-maps to FRR (over vtysh) end up killing bgpd process and being restarted.

The FRR version is 8.3 from deb.frrouting.org OS is Debian 11 with latest kernel updates and packages updated.

  • [X] Did you check if this is a duplicate issue?
  • [x] Did you test it on the latest FRRouting/frr master branch?

To Reproduce

Having multiple ASN trying to insert a route- map like this:

route-map cASNNUMBER-v4 permit 5
 description XXXX
 match community gshut
 on-match goto 40
 set local-preference 0
 set community 65535:0
exit
!
route-map cASNNUMBER-v4 permit 10
 description XXX
 match community blackhole
 match ip address prefix-list BARRA32
 set community 65500:100 65535:666 65500:6001 65500:6009
 set large-community 65500:2010:ASNNUMBER 65500:666:9 65500:100:1
 on-match goto 40
exit
!
route-map cASNNUMBER-v4 permit 15
 description XXX
 match community rfc7999
 match ip address prefix-list BARRA32
 set community 65500:100 65535:666 65500:6001 65500:6009
 set large-community 65500:2010:ASNNUMBER 65500:666:9 65500:100:1
 on-match goto 40
exit
!
route-map cASNNUMBER-v4 permit 40
 description XXX
 match community rfc7999
 match ip address prefix-list cASNNUMBER-v4-bh
 set community 65500:100 65500:206 65500:667 65500:8020 65500:8030 65500:8032 65500:9001 65535:666 65500:6001 65500:6009
 set large-community 65500:2010:ASNNUMBER 65500:666:9 65500:100:1
 set ip next-hop 192.0.2.1
 set local-preference 100
exit
!
route-map cASNNUMBER-v4 permit 50
 description XXX
 call rpki
 match ip address prefix-list cASNNUMBER-v4
 set community 65500:100 65500:201 65500:8020 65500:8030 65500:8032 65500:9001 65500:2010
 set large-community 65500:2010:ASNNUMBER 65500:100:1
 set local-preference 100
exit
!
route-map cASNNUMBER-v4 deny 500
exit
!
route-map cASNNUMBER-v6 permit 5
 description XXX
 match community gshut
 on-match goto 40
 set local-preference 0
 set community 65535:0
exit
!
route-map cASNNUMBER-v6 permit 10
 description XXX
 match community blackhole
 match ip address prefix-list BARRA128
 set community 65500:100 65535:666 65500:6001 65500:6009
 set large-community 65500:2010:ASNNUMBER 65500:666:9 65500:100:1
 on-match goto 40
exit
!
route-map cASNNUMBER-v6 permit 15
 description XXX
 match community rfc7999
 match ip address prefix-list BARRA128
 set community 65500:100 65535:666 65500:6001 65500:6009
 set large-community 65500:2010:ASNNUMBER 65500:666:9 65500:100:1
 on-match goto 40
exit
!
route-map cASNNUMBER-v6 permit 40
 description XXX
 match community rfc7999
 match ipv6 address prefix-list cASNNUMBER-v6-bh
 set community 65500:100 65500:206 65500:667 65500:8020 65500:8030 65500:8032 65500:9001 65535:666 65500:6001 65500:6009
 set large-community 65500:2010:ASNNUMBER 65500:666:9 65500:100:1
 set ipv6 next-hop global 0100::
 set local-preference 100
exit
!
route-map cASNNUMBER-v6 permit 50
 description XXX
 call rpki
 match ipv6 address prefix-list cASNNUMBER-v6
 set community 65500:100 65500:201 65500:8020 65500:8030 65500:8032 65500:9001 65500:2010
 set large-community 65500:2010:ASNNUMBER 65500:100:1
 set local-preference 100
exit
!
route-map cASNNUMBER-v6 deny 500
exit
!
route-map cASNNUMBER-out permit 10
 description XXX
 match community blackhole
 set community 65535:666 65500:6009
 set ip next-hop 192.0.2.1
 set ipv6 next-hop global 0100::
 on-match goto 100
exit
!
route-map cASNNUMBER-out permit 15
 match large-community OnlyPeering
exit
!
route-map cASNNUMBER-out permit 20
 match community OnlyPeering
exit
!
route-map cASNNUMBER-out deny 100
exit
!
end

Expected behavior To accept the route-map and be able to save it without crashing

Versions

  • OS Version: Debian 11.4
  • Kernel: 5.10.0-17-amd64
  • FRR Version: 8.3 (Package from deb.frrouting.org)

Additional context The crash in syslog show this:

Aug 31 20:10:26 rs2 BGP[76751]: Received signal 11 at 1661976626 (si_addr 0xcf08efda, PC 0x7f2351a922ee); aborting...
Aug 31 20:10:26 rs2 BGP[76751]: /usr/lib/x86_64-linux-gnu/frr/libfrr.so.0(zlog_backtrace_sigsafe+0x6d) [0x7f2351a59b4d]
Aug 31 20:10:26 rs2 BGP[76751]: /usr/lib/x86_64-linux-gnu/frr/libfrr.so.0(zlog_signal+0xf5) [0x7f2351a59d45]
Aug 31 20:10:26 rs2 BGP[76751]: /usr/lib/x86_64-linux-gnu/frr/libfrr.so.0(+0xcd691) [0x7f2351a86691]
Aug 31 20:10:26 rs2 BGP[76751]: /lib/x86_64-linux-gnu/libpthread.so.0(+0x13140) [0x7f235184f140]
Aug 31 20:10:26 rs2 BGP[76751]: /usr/lib/x86_64-linux-gnu/frr/libfrr.so.0(route_node_match+0x5e) [0x7f2351a922ee]
Aug 31 20:10:26 rs2 BGP[76751]: /usr/lib/x86_64-linux-gnu/frr/libfrr.so.0(route_map_apply_ext+0x5b0) [0x7f2351a7b1a0]
Aug 31 20:10:26 rs2 BGP[76751]: /usr/lib/frr/bgpd(+0x1440ae) [0x55ec1af320ae]
Aug 31 20:10:26 rs2 BGP[76751]: /usr/lib/frr/bgpd(bgp_update+0x843) [0x55ec1af3ef53]
Aug 31 20:10:26 rs2 BGP[76751]: /usr/lib/frr/bgpd(bgp_nlri_parse_ip+0xb7) [0x55ec1af41387]
Aug 31 20:10:26 rs2 BGP[76751]: /usr/lib/frr/bgpd(+0x1382e8) [0x55ec1af262e8]
Aug 31 20:10:26 rs2 BGP[76751]: /usr/lib/frr/bgpd(bgp_process_packet+0x515) [0x55ec1af29015]
Aug 31 20:10:26 rs2 BGP[76751]: /usr/lib/x86_64-linux-gnu/frr/libfrr.so.0(thread_call+0x7d) [0x7f2351a9842d]
Aug 31 20:10:26 rs2 BGP[76751]: /usr/lib/x86_64-linux-gnu/frr/libfrr.so.0(frr_run+0xe8) [0x7f2351a52088]
Aug 31 20:10:26 rs2 BGP[76751]: /usr/lib/frr/bgpd(main+0x356) [0x55ec1aecf336]
Aug 31 20:10:26 rs2 BGP[76751]: /lib/x86_64-linux-gnu/libc.so.6(__libc_start_main+0xea) [0x7f235168ad0a]
Aug 31 20:10:26 rs2 BGP[76751]: /usr/lib/frr/bgpd(_start+0x2a) [0x55ec1aed109a]
Aug 31 20:10:26 rs2 BGP[76751]: in thread bgp_process_packet scheduled from bgpd/bgp_io.c:269 bgp_process_reads()

aalmenar avatar Sep 01 '22 08:09 aalmenar

Could you narrow it down a little bit and check which one match causes this crash?

ton31337 avatar Sep 01 '22 08:09 ton31337

The change is applied:

  • Tried copy and pasting.
  • Tried vtysh < "file with changes"

When applying this happens:

rs2(config-route-map)#  set large-community 65500:2010:197518 65500:666:9 65500:100:1
rs2(config-route-map)#  on-match goto 40
rs2(config-route-map)# exit
rs2(config)# !
rs2(config)# route-map cASNUMBER-v6 permit 15
rs2(config-route-map)#  description XXXXXXXXXX
Warning: connecting to bgpd...failed!
Warning: closing connection to bgpd because of an I/O error!
rs2(config-route-map)#  match community rfc7999
Warning: connecting to bgpd...failed!
bgpd is not running
rs2(config-route-map)#  match ip address prefix-list BARRA128
Warning: connecting to bgpd...failed!

aalmenar avatar Sep 01 '22 09:09 aalmenar

Can you check if it's crashing or not if removing match or on-match statements?

ton31337 avatar Sep 01 '22 09:09 ton31337

In the script theres 3 route-maps being added.

If i split them into 3 different files (1 route-map per file, looks like it doesn't crash)

aalmenar avatar Sep 01 '22 10:09 aalmenar

Crashing only for cASNNUMBER-v6 or cASNNUMBER-v4 as well? I'm trying locally, but can't hit this crash.

Also:

call rpki

Can you show this route-map as well?

ton31337 avatar Sep 01 '22 11:09 ton31337

Only crashes on -v6, after placing each route-map with conf t "routemap-file" end wr the daemon doesnt crash.

Also as request rpki route-map:

route-map rpki permit 10
 description Dont check rpki on blackholes
 match community rfc7999
 set community 65500:1003 additive
 set large-community 65500:1000:3 additive
exit
!
route-map rpki permit 20
 description Match notfound rpki 
 match rpki notfound
 set community 65500:1002 additive
 set large-community 65500:1000:2 additive
exit
!
route-map rpki permit 30
 description Match valid rpki
 match rpki valid
 set community 65500:1001 additive
 set large-community 65500:1000:1 additive
exit
!
route-map rpki deny 40
 description Match invalid rpki and discard
 match rpki invalid
exit
!

aalmenar avatar Sep 01 '22 12:09 aalmenar

I managed to replicate this. stable/8.2 does not have this crash, but stable/8.3 and master have. Adding more details as a reference here:

(gdb) bt full
#0  0x00007fc197d3ffe1 in raise () from /lib/x86_64-linux-gnu/[libpthread.so](http://libpthread.so/).0
No symbol table info available.
#1  0x00007fc197f776cc in core_handler (signo=11, siginfo=0x7ffd6ab650f0, context=<optimized out>)
    at lib/sigevent.c:262
        pc = 0x7fc197f832ee <route_node_match+94>
        sa_default = {__sigaction_handler = {sa_handler = 0x0, sa_sigaction = 0x0}, sa_mask = {__val = {
              0 <repeats 16 times>}}, sa_flags = 0, sa_restorer = 0x0}
        sigset = {__val = {9216, 0 <repeats 15 times>}}
#2  <signal handler called>
No symbol table info available.
#3  0x00007fc197f832ee in route_node_match (table=table@entry=0x5612f410c0d0, pu=..., pu@entry=...) at lib/table.c:208
        p = 0x7ffd6ab65be0
        node = 0x8c8b61e7
        matched = 0x0
#4  0x00007fc197f6c1a0 in route_map_get_index_list (table=0x5612f410c0d0, prefix=0x7ffd6ab65be0,
    rn=<synthetic pointer>) at lib/routemap.c:1778
        tmp_rn = 0x0
        tmp_rn = <optimized out>
#5  route_map_get_index (match_ret=<synthetic pointer>, object=0x7ffd6ab65600, prefix=0x7ffd6ab65be0,
    map=0x5612f433e710) at lib/routemap.c:1835
        table = <optimized out>
        ret = <optimized out>
        candidate_rmap_list = <optimized out>
        rn = 0x0
        nn = <optimized out>
        family = <optimized out>
        ln = <optimized out>
        index = <optimized out>
        best_index = 0x0
        head_index = <optimized out>
        ret = <optimized out>
        candidate_rmap_list = <optimized out>
        rn = <optimized out>
        ln = <optimized out>
        nn = <optimized out>
        index = <optimized out>
        best_index = <optimized out>
        head_index = <optimized out>
        table = <optimized out>
        family = <optimized out>
        __func__ = "route_map_get_index"
        _xref = {xref = {xrefdata = 0x0, type = XREFT_ASSERT, line = 1843, file = 0x7fc197fcc964 "lib/routemap.c",
            func = 0x7fc197fcc120 <__func__.38> "route_map_get_index"},
          expr = 0x7fc197fccf60 "((candidate_rmap_list) ? ((candidate_rmap_list)->head) : ((void *)0))", extra = 0x0,
          args = 0x0}
        xref_p_14 = 0x7fc198030320 <_xref.42>
        _xref = {xref = {xrefdata = 0x0, type = XREFT_ASSERT, line = 1843, file = 0x7fc197fcc964 "lib/routemap.c",
            func = 0x7fc197fcc120 <__func__.38> "route_map_get_index"},
          expr = 0x7fc197fccf08 "(((candidate_rmap_list) ? ((candidate_rmap_list)->head) : ((void *)0)))->data != NULL", extra = 0x0, args = 0x0}
        xref_p_15 = 0x7fc1980302e0 <_xref.41>
        _xref = {xref = {xrefdata = 0x0, type = XREFT_ASSERT, line = 1851, file = 0x7fc197fcc964 "lib/routemap.c",
            func = 0x7fc197fcc120 <__func__.38> "route_map_get_index"}, expr = 0x7fc197fb19c0 "ln", extra = 0x0,
          args = 0x0}
        xref_p_16 = 0x7fc1980302a0 <_xref.40>
        _xref = {xref = {xrefdata = 0x0, type = XREFT_ASSERT, line = 1851, file = 0x7fc197fcc964 "lib/routemap.c",
            func = 0x7fc197fcc120 <__func__.38> "route_map_get_index"}, expr = 0x7fc197fb19ad "(ln)->data != NULL",
          extra = 0x0, args = 0x0}
--Type <RET> for more, q to quit, c to continue without paging--
        xref_p_17 = 0x7fc198030260 <_xref.39>
#6  route_map_apply_ext (map=0x5612f433e710, prefix=prefix@entry=0x7ffd6ab65be0,
    match_object=match_object@entry=0x7ffd6ab65600, set_object=set_object@entry=0x7ffd6ab65600) at lib/routemap.c:2570
        recursion = 0
        match_ret = <optimized out>
        ret = RMAP_PERMITMATCH
        index = 0x0
        set = 0x0
        skip_match_clause = false
        __func__ = {<optimized out> <repeats 20 times>}
#7  0x00005612f16420ae in bgp_input_modifier (peer=peer@entry=0x7fc18f1ef010, p=p@entry=0x7ffd6ab65be0,
    attr=attr@entry=0x7ffd6ab65900, afi=afi@entry=AFI_IP6, safi=safi@entry=SAFI_UNICAST,
    rmap_name=rmap_name@entry=0x0, label=0x0, num_labels=0, dest=0x5612f8519ae0) at bgpd/bgp_route.c:1612
        filter = <optimized out>
        rmap_path = {next = 0x0, prev = 0x0, nh_thread = {le_next = 0x0, le_prev = 0x0}, net = 0x5612f8519ae0,
          nexthop = 0x0, peer = 0x7fc18f1ef010, attr = 0x7ffd6ab65900, extra = 0x7ffd6ab65670, mpath = 0x0,
          uptime = 0, lock = 0, flags = 0, type = 0 '\000', sub_type = 0 '\000', instance = 0, addpath_rx_id = 0,
          tx_addpath = {addpath_tx_id = {0, 0}}}
        extra = {damp_info = 0x0, aggr_suppressors = 0x0, igpmetric = 0, label = {0, 0}, num_labels = 0,
          af_flags = 0, sid = {{sid = {__in6_u = {__u6_addr8 = '\000' <repeats 15 times>, __u6_addr16 = {0, 0, 0, 0,
                    0, 0, 0, 0}, __u6_addr32 = {0, 0, 0, 0}}}, loc_block_len = 0 '\000', loc_node_len = 0 '\000',
              func_len = 0 '\000', arg_len = 0 '\000', transposition_len = 0 '\000',
              transposition_offset = 0 '\000'}, {sid = {__in6_u = {__u6_addr8 = '\000' <repeats 15 times>,
                  __u6_addr16 = {0, 0, 0, 0, 0, 0, 0, 0}, __u6_addr32 = {0, 0, 0, 0}}}, loc_block_len = 0 '\000',
              loc_node_len = 0 '\000', func_len = 0 '\000', arg_len = 0 '\000', transposition_len = 0 '\000',
              transposition_offset = 0 '\000'}, {sid = {__in6_u = {__u6_addr8 = '\000' <repeats 15 times>,
                  __u6_addr16 = {0, 0, 0, 0, 0, 0, 0, 0}, __u6_addr32 = {0, 0, 0, 0}}}, loc_block_len = 0 '\000',
              loc_node_len = 0 '\000', func_len = 0 '\000', arg_len = 0 '\000', transposition_len = 0 '\000',
              transposition_offset = 0 '\000'}, {sid = {__in6_u = {__u6_addr8 = '\000' <repeats 15 times>,
                  __u6_addr16 = {0, 0, 0, 0, 0, 0, 0, 0}, __u6_addr32 = {0, 0, 0, 0}}}, loc_block_len = 0 '\000',
              loc_node_len = 0 '\000', func_len = 0 '\000', arg_len = 0 '\000', transposition_len = 0 '\000',
              transposition_offset = 0 '\000'}, {sid = {__in6_u = {__u6_addr8 = '\000' <repeats 15 times>,
                  __u6_addr16 = {0, 0, 0, 0, 0, 0, 0, 0}, __u6_addr32 = {0, 0, 0, 0}}}, loc_block_len = 0 '\000',
              loc_node_len = 0 '\000', func_len = 0 '\000', arg_len = 0 '\000', transposition_len = 0 '\000',
              transposition_offset = 0 '\000'}, {sid = {__in6_u = {__u6_addr8 = '\000' <repeats 15 times>,
                  __u6_addr16 = {0, 0, 0, 0, 0, 0, 0, 0}, __u6_addr32 = {0, 0, 0, 0}}}, loc_block_len = 0 '\000',
              loc_node_len = 0 '\000', func_len = 0 '\000', arg_len = 0 '\000', transposition_len = 0 '\000',
              transposition_offset = 0 '\000'}}, num_sids = 0, vnc = {export = {rfapi_handle = 0x0,
              local_nexthops = 0x0}, import = {timer = 0x0, hme = 0x0, rd = {family = 0 '\000', prefixlen = 0,
                val = "\000\000\000\000\000\000\000"}, un_family = 0 '\000', un = {addr4 = {s_addr = 0}, addr6 = {
                  __in6_u = {__u6_addr8 = '\000' <repeats 15 times>, __u6_addr16 = {0, 0, 0, 0, 0, 0, 0, 0},
                    __u6_addr32 = {0, 0, 0, 0}}}}, create_time = 0, aux_prefix = {family = 0 '\000', prefixlen = 0,
                u = {prefix = 0 '\000', prefix4 = {s_addr = 0}, prefix6 = {__in6_u = {
                      __u6_addr8 = '\000' <repeats 15 times>, __u6_addr16 = {0, 0, 0, 0, 0, 0, 0, 0}, __u6_addr32 = {
                        0, 0, 0, 0}}}, lp = {id = {s_addr = 0}, adv_router = {s_addr = 0}}, prefix_eth = {
                    octet = "\000\000\000\000\000"}, val = '\000' <repeats 15 times>, val32 = {0, 0, 0, 0}, ptr = 0,
                  prefix_evpn = {route_type = 0 '\000', u = {_ead_addr = {esi = {
                          val = "\000\000\000\000\000\000\000\000\000"}, eth_tag = 0, ip = {ipa_type = IPADDR_NONE,
                          ip = {addr = 0 '\000', _v4_addr = {s_addr = 0}, _v6_addr = {__in6_u = {
                                __u6_addr8 = '\000' <repeats 15 times>, __u6_addr16 = {0, 0, 0, 0, 0, 0, 0, 0},
                                __u6_addr32 = {0, 0, 0, 0}}}}}, frag_id = 0}, _macip_addr = {eth_tag = 0,
                        ip_prefix_length = 0 '\000', mac = {octet = "\000\000\000\000\000"}, ip = {
                          ipa_type = IPADDR_NONE, ip = {addr = 0 '\000', _v4_addr = {s_addr = 0}, _v6_addr = {
                              __in6_u = {__u6_addr8 = '\000' <repeats 15 times>, __u6_addr16 = {0, 0, 0, 0, 0, 0, 0,
                                  0}, __u6_addr32 = {0, 0, 0, 0}}}}}}, _imet_addr = {eth_tag = 0,
                        ip_prefix_length = 0 '\000', ip = {ipa_type = IPADDR_NONE, ip = {addr = 0 '\000', _v4_addr = {
                              s_addr = 0}, _v6_addr = {__in6_u = {__u6_addr8 = '\000' <repeats 15 times>,
                                __u6_addr16 = {0, 0, 0, 0, 0, 0, 0, 0}, __u6_addr32 = {0, 0, 0, 0}}}}}}, _es_addr = {
--Type <RET> for more, q to quit, c to continue without paging--
                        esi = {val = "\000\000\000\000\000\000\000\000\000"}, ip_prefix_length = 0 '\000', ip = {
                          ipa_type = IPADDR_NONE, ip = {addr = 0 '\000', _v4_addr = {s_addr = 0}, _v6_addr = {
                              __in6_u = {__u6_addr8 = '\000' <repeats 15 times>, __u6_addr16 = {0, 0, 0, 0, 0, 0, 0,
                                  0}, __u6_addr32 = {0, 0, 0, 0}}}}}}, _prefix_addr = {eth_tag = 0,
                        ip_prefix_length = 0 '\000', ip = {ipa_type = IPADDR_NONE, ip = {addr = 0 '\000', _v4_addr = {
                              s_addr = 0}, _v6_addr = {__in6_u = {__u6_addr8 = '\000' <repeats 15 times>,
                                __u6_addr16 = {0, 0, 0, 0, 0, 0, 0, 0}, __u6_addr32 = {0, 0, 0, 0}}}}}}}},
                  prefix_flowspec = {family = 0 '\000', prefixlen = 0, ptr = 0}}}}}, parent = 0x0, bgp_orig = 0x0,
          nexthop_orig = {family = 0 '\000', prefixlen = 0, u = {prefix = 0 '\000', prefix4 = {s_addr = 0},
              prefix6 = {__in6_u = {__u6_addr8 = '\000' <repeats 15 times>, __u6_addr16 = {0, 0, 0, 0, 0, 0, 0, 0},
                  __u6_addr32 = {0, 0, 0, 0}}}, lp = {id = {s_addr = 0}, adv_router = {s_addr = 0}}, prefix_eth = {
                octet = "\000\000\000\000\000"}, val = '\000' <repeats 15 times>, val32 = {0, 0, 0, 0}, ptr = 0,
              prefix_evpn = {route_type = 0 '\000', u = {_ead_addr = {esi = {
                      val = "\000\000\000\000\000\000\000\000\000"}, eth_tag = 0, ip = {ipa_type = IPADDR_NONE, ip = {
                        addr = 0 '\000', _v4_addr = {s_addr = 0}, _v6_addr = {__in6_u = {
                            __u6_addr8 = '\000' <repeats 15 times>, __u6_addr16 = {0, 0, 0, 0, 0, 0, 0, 0},
                            __u6_addr32 = {0, 0, 0, 0}}}}}, frag_id = 0}, _macip_addr = {eth_tag = 0,
                    ip_prefix_length = 0 '\000', mac = {octet = "\000\000\000\000\000"}, ip = {
                      ipa_type = IPADDR_NONE, ip = {addr = 0 '\000', _v4_addr = {s_addr = 0}, _v6_addr = {__in6_u = {
                            __u6_addr8 = '\000' <repeats 15 times>, __u6_addr16 = {0, 0, 0, 0, 0, 0, 0, 0},
                            __u6_addr32 = {0, 0, 0, 0}}}}}}, _imet_addr = {eth_tag = 0, ip_prefix_length = 0 '\000',
                    ip = {ipa_type = IPADDR_NONE, ip = {addr = 0 '\000', _v4_addr = {s_addr = 0}, _v6_addr = {
                          __in6_u = {__u6_addr8 = '\000' <repeats 15 times>, __u6_addr16 = {0, 0, 0, 0, 0, 0, 0, 0},
                            __u6_addr32 = {0, 0, 0, 0}}}}}}, _es_addr = {esi = {
                      val = "\000\000\000\000\000\000\000\000\000"}, ip_prefix_length = 0 '\000', ip = {
                      ipa_type = IPADDR_NONE, ip = {addr = 0 '\000', _v4_addr = {s_addr = 0}, _v6_addr = {__in6_u = {
                            __u6_addr8 = '\000' <repeats 15 times>, __u6_addr16 = {0, 0, 0, 0, 0, 0, 0, 0},
                            __u6_addr32 = {0, 0, 0, 0}}}}}}, _prefix_addr = {eth_tag = 0,
                    ip_prefix_length = 0 '\000', ip = {ipa_type = IPADDR_NONE, ip = {addr = 0 '\000', _v4_addr = {
                          s_addr = 0}, _v6_addr = {__in6_u = {__u6_addr8 = '\000' <repeats 15 times>, __u6_addr16 = {
                              0, 0, 0, 0, 0, 0, 0, 0}, __u6_addr32 = {0, 0, 0, 0}}}}}}}}, prefix_flowspec = {
                family = 0 '\000', prefixlen = 0, ptr = 0}}}, bgp_fs_pbr = 0x0, bgp_fs_iprule = 0x0, mh_info = 0x0}
        ret = <optimized out>
        rmap = <optimized out>
#8  0x00005612f164ef53 in bgp_update (peer=peer@entry=0x7fc18f1ef010, p=p@entry=0x7ffd6ab65be0,
    addpath_id=addpath_id@entry=0, attr=0x7ffd6ab65cf0, afi=afi@entry=AFI_IP6, safi=<optimized out>,
    safi@entry=SAFI_UNICAST, type=9, sub_type=0, prd=0x0, label=0x0, num_labels=0, soft_reconfig=0, evpn=0x0)
    at bgpd/bgp_route.c:3895
        ret = <optimized out>
        aspath_loop_count = <optimized out>
        dest = 0x5612f8519ae0
        bgp = 0x5612f366cfa0
        new_attr = {aspath = 0x5612f8519570, community = 0x5612f3d9b270, refcnt = 0, flag = 2147491971, nexthop = {
            s_addr = 0}, med = 0, local_pref = 0, nh_ifindex = 0, origin = 0 '\000',
          pmsi_tnl_type = PMSI_TNLTYPE_NO_INFO, rmap_change_flags = 0, mp_nexthop_global = {__in6_u = {
              __u6_addr8 = " \001\a\370\000~\000\000\000\003\003\216\000\000\000\001", __u6_addr16 = {288, 63495,
                32256, 0, 768, 36355, 0, 256}, __u6_addr32 = {4161208608, 32256, 2382562048, 16777216}}},
          mp_nexthop_local = {__in6_u = {__u6_addr8 = "\376\200\000\000\000\000\000\000\002]s\377\376\240\177\200",
              __u6_addr16 = {33022, 0, 0, 0, 23810, 65395, 41214, 32895}, __u6_addr32 = {33022, 0, 4285750530,
                2155847934}}}, nh_lla_ifindex = 3, ecommunity = 0x0, ipv6_ecommunity = 0x0,
          lcommunity = 0x5612f434eb90, cluster1 = 0x0, transit = 0x0, mp_nexthop_global_in = {s_addr = 0},
          aggregator_addr = {s_addr = 0}, originator_id = {s_addr = 0}, weight = 0, aggregator_as = 0,
          mp_nexthop_len = 32 ' ', mp_nexthop_prefer_global = 0 '\000', sticky = 0 '\000', default_gw = 0 '\000',
          router_flag = 0 '\000', es_flags = 0 '\000', tag = 0, label_index = 4294967295, label = 4294836223,
          srv6_vpn = 0x0, srv6_l3vpn = 0x0, encap_tunneltype = 0, encap_subtlvs = 0x0, vnc_subtlvs = 0x0,
          evpn_overlay = {type = OVERLAY_INDEX_TYPE_NONE, eth_s_id = {val = "\000\000\000\000\000\000\000\000\000"},
            gw_ip = {ipa_type = IPADDR_NONE, ip = {addr = 0 '\000', _v4_addr = {s_addr = 0}, _v6_addr = {__in6_u = {
                    __u6_addr8 = '\000' <repeats 15 times>, __u6_addr16 = {0, 0, 0, 0, 0, 0, 0, 0}, __u6_addr32 = {0,
--Type <RET> for more, q to quit, c to continue without paging--
                      0, 0, 0}}}}}}, mm_seqnum = 0, mm_sync_seqnum = 0, rmac = {octet = "\000\000\000\000\000"},
          distance = 0 '\000', rmap_table_id = 0, link_bw = 0, esi = {val = "\000\000\000\000\000\000\000\000\000"},
          srte_color = 0, df_pref = 0, df_alg = 0 '\000', nh_type = 0, bh_type = BLACKHOLE_UNSPEC}
        attr_new = 0x5612fb79b5e0
        pi = <optimized out>
        new = <optimized out>
        extra = <optimized out>
        reason = <optimized out>
        pfx_buf = "\030\000\000\000\000\000\000\000\200\315\063J\023V\000\000\f\000\000\000\000\000\000\000\250\351\364\227\301\177\000\000\340;\343{\023V\000\000\376\377\377\377\000\000\000\000\002\000\000\000\000\000\000\000\000\346\276QL\032JB\360Z\266j\375\177\000\000\b[\266j\375\177\000\000\360Z\266j\375\177\000\000\220\353\064\364\022V\000\000\030\235\000\220\301\177\000\000 \000\000\000\000\000\000\000\340\000\000\000\000\000\000\000wXb\361\022V\000\000 \000\000\000\000\000\000\000\340\000\000\000\000\000\000\000 [\266j\375\177\000\000\234Zb\361\022V\000\000\340", '\000' <repeats 15 times>, "\360\\\266j\375\177\000\000\020\360\036\217"
        connected = 0
        do_loop_check = <optimized out>
        has_valid_label = <optimized out>
        nh_afi = <optimized out>
        pi_type = 0 '\000'
        pi_sub_type = 0 '\000'
        force_evpn_import = false
        orig_safi = SAFI_UNICAST
        vnc_implicit_withdraw = 0
        same_attr = 0
        __func__ = "bgp_update"
        pfxprint = {<optimized out> <repeats 80 times>}
        label_decoded = <optimized out>
(gdb) frame 3
#3  0x00007fc197f832ee in route_node_match (table=table@entry=0x5612f410c0d0, pu=..., pu@entry=...) at lib/table.c:208
208	lib/table.c: No such file or directory.
(gdb) p *node
Cannot access memory at address 0x8c8b61e7

ton31337 avatar Sep 02 '22 10:09 ton31337

Something is really wrong:

(gdb) p *peer->filter[2][1]->map
$9 = {name = 0x55ebd3e71e20 "c197518-v6", map = 0x55ebd399c560}
(gdb) p *peer->filter[2][1]->map->map
$10 = {name = 0x55ebd43ad050 "\340\236\260 \002", head = 0x55ebd43ad6e0, tail = 0x55ebd43ad280, 
  next = 0x55ebd450a9e0, prev = 0x55ebd450be00, to_be_processed = 128, deleted = 73, 
  optimization_disabled = 80, applied = 94471367737185, applied_clear = 94471366301072, 
  use_count = 56416, ipv4_prefix_table = 0x55ebd450edb0, ipv6_prefix_table = 0x55ebd450cce0, 
  qobj_node = {nid = 94471367758176, nodehash = {hi = {next = 0x55ebd450cbb0, hashval = 3562081872}}, 
    type = 0x55ebd4510410}}
(gdb) 

ton31337 avatar Sep 06 '22 10:09 ton31337

Got one more different crash...:

(gdb) bt
#0  raise (sig=sig@entry=11) at ../sysdeps/unix/sysv/linux/raise.c:50
#1  0x00007f96dd73e450 in core_handler (signo=11, siginfo=0x7ffdfe6aecf0, context=<optimized out>) at lib/sigevent.c:261
#2  <signal handler called>
#3  route_map_apply_ext (map=<optimized out>, prefix=prefix@entry=0x7ffdfe6af8e0, match_object=match_object@entry=0x7ffdfe6af2e0, 
    set_object=set_object@entry=0x7ffdfe6af2e0, pref=pref@entry=0x0) at lib/routemap.c:2603
#4  0x000055ee482f6d22 in bgp_input_modifier (peer=peer@entry=0x7f96db071010, p=p@entry=0x7ffdfe6af8e0, 
    attr=attr@entry=0x7ffdfe6af5f0, afi=afi@entry=AFI_IP6, safi=safi@entry=SAFI_UNICAST, rmap_name=rmap_name@entry=0x0, label=0x0, 
    num_labels=0, dest=0x55ee49d09730) at bgpd/bgp_route.c:1654
#5  0x000055ee48303db7 in bgp_update (peer=peer@entry=0x7f96db071010, p=p@entry=0x7ffdfe6af8e0, addpath_id=addpath_id@entry=0, 
    attr=0x7ffdfe6afa00, afi=afi@entry=AFI_IP6, safi=<optimized out>, safi@entry=SAFI_UNICAST, type=9, sub_type=0, prd=0x0, label=0x0, 
    num_labels=0, soft_reconfig=0, evpn=0x0) at bgpd/bgp_route.c:3989
#6  0x000055ee483062f7 in bgp_nlri_parse_ip (peer=0x7f96db071010, attr=attr@entry=0x7ffdfe6afa00, packet=0x7ffdfe6af9d0)
    at bgpd/bgp_route.c:5887
#7  0x000055ee482ea210 in bgp_nlri_parse (peer=peer@entry=0x7f96db071010, attr=attr@entry=0x7ffdfe6afa00, 
    packet=packet@entry=0x7ffdfe6af9d0, mp_withdraw=mp_withdraw@entry=0) at bgpd/bgp_packet.c:347
#8  0x000055ee482eae84 in bgp_update_receive (peer=peer@entry=0x7f96db071010, size=size@entry=114) at bgpd/bgp_packet.c:1921
#9  0x000055ee482edba3 in bgp_process_packet (thread=<optimized out>) at bgpd/bgp_packet.c:2822
#10 0x00007f96dd7506f6 in thread_call (thread=thread@entry=0x7ffdfe6afd30) at lib/thread.c:2008
#11 0x00007f96dd709b78 in frr_run (master=0x55ee49689f70) at lib/libfrr.c:1198
#12 0x000055ee482945f8 in main (argc=<optimized out>, argv=0x7ffdfe6b0098) at bgpd/bgp_main.c:518
(gdb) 

ton31337 avatar Sep 06 '22 11:09 ton31337

This issue is stale because it has been open 180 days with no activity. Comment or remove the autoclose label in order to avoid having this issue closed.

github-actions[bot] avatar Mar 06 '23 02:03 github-actions[bot]

This issue will be automatically closed in the specified period unless there is further activity.

frrbot[bot] avatar Mar 06 '23 02:03 frrbot[bot]

Hit similar crash on 8.4.2 after delete and recreate routemap.

BGP: Received signal 11 at 1677816637 (si_addr 0x50006, PC 0x7f6a4c05a1f6); aborting...
BGP: /opt/nova/lib/libfrr.so.0(zlog_backtrace_sigsafe+0x71) [0x7f6a4c021691]
BGP: /opt/nova/lib/libfrr.so.0(zlog_signal+0xf9) [0x7f6a4c021899]
BGP: /opt/nova/lib/libfrr.so.0(+0xd21c5) [0x7f6a4c04e1c5]
BGP: /lib/x86_64-linux-gnu/libpthread.so.0(+0x153c0) [0x7f6a4be053c0]
BGP: /opt/nova/lib/libfrr.so.0(route_node_match+0x66) [0x7f6a4c05a1f6]
BGP: /opt/nova/lib/libfrr.so.0(route_map_apply_ext+0x5fc) [0x7f6a4c04318c]
BGP: /opt/nova/sbin/bgpd(+0x148672) [0x5575aec4a672]
BGP: /opt/nova/sbin/bgpd(bgp_update+0x856) [0x5575aec57686]
BGP: /opt/nova/sbin/bgpd(bgp_nlri_parse_ip+0xb7) [0x5575aec59ba7]
BGP: /opt/nova/sbin/bgpd(+0x13c3a4) [0x5575aec3e3a4]
BGP: /opt/nova/sbin/bgpd(bgp_process_packet+0x522) [0x5575aec412a2]
BGP: /opt/nova/lib/libfrr.so.0(thread_call+0x81) [0x7f6a4c060431]
BGP: /opt/nova/lib/libfrr.so.0(frr_run+0xe8) [0x7f6a4c019af8]
BGP: /opt/nova/sbin/bgpd(main+0x37d) [0x5575aebe770d]
BGP: /lib/x86_64-linux-gnu/libc.so.6(__libc_start_main+0xf3) [0x7f6a4bc250b3]
BGP: /opt/nova/sbin/bgpd(_start+0x2e) [0x5575aebe9bde]
BGP: in thread bgp_process_packet scheduled from bgpd/bgp_io.c:269 bgp_process_reads()

This remembers me of #10708, and found a potential issue after a look:

In route_map_add(), lib/routemap.c:

	exist = hash_release(route_map_master_hash, map);
	if (exist) {
		map->to_be_processed = exist->to_be_processed;
		route_map_free_map(exist);
	}

The old route-map is freed here, but bgpd might still be holding dangling references.

leonshaw avatar Mar 06 '23 07:03 leonshaw

This issue will no longer be automatically closed.

frrbot[bot] avatar Mar 06 '23 07:03 frrbot[bot]

This issue is stale because it has been open 180 days with no activity. Comment or remove the autoclose label in order to avoid having this issue closed.

github-actions[bot] avatar Sep 03 '23 01:09 github-actions[bot]

This issue will be automatically closed in the specified period unless there is further activity.

frrbot[bot] avatar Sep 03 '23 01:09 frrbot[bot]