frr
frr copied to clipboard
FRRouting
OSPF6 Crashes if ospf6d-gr.json and ospf6d-at-seq-no.dat exist
FRR VERSION: 8.3 OPERATING SYSTEM VERSION: *****UTM3 KERNEL VERSION: 4.19.204-*****UTM3
jbfw9182:/usr/local/frrouting/etc# gdb --core=dump --args ospf6d -t --log stdout
gdb: Symbol `acs_map' has different size in shared object, consider re-linking
Could not find platform independent libraries <prefix>
Could not find platform dependent libraries <exec_prefix>
Consider setting $PYTHONHOME to <prefix>[:<exec_prefix>]
'import site' failed; use -v for traceback
Python Exception <type 'exceptions.ImportError'> No module named gdb:
gdb: warning:
Could not load the Python gdb module from `/usr/local/share/gdb/python'.
Limited Python support is available from the _gdb module.
Suggest passing --data-directory=/path/to/gdb/data-directory.
GNU gdb (GDB) 7.12
Copyright (C) 2016 Free Software Foundation, Inc.
License GPLv3+: GNU GPL version 3 or later <http://gnu.org/licenses/gpl.html>
This is free software: you are free to change and redistribute it.
There is NO WARRANTY, to the extent permitted by law. Type "show copying"
and "show warranty" for details.
This GDB was configured as "x86_64-pc-linux-gnu".
Type "show configuration" for configuration details.
For bug reporting instructions, please see:
<http://www.gnu.org/software/gdb/bugs/>.
Find the GDB manual and other documentation resources online at:
<http://www.gnu.org/software/gdb/documentation/>.
For help, type "help".
Type "apropos word" to search for commands related to "word"...
Reading symbols from ospf6d...(no debugging symbols found)...done.
/usr/local/frrouting/etc/dump: No such file or directory.
(gdb) quit
jbfw9182:/usr/local/frrouting/etc# gdb --args ospf6d -t --log stdout
gdb: Symbol `acs_map' has different size in shared object, consider re-linking
Could not find platform independent libraries <prefix>
Could not find platform dependent libraries <exec_prefix>
Consider setting $PYTHONHOME to <prefix>[:<exec_prefix>]
'import site' failed; use -v for traceback
Python Exception <type 'exceptions.ImportError'> No module named gdb:
gdb: warning:
Could not load the Python gdb module from `/usr/local/share/gdb/python'.
Limited Python support is available from the _gdb module.
Suggest passing --data-directory=/path/to/gdb/data-directory.
GNU gdb (GDB) 7.12
Copyright (C) 2016 Free Software Foundation, Inc.
License GPLv3+: GNU GPL version 3 or later <http://gnu.org/licenses/gpl.html>
This is free software: you are free to change and redistribute it.
There is NO WARRANTY, to the extent permitted by law. Type "show copying"
and "show warranty" for details.
This GDB was configured as "x86_64-pc-linux-gnu".
Type "show configuration" for configuration details.
For bug reporting instructions, please see:
<http://www.gnu.org/software/gdb/bugs/>.
Find the GDB manual and other documentation resources online at:
<http://www.gnu.org/software/gdb/documentation/>.
For help, type "help".
Type "apropos word" to search for commands related to "word"...
Reading symbols from ospf6d...(no debugging symbols found)...done.
(gdb) run
Starting program: /usr/local/sbin/ospf6d -t --log stdout
Python Exception <type 'exceptions.NameError'> Installation error: gdb.execute_unwinders function is missing:
Python Exception <type 'exceptions.NameError'> Installation error: gdb.execute_unwinders function is missing:
Python Exception <type 'exceptions.NameError'> Installation error: gdb.execute_unwinders function is missing:
warning: Unable to find libthread_db matching inferior's thread library, thread debugging will not be available.
Python Exception <type 'exceptions.NameError'> Installation error: gdb.execute_unwinders function is missing:
Python Exception <type 'exceptions.NameError'> Installation error: gdb.execute_unwinders function is missing:
Python Exception <type 'exceptions.NameError'> Installation error: gdb.execute_unwinders function is missing:
Python Exception <type 'exceptions.NameError'> Installation error: gdb.execute_unwinders function is missing:
Python Exception <type 'exceptions.NameError'> Installation error: gdb.execute_unwinders function is missing:
Python Exception <type 'exceptions.NameError'> Installation error: gdb.execute_unwinders function is missing:
Python Exception <type 'exceptions.NameError'> Installation error: gdb.execute_unwinders function is missing:
2022/08/17 18:47:39 OSPF6: [T83RR-8SM5G] ospf6d 8.3 starting: vty@2606
jbfw9182.ddns.additionnetworks.net# conf
jbfw9182.ddns.additionnetworks.net# configure
jbfw9182.ddns.additionnetworks.net(config)# roo
jbfw9182.ddns.additionnetworks.net(config)# router ospf
jbfw9182.ddns.additionnetworks.net(config)# router ospf6
OSPF6: Received signal 11 at 1660776529 (si_addr 0x7f, PC 0x7ffff7586f5f); aborting...
OSPF6: zlog_signal+0xdf 7ffff7b85dcf 7fffffffaab0 /usr/local/frrouting/lib/libfrr.so.0 (mapped at 0x7ffff7ae4000)
OSPF6: PBKDF2_SHA256+0x453 7ffff7baff03 7fffffffabf0 /usr/local/frrouting/lib/libfrr.so.0 (mapped at 0x7ffff7ae4000)
OSPF6: killpg+0x40 7ffff758e530 7fffffffad40 /lib64/libc.so.6 (mapped at 0x7ffff755b000)
OSPF6: ---- signal ----
OSPF6: duplocale+0x3f 7ffff7586f5f 7fffffffb190 /lib64/libc.so.6 (mapped at 0x7ffff755b000)
OSPF6: json_tokener_parse_ex+0xbb 7ffff6a88bd4 7fffffffb1c0 /lib64/libjson-c.so.5 (mapped at 0x7ffff6a80000)
OSPF6: json_object_from_fd_ex+0x159 7ffff6a8d9d0 7fffffffb300 /lib64/libjson-c.so.5 (mapped at 0x7ffff6a80000)
OSPF6: json_object_from_fd+0x1a 7ffff6a8d875 7fffffffc350 /lib64/libjson-c.so.5 (mapped at 0x7ffff6a80000)
OSPF6: json_object_from_file+0x65 7ffff6a8da85 7fffffffc370 /lib64/libjson-c.so.5 (mapped at 0x7ffff6a80000)
OSPF6: ospf6_auth_seqno_nvm_read+0x27 448b37 7fffffffc3a0 /usr/local/sbin/ospf6d (mapped at 0x400000)
OSPF6: ospf6_instance_create+0x330 4410e0 7fffffffc3f0 /usr/local/sbin/ospf6d (mapped at 0x400000)
OSPF6: ospf6_instance_create+0x3f8 4411a8 7fffffffc430 /usr/local/sbin/ospf6d (mapped at 0x400000)
OSPF6: cmd_exit+0x20f 7ffff7b5af6f 7fffffffc470 /usr/local/frrouting/lib/libfrr.so.0 (mapped at 0x7ffff7ae4000)
OSPF6: cmd_execute_command+0xe1 7ffff7b5b0e1 7fffffffc4d0 /usr/local/frrouting/lib/libfrr.so.0 (mapped at 0x7ffff7ae4000)
OSPF6: cmd_execute+0xb9 7ffff7b5b299 7fffffffc520 /usr/local/frrouting/lib/libfrr.so.0 (mapped at 0x7ffff7ae4000)
OSPF6: vty_set_include+0x387 7ffff7bc6297 7fffffffc580 /usr/local/frrouting/lib/libfrr.so.0 (mapped at 0x7ffff7ae4000)
OSPF6: vty_set_include+0x616 7ffff7bc6526 7fffffffe5e0 /usr/local/frrouting/lib/libfrr.so.0 (mapped at 0x7ffff7ae4000)
OSPF6: vty_close+0x9bc 7ffff7bc82dc 7fffffffe610 /usr/local/frrouting/lib/libfrr.so.0 (mapped at 0x7ffff7ae4000)
OSPF6: thread_call+0x89 7ffff7bc0fe9 7fffffffe8b0 /usr/local/frrouting/lib/libfrr.so.0 (mapped at 0x7ffff7ae4000)
OSPF6: frr_run+0xc8 7ffff7b7e168 7fffffffe940 /usr/local/frrouting/lib/libfrr.so.0 (mapped at 0x7ffff7ae4000)
OSPF6: main+0xc3 40e923 7fffffffeac0 /usr/local/sbin/ospf6d (mapped at 0x400000)
OSPF6: __libc_start_main+0xfd 7ffff7579abd 7fffffffeae0 /lib64/libc.so.6 (mapped at 0x7ffff755b000)
OSPF6: _start+0x29 40f0bd 7fffffffeba0 /usr/local/sbin/ospf6d (mapped at 0x400000)
OSPF6: in thread vty_read scheduled from lib/vty.c:2720 vty_event()
core_handler: showing active allocations in memory group libfrr
core_handler: memstats: Buffer : 5 * 24
core_handler: memstats: Host config : 6 * (variably sized)
core_handler: memstats: Command Tokens : 3971 * 72
core_handler: memstats: Command Token Text : 2823 * (variably sized)
core_handler: memstats: Command Token Help : 2823 * (variably sized)
core_handler: memstats: Command Argument : 2 * (variably sized)
core_handler: memstats: Command Argument Name : 674 * (variably sized)
core_handler: memstats: Graph : 22 * 8
core_handler: memstats: Graph Node : 4629 * 32
core_handler: memstats: Hash : 260 * (variably sized)
core_handler: memstats: Hash Bucket : 730 * 32
core_handler: memstats: Hash Index : 130 * (variably sized)
core_handler: memstats: Interface : 30 * 272
core_handler: memstats: Connected : 17 * 48
core_handler: memstats: Link List : 73 * 40
core_handler: memstats: Link Node : 175 * 24
core_handler: memstats: Temporary memory : 3 * (variably sized)
core_handler: memstats: Northbound Node : 135 * 1192
core_handler: memstats: Northbound Configuration : 2 * 16
core_handler: memstats: Prefix : 17 * 56
core_handler: memstats: Privilege information : 3 * (variably sized)
core_handler: memstats: Stream : 2 * 16416
core_handler: memstats: Route table : 7 * 56
core_handler: memstats: Thread : 8 * 160
core_handler: memstats: Thread master : 4 * (variably sized)
core_handler: memstats: Thread Poll Info : 2 * 8192
core_handler: memstats: Thread stats : 9 * 96
core_handler: memstats: Typed-hash bucket : 3 * (variably sized)
core_handler: memstats: Vector : 9304 * 24
core_handler: memstats: Vector index : 9304 * (variably sized)
core_handler: memstats: VRF : 1 * 216
core_handler: memstats: VRF bit-map : 3 * 8
core_handler: memstats: VTY : 4 * (variably sized)
core_handler: memstats: VTY server : 3 * 32
core_handler: memstats: VTY history : 1 * 11
core_handler: memstats: YANG module : 6 * 48
core_handler: memstats: Zclient : 1 * 3144
core_handler: memstats: Redistribution instance IDs : 3 * 2
core_handler: memstats: log thread-local buffer : 1 * 24608
core_handler: showing active allocations in memory group logging subsystem
core_handler: memstats: log file target : 1 * 88
core_handler: showing active allocations in memory group ospf6d
core_handler: memstats: OSPF6 LSA database : 2 * 88
core_handler: memstats: OSPF6 route table : 3 * 56
core_handler: memstats: OSPF6 top : 2 * (variably sized)
ospf6d crashed in startup, signal 11
[Inferior 1 (process 22795) exited with code 01]
- It's the 8.3 release so it should be: d66a1ca8da269a4de0084db45c5a9fc3352ae16c
The issue is ospf6d will run after a reboot but if you kill or stop ospf6d either with frrinit.sh or manually. It will crash on restart. unless you delete these files in /var/run ospf6d-gr.json, ospf6d-at-seq-no.dat and ospf6d.pid. I'm not sure if the issue is just it's not removing those files on shutdown or what the issue is. Files don't really seem to have much in them.
ospf6d-gr.json:
/var/run# cat ospf6d-gr.json
{
"instances":{
}
}
ospf6d-at-seq-no.dat:
/var/run# cat ospf6d-at-seq-no.dat
{
"instances":{
"default":{
"sequence_number":0
}
}
}
The pid just has the process ID of the old process in it, maybe it's not getting shutdown correctly if the pid is till there or it's crashing on shutdown? not sure, did check and it's not running.
- [X] Did you check if this is a duplicate issue?
- [X] Did you test it on the latest FRRouting/frr master branch?
To Reproduce
- start frr using: frrinit.sh start
- services come up
- Check to see if ospf6 is running in vtysh do: show ipv6 ospf6 (should show ospf6 status)
- stop frr frrinit.sh stop
- restart frr frrinit.sh start
- Check to see if ospf6 is running in vtysh do: show ipv6 ospf6 This time it will say ospf6d is not running
And these results seem independent of any configuration. I've tested it with and without our production config. In the example above when I running the ospf6d bin in gdb I didn't specify a config the process started but as soon as I typed router ospf6 to create the instance it crashed.
Expected behavior ospf6d should be able to be stopped and restarted like the other processes. (frrinit.sh)
Versions OPERATING SYSTEM VERSION: ******UTM3 KERNEL VERSION: 4.19.204-*****UTM3 FRR-8.3-RELEASE
Hi @JGCR1,
I'm not sure if the issue is just it's not removing those files on shutdown or what the issue is.
Those two files (ospf6d-gr.json and ospf6d-at-seq-no.dat) are intended to be preserved across restarts. ospf6d uses them as non-volatile storage, so removing them manually shouldn't be necessary (nor recommended).
Files don't really seem to have much in them.
ospf6d-gr.json:
/var/run# cat ospf6d-gr.json { "instances":{ } }ospf6d-at-seq-no.dat:
/var/run# cat ospf6d-at-seq-no.dat { "instances":{ "default":{ "sequence_number":0 } } }
Those files look ok.
Given the crash is happening inside the json_object_from_file function, before FRR has a chance to do anything with the file, it's possible your libjson installation might be broken. In normal circumstances, json_object_from_file would return -1 in the case of an error, instead of crashing.
A few questions:
- Which libjson version are you using?
- Did you cross-compile the library?
- Isn't the same problem occurring with ospfd? That daemon uses a similar file called
ospfd-gr.json.
Could be something to do with locale settings: below is the code where its failing.
#ifdef HAVE_USELOCALE { locale_t duploc = duplocale(oldlocale); ====> this is failing. so something to do with setting the locale? newloc = newlocale(LC_NUMERIC_MASK, "C", duploc); if (newloc == NULL) { freelocale(duploc); return NULL; } uselocale(newloc); }
OSPF6: ---- signal ---- OSPF6: duplocale+0x3f 7ffff7586f5f 7fffffffb190 /lib64/libc.so.6 (mapped at 0x7ffff755b000) OSPF6: json_tokener_parse_ex+0xbb 7ffff6a88bd4 7fffffffb1c0 /lib64/libjson-c.so.5 (mapped at 0x7ffff6a80000) OSPF6: json_object_from_fd_ex+0x159 7ffff6a8d9d0 7fffffffb300 /lib64/libjson-c.so.5 (mapped at 0x7ffff6a80000) OSPF6: json_object_from_fd+0x1a 7ffff6a8d875 7fffffffc350 /lib64/libjson-c.so.5 (mapped at 0x7ffff6a80000) OSPF6: json_object_from_file+0x65 7ffff6a8da85 7fffffffc370 /lib64/libjson-c.so.5 (mapped at 0x7ffff6a80000) OSPF6: ospf6_auth_seqno_nvm_read+0x27
This issue is stale because it has been open 180 days with no activity. Comment or remove the autoclose label in order to avoid having this issue closed.
![github-actions[bot] avatar](https://avatars.githubusercontent.com/in/15368?v=4 "Published by a pub.dev verified publisher"){kind=small}
This issue will be automatically closed in the specified period unless there is further activity.
![frrbot[bot] avatar](https://avatars.githubusercontent.com/in/136408?v=4 "Published by a pub.dev verified publisher"){kind=small}