Crash in netopeer2-server during get-data rpc call

[meghna-doshi]

Hi, netopeer2-server 2.1.10 sysrepoctl - sysrepo YANG schema manipulation tool, compiled with libsysrepo v2.1.15 (SO v7.1.3)

Crash bt:

Core was generated by netopeer2-server -dv2 -t 60'.
Program terminated with signal 11, Segmentation fault.
#0  0x00007fc0aea2054f in sr_edit_apply_r (first_node=0x38, parent_node=0x0,
edit_node=0x7fc088030e70, parent_op=EDIT_REMOVE, diff_parent=0x0,
diff_root=0x0, oper_edit=1, flags=0, change=0x0)
at /workspace/siqbuild/PRG-BUILDS/onecell_integration_6.5/6.5.3.006/DUOTA/ran_3rdparty/open_source/netconf/sysrepo/src/edit_diff.c:2342
2342      /workspace/siqbuild/PRG-BUILDS/onecell_integration_6.5/6.5.3.006/DUOTA/ran_3rdparty/open_source/netconf/sysrepo/src/edit_diff.c: No such file or directory.
#0  0x00007fc0aea2054f in sr_edit_apply_r (first_node=0x38, parent_node=0x0,
edit_node=0x7fc088030e70, parent_op=EDIT_REMOVE, diff_parent=0x0,
diff_root=0x0, oper_edit=1, flags=0, change=0x0)
at /workspace/siqbuild/PRG-BUILDS/onecell_integration_6.5/6.5.3.006/DUOTA/ran_3rdparty/open_source/netconf/sysrepo/src/edit_diff.c:2342
#1  0x00007fc0aea20d9f in sr_edit_apply_r (first_node=0x7fc088001d28,
parent_node=0x7fc088001cf0, edit_node=0x7fc088031000,
parent_op=EDIT_MERGE, diff_parent=0x0, diff_root=0x0, oper_edit=1,
flags=0, change=0x0)
at /workspace/siqbuild/PRG-BUILDS/onecell_integration_6.5/6.5.3.006/DUOTA/ran_3rdparty/open_source/netconf/sysrepo/src/edit_diff.c:2505
#2  0x00007fc0aea20d9f in sr_edit_apply_r (first_node=0x7fc0aa0278e0,
parent_node=0x0, edit_node=0x7fc088010300, parent_op=EDIT_CONTINUE,
diff_parent=0x0, diff_root=0x0, oper_edit=1, flags=0, change=0x0)
at /workspace/siqbuild/PRG-BUILDS/onecell_integration_6.5/6.5.3.006/DUOTA/ran_3rdparty/open_source/netconf/sysrepo/src/edit_diff.c:2505
#3  0x00007fc0aea20f41 in sr_edit_mod_apply (edit=0x7fc088010300,
ly_mod=0x26e32c0, oper_edit=1, data=0x7fc0aa0278e0, diff=0x0, change=0x0)
at /workspace/siqbuild/PRG-BUILDS/onecell_integration_6.5/6.5.3.006/DUOTA/ran_3rdparty/open_source/netconf/sysrepo/src/edit_diff.c:2550
#4  0x00007fc0aea1552a in sr_module_oper_data_update (mod=0x7fc088003780,
orig_name=0x2775850 "netopeer2", orig_data=0x27fe720, conn=0x25ff2f0,
timeout_ms=60000, opts=0, data=0x7fc0aa0278e0)
at /workspace/siqbuild/PRG-BUILDS/onecell_integration_6.5/6.5.3.006/DUOTA/ran_3rdparty/open_source/netconf/sysrepo/src/modinfo.c:770
#5  0x00007fc0aea18889 in sr_modinfo_module_data_load (
mod_info=0x7fc0aa0278d0, mod=0x7fc088003780,
orig_name=0x2775850 "netopeer2", orig_data=0x27fe720, timeout_ms=60000,
opts=0)
at /workspace/siqbuild/PRG-BUILDS/onecell_integration_6.5/6.5.3.006/DUOTA/ran_3rdparty/open_source/netconf/sysrepo/src/modinfo.c:1819
#6  0x00007fc0aea18f05 in sr_modinfo_data_load (mod_info=0x7fc0aa0278d0,
cache=4, orig_name=0x2775850 "netopeer2", orig_data=0x27fe720,
timeout_ms=60000, opts=0)
at /workspace/siqbuild/PRG-BUILDS/onecell_integration_6.5/6.5.3.006/DUOTA/ran_3rdparty/open_source/netconf/sysrepo/src/modinfo.c:2000
#7  0x00007fc0aea191ff in sr_modinfo_consolidate (mod_info=0x7fc0aa0278d0,
mod_deps=0, mod_lock=SR_LOCK_READ, mi_opts=68, sid=104,
orig_name=0x2775850 "netopeer2", orig_data=0x27fe720, timeout_ms=60000,
ds_lock_timeout_ms=0, get_opts=0)
at /workspace/siqbuild/PRG-BUILDS/onecell_integration_6.5/6.5.3.006/DUOTA/ran_3rdparty/open_source/netconf/sysrepo/src/modinfo.c:2092
#8  0x00007fc0ae9f9f49 in sr_get_data (session=0x27f9570,
xpath=0x7fc088001d40 "/o-ran-aggregation-base:aggregated-o-ru/aggregation/ru-instance", max_depth=0, timeout_ms=60000, opts=0, data=0x7fc0aa027960)
at /workspace/siqbuild/PRG-BUILDS/onecell_integration_6.5/6.5.3.006/DUOTA/ran_3rdparty/open_source/netconf/sysrepo/src/sysrepo.c:2672
#9  0x000000000040ffb0 in op_filter_data_get (session=0x27f9570, max_depth=0,
get_opts=0, filter=0x7fc0aa0279e0, ev_sess=0x7fc088001790,
data=0x7fc0aa0279f0)
at /workspace/siqbuild/PRG-BUILDS/onecell_integration_6.5/6.5.3.006/DUOTA/ran_3rdparty/open_source/netconf/netopeer2/src/common.c:1205
#10 0x0000000000418785 in np2srv_rpc_getdata_cb (session=0x7fc088001790,
UNUSED_sub_id=14, UNUSED_op_path=0x2819d40 "/ietf-netconf-nmda:get-data",
input=0x7fc088001ed0, event=SR_EV_RPC, UNUSED_request_id=1,
output=0x7fc088000ee0, UNUSED_private_data=0x0)
at /workspace/siqbuild/PRG-BUILDS/onecell_integration_6.5/6.5.3.006/DUOTA/ran_3rdparty/open_source/netconf/netopeer2/src/netconf_nmda.c:218
#11 0x00007fc0aea3aece in sr_shmsub_rpc_listen_call_callback (
rpc_sub=0x2819c70, ev_sess=0x7fc088001790, input_op=0x7fc088001ed0,
event=SR_SUB_EV_RPC, request_id=1, output_op=0x7fc0aa027ba0,
err_code=0x7fc0aa027b94)
at /workspace/siqbuild/PRG-BUILDS/onecell_integration_6.5/6.5.3.006/DUOTA/ran_3rdparty/open_source/netconf/sysrepo/src/shm_sub.c:2972
#12 0x00007fc0aea3bb48 in sr_shmsub_rpc_listen_process_rpc_events (
rpc_subs=0x27b88b0, conn=0x25ff2f0)
at /workspace/siqbuild/PRG-BUILDS/onecell_integration_6.5/6.5.3.006/DUOTA/ran_3rdparty/open_source/netconf/sysrepo/src/shm_sub.c:3257
#13 0x00007fc0ae9fd93a in sr_subscription_process_events (
subscription=0x27c12c0, session=0x0, stop_time_in=0x7fc0aa027cb0)
at /workspace/siqbuild/PRG-BUILDS/onecell_integration_6.5/6.5.3.006/DUOTA/ran_3rdparty/open_source/netconf/sysrepo/src/sysrepo.c:4095
#14 0x00007fc0aea3ca75 in sr_shmsub_listen_thread (arg=0x27c12c0)
at /workspace/siqbuild/PRG-BUILDS/onecell_integration_6.5/6.5.3.006/DUOTA/ran_3rdparty/open_source/netconf/sysrepo/src/shm_sub.c:3650
#15 0x00007fc0aefc2ea5 in start_thread () from /usr/lib64/libpthread.so.0
#16 0x00007fc0ae71cb0d in clone () from /usr/lib64/libc.so.6
GDB Binary File Name : netopeer2-server

Scenario to reproduce (this is mostly happening when a list entry added in both operational and running datastore but removal done only from operational datastore) Used netopeer2-cli, netopeer2-server,

1. install the yangs: ru-yang.zip

$SYSREPOCTL -i $MODDIR/ietf-hardware.yang -s $MODDIR -e hardware-state -v4
    $SYSREPOCTL -i $MODDIR/o-ran-interfaces.yang -s $MODDIR -e UDPIP-BASED-CU-PLANE -v4
    $SYSREPOCTL -i $MODDIR/o-ran-agg-sync.yang -s $MODDIR -v4
    $SYSREPOCTL -i $MODDIR/o-ran-agg-uplane-conf.yang -s $MODDIR -v4
    $SYSREPOCTL -i $MODDIR/o-ran-ru.yang -s $MODDIR -v4

2. run netopeer2-server

3. run netopeer2-cli

4. run application_changes_example o-ran-aggregation-base

5. trigger operational datastore update (sr_set_item_str ) make entry in operational datastore : xpath,value refer attached ruoperfull-22.txt

6. operational datastore update (sr_set_item_str) :
   ruoperfull-33.txt

7. trigger edit-config from netopeer2-cli :
   ruedit.txt

8. operational datastore delete one list entry : ruoperdel-22.txt

9. from netopeer2-cli do get-data --datastore operational --filter-subtree

<aggregated-o-ru xmlns="urn:o-ran:agg-base:1.0">
            <aggregation>
               <ru-instance/>
               <hardware-model xmlns="urn:o-ran:agg-hardware:1.0">
                  <hardware>
                     <component>
                        <name/>
                        <product-code xmlns="urn:o-ran:hardware:1.0"/>
                        <serial-num/>
                     </component>
                  </hardware>
               </hardware-model>
               <cell-id/>
            </aggregation>
         </aggregated-o-ru>

After giving (9), netopeer2-server crashes with above bt.

[michalvasko]

Does not really look like a simple reproducer so you will have to update to the latest version and reproduce it there yourself for me to try it. Otherwise I will just assume it has been fixed already.

[meghna-doshi]

Hi Michal, Did you give it a try on the latest version? It will take sometime for us to upgrade to latest version and confirm whether issue still exists. If you can give a try, it will be very useful.

[michalvasko]

I have not but I wanted to except you have not provided a full reproducer. Those text files with XPaths are not particularly useful, you cannot see the actual change. But I do not understand why it is a problem to test it yourself, it is really simple to use a virtual machine or even just some container where you install all the dependencies and test the use-case with the latest version.

[meghna-doshi]

Hi Michal, Ok, so exactly what more information do you need on top of the steps given above to reproduce. We will test with latest version and provide the information.

[michalvasko]

For example

trigger operational datastore update (sr_set_item_str ) make entry in operational datastore : xpath,value refer attached ruoperfull-22.txt

What is that supposed to mean? Are they all the XPaths and values that you have stored using sr_set_item_str()? To actually perform that, I would need to write my own example app, it would be nice if you provided it instead.

[meghna-doshi]

Oh ok, please find attached the sample app to trigger operational datastore updates using the xpath, value pair files. oper_data_example_gov.zip

[michalvasko]

Okay, using the files in your testing app seemed to work and after performing all the mentioned steps the server continued to work and provided a reply, no crash occurred.