srsRAN_Project icon indicating copy to clipboard operation
srsRAN_Project copied to clipboard

Published 20 hours ago • verified publisher icon srsran

Most Flexric xAPP are crashing with srsRAN

Open fklement opened this issue 11 months ago • 12 comments

Issue Description

I have tried all xAPPs from the br-flexric branch of the Flexric repo. Only the helloworld xAPP works. Otherwise the near-RT RIC crashes.

It seems that something is wrong with the subscription request. Unfortunately I do not have enough knowledge and understanding about the code details to understand the error.

Setup Details

Normal setup according to the srsRAN documentation.

Expected Behavior

I expect that the xAPPs are running successfully.

Actual Behaviour

Python Examples: - xapp_all_moni.py:

xAPP Terminal:
[xApp]: E42 SETUP-REQUEST sent
adding event fd = 6 ev-> 5
[xApp]: E42 SETUP-RESPONSE received
[xApp]: xApp ID = 7
Registered E2 Nodes = 1
Pending event size before remove = 1
+-------+---------+-------+-----------+
|   idx |   nb_id |   mcc | mnc       |
+=======+=========+=======+===========+
|   411 |       1 |     1 | ngran_gNB |
+-------+---------+-------+-----------+
<<<< Subscribe to MAC with time period 100_ms >>>>
Generated of req_id = 1
E42_RIC_SUBSCRIPTION_REQUEST 31
adding event fd = 6 ev-> 6
[xApp]: RIC SUBSCRIPTION REQUEST sent
python3: /root/flexric/src/xApp/e42_xapp.c:268: e2_event_loop_xapp: Assertion `*e.p_ev != E42_RIC_SUBSCRIPTION_REQUEST_PENDING_EVENT && "Timeout waiting for Report. Connection lost with the RIC?"' failed.
Aborted (core dumped)

Near-RT RIC:
[iApp]: E42 SETUP-REQUEST received
[iApp]: E42 SETUP-RESPONSE sent
[iApp]: SUBSCRIPTION-REQUEST xapp_ric_id->ric_id.ran_func_id 142
[E2AP] SUBSCRIPTION REQUEST generated
[NEAR-RIC]: nb_id 411 port = 34768
nearRT-RIC: /root/flexric/src/ric/msg_handler_ric.c:236: e2ap_handle_subscription_failure_ric: Assertion `0!=0 && "Not implemented"' failed.
Aborted (core dumped)

- xapp_cust_moni.py:

xAPP Terminal:
[xApp]: E42 SETUP-REQUEST sent
adding event fd = 6 ev-> 5
[xApp]: E42 SETUP-RESPONSE received
[xApp]: xApp ID = 7
Registered E2 Nodes = 1
Pending event size before remove = 1
Global E2 Node [0]: PLMN MCC = 1
Global E2 Node [0]: PLMN MNC = 1
Generated of req_id = 1
E42_RIC_SUBSCRIPTION_REQUEST 31
adding event fd = 6 ev-> 6
[xApp]: RIC SUBSCRIPTION REQUEST sent
python3: /root/flexric/src/xApp/e42_xapp.c:268: e2_event_loop_xapp: Assertion `*e.p_ev != E42_RIC_SUBSCRIPTION_REQUEST_PENDING_EVENT && "Timeout waiting for Report. Connection lost with the RIC?"' failed.
Aborted (core dumped)

Near-RT RIC:
[iApp]: Initializing ...
[iApp]: nearRT-RIC IP Address = 127.0.0.1, PORT = 36422
fd created with 6
[NEAR-RIC]: Initializing Task Manager with 2 threads
Received message with id = 411, port = 18878
[E2AP] Received SETUP-REQUEST from PLMN   1. 1 Node ID 411 RAN type ngran_gNB
[NEAR-RIC]: Accepting RAN function ID 2 with def = ORAN-E2SM-KPM
[iApp]: no xApp connected, no need to generate E42 UPDATE-E2-NODE
[iApp]: E42 SETUP-REQUEST received
[iApp]: E42 SETUP-RESPONSE sent
[iApp]: SUBSCRIPTION-REQUEST xapp_ric_id->ric_id.ran_func_id 142
[E2AP] SUBSCRIPTION REQUEST generated
[NEAR-RIC]: nb_id 411 port = 18878
nearRT-RIC: /root/flexric/src/ric/msg_handler_ric.c:236: e2ap_handle_subscription_failure_ric: Assertion `0!=0 && "Not implemented"' failed.
Aborted (core dumped)

- xapp_slice_moni_ctrl.py:

xAPP Terminal:
[xApp]: E42 SETUP-REQUEST sent
adding event fd = 5 ev-> 5
[xApp]: E42 SETUP-RESPONSE received
[xApp]: xApp ID = 7
Registered E2 Nodes = 1
Pending event size before remove = 1
Generated of req_id = 1
E42_RIC_SUBSCRIPTION_REQUEST 31
adding event fd = 5 ev-> 6
[xApp]: RIC SUBSCRIPTION REQUEST sent
python3: /root/flexric/src/xApp/e42_xapp.c:268: e2_event_loop_xapp: Assertion `*e.p_ev != E42_RIC_SUBSCRIPTION_REQUEST_PENDING_EVENT && "Timeout waiting for Report. Connection lost with the RIC?"' failed.
Aborted (core dumped)

Near-RT RIC:
[iApp]: E42 SETUP-REQUEST received
[iApp]: E42 SETUP-RESPONSE sent
[iApp]: SUBSCRIPTION-REQUEST xapp_ric_id->ric_id.ran_func_id 145
[E2AP] SUBSCRIPTION REQUEST generated
[NEAR-RIC]: nb_id 411 port = 14724
nearRT-RIC: /root/flexric/src/ric/msg_handler_ric.c:236: e2ap_handle_subscription_failure_ric: Assertion `0!=0 && "Not implemented"' failed.
Aborted (core dumped)

C Examples: -xapp_all_moni.c:

xAPP Terminal:
[xApp]: E42 SETUP-REQUEST sent
adding event fd = 6 ev-> 5
[xApp]: E42 SETUP-RESPONSE received
[xApp]: xApp ID = 7
Registered E2 Nodes = 1
Pending event size before remove = 1
E2 node idx 0 info: nb_id 411, mcc 1, mnc 1, mnc_digit_len 2, ran_type ngran_gNB
Registered E2 node idx 0, supported RAN Func ID = 2
 [xApp]: reporting period = 1000 [ms]
[xApp]: Filter UEs by S-NSSAI criteria where SST = 1
xApp subscribes RAN Func ID 2 in E2 node idx 0
Generated of req_id = 1
E42_RIC_SUBSCRIPTION_REQUEST 31
adding event fd = 6 ev-> 6
[xApp]: RIC SUBSCRIPTION REQUEST sent
xapp_all_moni: /root/flexric/src/xApp/e42_xapp.c:268: e2_event_loop_xapp: Assertion `*e.p_ev != E42_RIC_SUBSCRIPTION_REQUEST_PENDING_EVENT && "Timeout waiting for Report. Connection lost with the RIC?"' failed.
Aborted (core dumped)

Near-RT RIC:
[E2AP] Received SETUP-REQUEST from PLMN   1. 1 Node ID 411 RAN type ngran_gNB
[NEAR-RIC]: Accepting RAN function ID 2 with def = ORAN-E2SM-KPM
[iApp]: no xApp connected, no need to generate E42 UPDATE-E2-NODE
[iApp]: E42 SETUP-REQUEST received
[iApp]: E42 SETUP-RESPONSE sent
[iApp]: SUBSCRIPTION-REQUEST xapp_ric_id->ric_id.ran_func_id 2
[E2AP] SUBSCRIPTION REQUEST generated
[NEAR-RIC]: nb_id 411 port = 64955
Pending event timeout happened. Communication with E2 Node lost?
nearRT-RIC: /root/flexric/src/ric/near_ric.c:270: consume_fd: Assertion `fd > 0' failed.
Aborted (core dumped)

For the C xAPPs the gNB prints out the following: "Failed to unpack E2SM KPM Action Definition"

Steps to reproduce the problem

  1. Follow the documentation on how to setup srsRAN.
  2. Start the Flexric Near-RT RIC
  3. Connect the srsRAN gNB
  4. Connect the srsrue
  5. Start the xAPP

fklement avatar Mar 22 '24 09:03 fklement

Share the logs and pcaps please.

On Fri, 22 Mar 2024, 09:22 Felix Klement, @.***> wrote:

Issue Description

I have tried all xAPPs from the br-flexric branch of the Flexric repo. Only the helloworld xAPP works. Otherwise the near-RT RIC crashes.

It seems that something is wrong with the subscription request. Unfortunately I do not have enough knowledge and understanding about the code details to understand the error. Setup Details

Normal setup according to the srsRAN documentation. Expected Behavior

I expect that the xAPPs are running successfully. Actual Behaviour

Python Examples: - xapp_all_moni.py:

xAPP Terminal: [xApp]: E42 SETUP-REQUEST sent adding event fd = 6 ev-> 5 [xApp]: E42 SETUP-RESPONSE received [xApp]: xApp ID = 7 Registered E2 Nodes = 1 Pending event size before remove = 1 +-------+---------+-------+-----------+ | idx | nb_id | mcc | mnc | +=======+=========+=======+===========+ | 411 | 1 | 1 | ngran_gNB | +-------+---------+-------+-----------+ <<<< Subscribe to MAC with time period 100_ms >>>> Generated of req_id = 1 E42_RIC_SUBSCRIPTION_REQUEST 31 adding event fd = 6 ev-> 6 [xApp]: RIC SUBSCRIPTION REQUEST sent python3: /root/flexric/src/xApp/e42_xapp.c:268: e2_event_loop_xapp: Assertion `*e.p_ev != E42_RIC_SUBSCRIPTION_REQUEST_PENDING_EVENT && "Timeout waiting for Report. Connection lost with the RIC?"' failed. Aborted (core dumped)

Near-RT RIC: [iApp]: E42 SETUP-REQUEST received [iApp]: E42 SETUP-RESPONSE sent [iApp]: SUBSCRIPTION-REQUEST xapp_ric_id->ric_id.ran_func_id 142 [E2AP] SUBSCRIPTION REQUEST generated [NEAR-RIC]: nb_id 411 port = 34768 nearRT-RIC: /root/flexric/src/ric/msg_handler_ric.c:236: e2ap_handle_subscription_failure_ric: Assertion `0!=0 && "Not implemented"' failed. Aborted (core dumped)

- xapp_cust_moni.py:

xAPP Terminal: [xApp]: E42 SETUP-REQUEST sent adding event fd = 6 ev-> 5 [xApp]: E42 SETUP-RESPONSE received [xApp]: xApp ID = 7 Registered E2 Nodes = 1 Pending event size before remove = 1 Global E2 Node [0]: PLMN MCC = 1 Global E2 Node [0]: PLMN MNC = 1 Generated of req_id = 1 E42_RIC_SUBSCRIPTION_REQUEST 31 adding event fd = 6 ev-> 6 [xApp]: RIC SUBSCRIPTION REQUEST sent python3: /root/flexric/src/xApp/e42_xapp.c:268: e2_event_loop_xapp: Assertion `*e.p_ev != E42_RIC_SUBSCRIPTION_REQUEST_PENDING_EVENT && "Timeout waiting for Report. Connection lost with the RIC?"' failed. Aborted (core dumped)

Near-RT RIC: [iApp]: Initializing ... [iApp]: nearRT-RIC IP Address = 127.0.0.1, PORT = 36422 fd created with 6 [NEAR-RIC]: Initializing Task Manager with 2 threads Received message with id = 411, port = 18878 [E2AP] Received SETUP-REQUEST from PLMN 1. 1 Node ID 411 RAN type ngran_gNB [NEAR-RIC]: Accepting RAN function ID 2 with def = ORAN-E2SM-KPM [iApp]: no xApp connected, no need to generate E42 UPDATE-E2-NODE [iApp]: E42 SETUP-REQUEST received [iApp]: E42 SETUP-RESPONSE sent [iApp]: SUBSCRIPTION-REQUEST xapp_ric_id->ric_id.ran_func_id 142 [E2AP] SUBSCRIPTION REQUEST generated [NEAR-RIC]: nb_id 411 port = 18878 nearRT-RIC: /root/flexric/src/ric/msg_handler_ric.c:236: e2ap_handle_subscription_failure_ric: Assertion `0!=0 && "Not implemented"' failed. Aborted (core dumped)

- xapp_slice_moni_ctrl.py:

xAPP Terminal: [xApp]: E42 SETUP-REQUEST sent adding event fd = 5 ev-> 5 [xApp]: E42 SETUP-RESPONSE received [xApp]: xApp ID = 7 Registered E2 Nodes = 1 Pending event size before remove = 1 Generated of req_id = 1 E42_RIC_SUBSCRIPTION_REQUEST 31 adding event fd = 5 ev-> 6 [xApp]: RIC SUBSCRIPTION REQUEST sent python3: /root/flexric/src/xApp/e42_xapp.c:268: e2_event_loop_xapp: Assertion `*e.p_ev != E42_RIC_SUBSCRIPTION_REQUEST_PENDING_EVENT && "Timeout waiting for Report. Connection lost with the RIC?"' failed. Aborted (core dumped)

Near-RT RIC: [iApp]: E42 SETUP-REQUEST received [iApp]: E42 SETUP-RESPONSE sent [iApp]: SUBSCRIPTION-REQUEST xapp_ric_id->ric_id.ran_func_id 145 [E2AP] SUBSCRIPTION REQUEST generated [NEAR-RIC]: nb_id 411 port = 14724 nearRT-RIC: /root/flexric/src/ric/msg_handler_ric.c:236: e2ap_handle_subscription_failure_ric: Assertion `0!=0 && "Not implemented"' failed. Aborted (core dumped)

C Examples: -xapp_all_moni.c:

xAPP Terminal: [xApp]: E42 SETUP-REQUEST sent adding event fd = 6 ev-> 5 [xApp]: E42 SETUP-RESPONSE received [xApp]: xApp ID = 7 Registered E2 Nodes = 1 Pending event size before remove = 1 E2 node idx 0 info: nb_id 411, mcc 1, mnc 1, mnc_digit_len 2, ran_type ngran_gNB Registered E2 node idx 0, supported RAN Func ID = 2 [xApp]: reporting period = 1000 [ms] [xApp]: Filter UEs by S-NSSAI criteria where SST = 1 xApp subscribes RAN Func ID 2 in E2 node idx 0 Generated of req_id = 1 E42_RIC_SUBSCRIPTION_REQUEST 31 adding event fd = 6 ev-> 6 [xApp]: RIC SUBSCRIPTION REQUEST sent xapp_all_moni: /root/flexric/src/xApp/e42_xapp.c:268: e2_event_loop_xapp: Assertion `*e.p_ev != E42_RIC_SUBSCRIPTION_REQUEST_PENDING_EVENT && "Timeout waiting for Report. Connection lost with the RIC?"' failed. Aborted (core dumped)

Near-RT RIC: [E2AP] Received SETUP-REQUEST from PLMN 1. 1 Node ID 411 RAN type ngran_gNB [NEAR-RIC]: Accepting RAN function ID 2 with def = ORAN-E2SM-KPM [iApp]: no xApp connected, no need to generate E42 UPDATE-E2-NODE [iApp]: E42 SETUP-REQUEST received [iApp]: E42 SETUP-RESPONSE sent [iApp]: SUBSCRIPTION-REQUEST xapp_ric_id->ric_id.ran_func_id 2 [E2AP] SUBSCRIPTION REQUEST generated [NEAR-RIC]: nb_id 411 port = 64955 Pending event timeout happened. Communication with E2 Node lost? nearRT-RIC: /root/flexric/src/ric/near_ric.c:270: consume_fd: Assertion `fd > 0' failed. Aborted (core dumped)

For the C xAPPs the gNB prints out the following: "Failed to unpack E2SM KPM Action Definition" Steps to reproduce the problem

  1. Follow the documentation on how to setup srsRAN.
  2. Start the Flexric Near-RT RIC
  3. Connect the srsRAN gNB
  4. Connect the srsrue
  5. Start the xAPP

— Reply to this email directly, view it on GitHub https://github.com/srsran/srsRAN_Project/issues/525, or unsubscribe https://github.com/notifications/unsubscribe-auth/AATLGFFC2SHFQXN2GWRBUH3YZPZ6BAVCNFSM6AAAAABFC65MAOVHI2DSMVQWIX3LMV43ASLTON2WKOZSGIYDEMBVGQZTINY . You are receiving this because you are subscribed to this thread.Message ID: @.***>

yagoda avatar Mar 22 '24 09:03 yagoda

@yagoda I uploaded one cap file for the xapp_all_moni here: https://file.io/QoumFw8zM2fo What logs despite the console outputs I already posted do you need?

fklement avatar Mar 22 '24 09:03 fklement

The gnb logs are usually recorded at /tmp/gnb.log, just make sure to enable logging in the gnb config

yagoda avatar Mar 22 '24 09:03 yagoda

Ah ok sorry... sure. Here the gnb.log gnb.log Here the link to the pcap files:

If you need more logs from the other xAPPs that failed, just let me know.

fklement avatar Mar 22 '24 09:03 fklement

I think the problem is occurring because of the following:

2024-03-22T09:42:05.856981 [E2-ASN1-PCK] [I] Received PDU of 204 bytes
2024-03-22T09:42:05.857006 [E2      ] [I] Handling E2 PDU of type initiatingMessage
2024-03-22T09:42:05.857007 [E2      ] [I] E2AP SDU, "initiatingMessage.RICsubscriptionRequest"
2024-03-22T09:42:05.857008 [E2      ] [I] Received RIC Subscription Request
2024-03-22T09:42:05.857010 [E2      ] [I] E2AP: Received subscription request
2024-03-22T09:42:05.857467 [ASN1    ] [E] The provided enum is not within the range of possible values (38>=2)
2024-03-22T09:42:05.857469 [ASN1    ] [E] Invalid choice id=38 for choice type meas_type_c
2024-03-22T09:42:05.857469 [ASN1    ] [E] Invalid choice id=38 for choice type meas_type_c
2024-03-22T09:42:05.857469 [ASN1    ] [E] [/root/srsRAN_Project/lib/asn1/e2ap/e2sm_kpm.cpp][2962] Decoding failure.
2024-03-22T09:42:05.857470 [ASN1    ] [E] [/root/srsRAN_Project/include/srsran/asn1/e2ap/../asn1_utils.h][1155] Decoding failure.
2024-03-22T09:42:05.857470 [ASN1    ] [E] [/root/srsRAN_Project/lib/asn1/e2ap/e2sm_kpm.cpp][3395] Decoding failure.
2024-03-22T09:42:05.857471 [ASN1    ] [E] [/root/srsRAN_Project/lib/asn1/e2ap/e2sm_kpm.cpp][3450] Decoding failure.
2024-03-22T09:42:05.857471 [ASN1    ] [E] [/root/srsRAN_Project/lib/asn1/e2ap/e2sm_kpm.cpp][3795] Decoding failure.
2024-03-22T09:42:05.857471 [ASN1    ] [E] [/root/srsRAN_Project/lib/asn1/e2ap/e2sm_kpm.cpp][3580] Decoding failure.
2024-03-22T09:42:05.857482 [E2      ] [E] Unknown service model 2 -> do not admit action 0  (type 0)
2024-03-22T09:42:05.857623 [E2-SUBSCRIBER] [E] Action not supported 0
2024-03-22T09:42:05.857624 [E2-SUBSCRIBER] [E] Failed to setup subscription
2024-03-22T09:42:05.857636 [ASN1    ] [E] The provided enum is not within the range of possible values (6>=6)
2024-03-22T09:42:05.857636 [ASN1    ] [E] Invalid choice id=6 for choice type cause_c
2024-03-22T09:42:05.857637 [ASN1    ] [E] [/root/srsRAN_Project/include/srsran/asn1/asn1_ap_utils.h][197] Encoding failure.
2024-03-22T09:42:05.857638 [ASN1    ] [E] [/root/srsRAN_Project/lib/asn1/e2ap/e2ap.cpp][13947] Encoding failure.
2024-03-22T09:42:05.857638 [ASN1    ] [E] [/root/srsRAN_Project/include/srsran/asn1/asn1_ap_utils.h][335] Encoding failure.
2024-03-22T09:42:05.857638 [ASN1    ] [E] [/root/srsRAN_Project/lib/asn1/e2ap/e2ap.cpp][15652] Encoding failure.
2024-03-22T09:42:05.857638 [ASN1    ] [E] [/root/srsRAN_Project/lib/asn1/e2ap/e2ap.cpp][15813] Encoding failure.
2024-03-22T09:42:05.857639 [ASN1    ] [E] [/root/srsRAN_Project/lib/asn1/e2ap/e2ap.cpp][15973] Encoding failure.
2024-03-22T09:42:05.857639 [E2-ASN1-PCK] [E] Failed to pack E2 PDU
2024-03-22T09:42:05.857639 [E2      ] [I] E2AP: Sending subscription failure

fklement avatar Mar 22 '24 09:03 fklement

Hey, it would be nice if we got a pcap file of the E2AP. The one shared above doesn't seem to have the problematic E2AP message. You can enable pcap writing via pcap --e2ap_enable=true

frankist avatar Mar 25 '24 10:03 frankist

@frankist I can try to upload the pcap file tomorrow, can't get to it today.

What I also noticed is that in (https://docs.srsran.com/projects/project/en/latest/tutorials/source/flexric/source/index.html#xapp) there is only talk of the xapp_oran_moni xAPP. Do you currently only support this one xAPP?

fklement avatar Mar 26 '24 13:03 fklement

Hi, indeed, currently we just support xapp_oran_moni xapp from flexric framework. We are currently working to support also control requests.

pgawlowicz avatar Apr 04 '24 15:04 pgawlowicz

Hi @fklement, We have just released a repository that allows running ORAN-SC-RIC as a multi-container Docker application with a single docker command. You can find it here: https://github.com/srsran/oran-sc-ric

In addition, the repository provides two example Python-based xApps that allow requesting and monitoring E2 metrics from the gNB using all reporting styles (1-5) of the e2sm-kpm module (which was updated to version 3).

The Quick Start section in the readme file demonstrates how to use it. Note that until the next code release, you have to use the test branch from the srsRAN_Project repository.

pgawlowicz avatar Apr 11 '24 14:04 pgawlowicz

Hi @fklement, We have just released a repository that allows running ORAN-SC-RIC as a multi-container Docker application with a single docker command. You can find it here: https://github.com/srsran/oran-sc-ric

In addition, the repository provides two example Python-based xApps that allow requesting and monitoring E2 metrics from the gNB using all reporting styles (1-5) of the e2sm-kpm module (which was updated to version 3).

The Quick Start section in the readme file demonstrates how to use it. Note that until the next code release, you have to use the test branch from the srsRAN_Project repository.

Hi I've just seen it! Great work. Thanks for updating me.

fklement avatar Apr 11 '24 14:04 fklement

I'm experiencing the same error as @fklement, is it preferred to use the ORAN-SC-RIC setup going forward?

corbanvilla avatar May 04 '24 18:05 corbanvilla

Hi, Please note that we support only KPM and RC modules. Therefore, if you try to run an xApp using a custom module, it will not work. Also please make sure that you are requesting metrics supported by srsRAN using KPM module, and control only supported parameters with RC module (currently only number of PRBs per UE).

pgawlowicz avatar May 13 '24 09:05 pgawlowicz