srsRAN_Project
srsRAN_Project copied to clipboard
srsran
Large number of DL U-Plane messages not arriving Foxconn RU on time
Issue Description
During our integration of Open5gs + srsRAN 24.04 + Foxconn RU, although the srsRAN DU and Foxconn RU have successfully connected, and the DU is sending C/U-Plane messages to the RU, with the RU also sending back U-Plane messages to the DU, the RU logs show that a large number of U-Plane messages are early and late.
- I have tried adjusting the t1a_max_up and t1a_min_up parameters, but many U-Plane messages still cannot arrive at the RU on time.
- Some of the UL U-Plane messages received by srsRAN are also arriving late.
I want to inquire about what the problem might be and how I should adjust the settings.
RU log:
[2024-08-27 04:56:03.017] xRN: total=0 c_early=0 c_on=0 c_late=0 err_tci=0 err_ecpri=0 err_port=0 err_sct=0 err_total=0
[2024-08-27 04:56:03.168] xRN: u-plane u_early=0 u_on=0 u_late=0
[2024-08-27 04:56:04.020] xRN: total=32036 c_early=0 c_on=2568 c_late=0 err_tci=0 err_ecpri=0 err_port=0 err_sct=0 err_total=16988
[2024-08-27 04:56:04.221] xRN: u-plane u_early=12428 u_on=15895 u_late=9179
[2024-08-27 04:56:05.023] xRN: total=72032 c_early=0 c_on=5768 c_late=0 err_tci=0 err_ecpri=0 err_port=0 err_sct=0 err_total=38184
[2024-08-27 04:56:05.225] xRN: u-plane u_early=24447 u_on=31278 u_late=18059
srsRAN OFH log:
2024-08-27T04:55:26.196070 [OFH ] [I] Started the operation of the Open Fronthaul interface
2024-08-27T04:55:26.196748 [SCHED ] [I] [ 339.18] Processed slot events pci=1: Cell creation idx=0 pci=1
2024-08-27T04:55:27.196041 [OFH ] [I] Received packets: rx_total=4800 rx_early=0, rx_on_time=4707, rx_late=93
2024-08-27T04:55:27.196079 [METRICS ] [I] Cell Scheduler Metrics: error_indications=0 mean_latency=5usec latency_hist=[1999, 0, 0, 1, 0, 0, 0, 0, 0, 0]
2024-08-27T04:55:28.196040 [OFH ] [I] Received packets: rx_total=4800 rx_early=0, rx_on_time=4684, rx_late=116
2024-08-27T04:55:28.196110 [METRICS ] [I] Cell Scheduler Metrics: error_indications=0 mean_latency=5usec latency_hist=[2000, 0, 0, 0, 0, 0, 0, 0, 0, 0]
Setup Details
Topology
gNB
Hardware:
| Item | Info |
|---|---|
| CPU | Intel(R) Xeon(R) CPU E5-2695 v4 @ 2.10GHz |
| Memory | 64G |
| Disk | 240G |
| NIC | Intel E810 (driver: ice, firmware-version: 3.20 0x8000d853 1.3146.0, bus-info: 0000:83:00.0) |
| Server Model | Supermicro SSG-6028R-E1CR12L |
Software:
| Item | Info |
|---|---|
| OS | ubuntu 22.04 |
| Kernel | Linux 5.15.0-1067-realtime |
| DPDK | dpdk-23.11 |
| LinuxPTP | linuxptp-3.1.1 |
| gNB Branch | srsRAN 24.04 |
RU
| Item | Info |
|---|---|
| product_name | Foxconn RPQN-7801 |
| model id | |
| firmware version | v3.1.15q |
Expected Behavior
- All U-Plane messages arrive at the RU on time, and the U-Plane messages sent back by the RU also arrive on time.
Actual Behaviour
- A large number of U-Plane messages received by the RU are early and late, and a small portion of UL U-Plane messages are late
- Foxconn RU, srsRAN, ptp log
Steps to reproduce the problem
- srsRAN Foxconn RU config
# This example configuration outlines how to configure the srsRAN Project CU/DU to use an O-RU and split 7.2. This is specifically for use
# with the Foxconn RPQN O-RU. This config will create a single TDD cell transmitting in band n78, with 20 MHz bandwidth and 30 kHz sub-carrier-spacing.
# The parameters used to configure the RU are found in the `ru_ofh` sub-section. This configuration makes used of the OFH Lib from SRS to enable split 7.2.
amf:
addr: 192.168.8.108
bind_addr: 192.168.8.53
ru_ofh:
t1a_max_cp_dl: 470 #420
t1a_min_cp_dl: 285 #280 #250
t1a_max_cp_ul: 429
t1a_min_cp_ul: 285 # 280 # 250
t1a_max_up: 230 # 196
t1a_min_up: 140 # 96 # 80
ta4_max: 200 # 75
ta4_min: 25
is_prach_cp_enabled: true
ignore_ecpri_payload_size: true
compr_method_ul: bfp
compr_bitwidth_ul: 9
compr_method_dl: bfp
compr_bitwidth_dl: 9
compr_method_prach: bfp
compr_bitwidth_prach: 9
enable_ul_static_compr_hdr: false
enable_dl_static_compr_hdr: false
iq_scaling: 1.0
cells:
- network_interface: enp131s0f0
ru_mac_addr: 6c:ad:ad:00:08:6c
du_mac_addr: 50:7c:6f:3c:35:1a
vlan_tag_cp: 1
vlan_tag_up: 1
prach_port_id: [4, 5, 6, 7]
dl_port_id: [0, 1]
ul_port_id: [0, 1, 2, 3]
cell_cfg:
dl_arfcn: 649980
band: 78
channel_bandwidth_MHz: 20
common_scs: 30
plmn: "00101"
tac: 1
pci: 1
nof_antennas_dl: 2
nof_antennas_ul: 4
prach:
prach_config_index: 159
prach_root_sequence_index: 1
zero_correlation_zone: 0
prach_frequency_start: 0
pdsch:
mcs_table: qam256
log:
filename: gnb.log
all_level: info
pcap:
mac_enable: false
mac_filename: gnb_mac.pcap
ngap_enable: true
ngap_filename: gnb_ngap.pcap
- Foxconn RU RRH config
root@arria10:~/test# cat RRHconfig_xran.xml
<!-- -->
<!-- Common -->
<!-- -->
<!-- RRH_DST_MAC_ADDR: Destination MAC address, fill with 6 bytes and separate each others by colon -->
RRH_DST_MAC_ADDR = 00:11:22:33:44:66
<!-- RRH_SRC_MAC_ADDR: Source MAC address, fill with 6 bytes and separate each others by colon -->
RRH_SRC_MAC_ADDR = 6c:ad:ad:00:08:6c
<!-- RRH_EN_EAXC_ID: Enable using eAxC ID field defined in O-RAN spec. -->
<!-- When 0 is set, RU port ID=0,1,2,3 are used for PDSCH/PUSCH if RRH_TRX_EN_BIT_MASK = 0x0F -->
<!-- When 0 is set, RU port ID=4,5,6,7 are used for PRACH if RRH_TRX_EN_BIT_MASK = 0x0F -->
<!-- When 0 is set, RU port ID=0,1 are used for PDSCH/PUSCH if RRH_TRX_EN_BIT_MASK = 0x03 -->
<!-- When 0 is set, RU port ID=2,3 are used for PRACH if RRH_TRX_EN_BIT_MASK = 0x03 -->
RRH_EN_EAXC_ID = 0
<!-- RRH_EAXC_ID_TYPE1: Specify the eAxC ID for type1 message -->
RRH_EAXC_ID_TYPE1 = 0x0, 0x1, 0x2, 0x3
<!-- RRH_EAXC_ID_TYPE3: Specify the eAxC ID for type3 message -->
RRH_EAXC_ID_TYPE3 = 0x8, 0x9, 0xA, 0xB
<!-- RRH_EN_SPC: Enable SPC or not, 0:OFF, 1:ON -->
RRH_EN_SPC = 1
<!-- RRH_LTE_OR_NR: Indicate the spec of xRAN, 0:LTE, 1:NR -->
RRH_LTE_OR_NR = 1
<!-- RRH_TRX_EN_BIT_MASK: Bit-mask of 4 TRx, bit 0: TRx0, bit1: TRx1, bit2: TRx2, bit3: TRx3 -->
RRH_TRX_EN_BIT_MASK = 0x0f
<!-- RRH_RF_EN_BIT_MASK: Bit-mask of 4 PA/LNA, bit0: PA0/LNA0, bit1: PA1/LNA1, bit2: PA2/LNA2, bit3: PA3/LNA3 -->
RRH_RF_EN_BIT_MASK = 0x0f
<!-- RRH_CMPR_HDR_PRESENT: Indicate the UdCompHdr/reserved field is present or not, 0:no present; 1:present -->
RRH_CMPR_HDR_PRESENT = 1
<!-- RRH_CMPR_TYPE: Indicate compress type. 1st for PDSCH/PUSCH, 2nd for PRACH. 0: No Cmpr; 1:block-floating -->
RRH_CMPR_TYPE = 1, 1
<!-- RRH_CMPR_BIT_LENGTH: Indicate the bit length after compression. 1st for PDSCH/PUSCH, 2nd for PRACH. -->
RRH_CMPR_BIT_LENGTH = 9, 9
<!-- RRH_TX_TRUNC_BITS: The extra truncation in fractional part of IFFT output -->
RRH_TX_TRUNC_BITS = 4
<!-- RRH_RX_TRUNC_BITS: The extra truncation in fractional part of FFT output -->
RRH_RX_TRUNC_BITS = 4
<!-- RRH_MAX_PRB: Maximum PRBs -->
RRH_MAX_PRB = 51
<!-- RRH_C_PLANE_VLAN_TAG: C-plane V-LAN tag express by hex number -->
RRH_C_PLANE_VLAN_TAG = 0x0001
<!-- RRH_U_PLANE_VLAN_TAG: U-plane V-LAN tag express by hex number -->
RRH_U_PLANE_VLAN_TAG = 0x0001
<!-- RRH_SLOT_TICKS_IN_SEC: Number of slot tick in a second (2000 slots for u=1) -->
RRH_SLOT_TICKS_IN_SEC = 2000
<!-- RRH_SLOT_PERIOD_IN_SAMPLE: Slot period in 122.88MHz (61440 for u=1) -->
RRH_SLOT_PERIOD_IN_SAMPLE = 61440
<!-- RRH_LO_FREQUENCY_KHZ: Tx and Rx PLL LO Frequency in kHz(internal or external LO) -->
RRH_LO_FREQUENCY_KHZ = 3749700, 0
<!-- RRH_TX_POWER: Target Tx power (Default=24dBm. Range:24dBm ~ 4dBm. It can be applied only when Tx power of a RRH is calibrated) -->
RRH_TX_POWER = 24, 24
<!-- RRH_TX_ATTENUATION: Tx attenuation value with 1-digit fraction for each layer (>10dB. NOTE: The attenuation value must be larget than 10dB) -->
RRH_TX_ATTENUATION = 12.0, 12.0, 12.0, 12.0
<!-- RRH_RX_ATTENUATION: Rx attenuation value with 1-digit fraction for each layer (<30dB) -->
RRH_RX_ATTENUATION = 0.0, 0.0, 0.0, 0.0
<!-- RRH_BB_GENERAL_CTRL: General control words for Baseband -->
<!-- Bit[0] of 1st word: Enable the filtering of MAC address -->
<!-- Bit[1] of 1st word: Enable the UL slot tick packets per port -->
<!-- Others: Reserved -->
RRH_BB_GENERAL_CTRL = 0x0, 0x0, 0x0, 0x0
<!-- RRH_RF_GENERAL_CTRL: General control words for RF -->
<!-- Bit[2:0] of 1st word: 7=DPD ON and init port by RRH_RF_EN_BIT_MASK, 3=DPD ON, 0=DPD off -->
<!-- Bit[0] of 2st word: 1=CLGC ON, 0=CLGC off -->
<!-- Bit[0] of 4th word: 1=for N77, 0=else -->
RRH_RF_GENERAL_CTRL = 0x3, 0x1, 0x0, 0x0
<!-- -->
<!-- PTPV2 Related -->
<!-- -->
<!-- RRH_PTPV2_GRAND_MASTER_MODE: 0: Unicast over 1G, 1:Multicast over 1G; 2: Unicast over 10G; 3: Multicast over 10G -->
RRH_PTPV2_GRAND_MASTER_MODE = 1
<!-- RRH_PTPV2_JITTER_LEVEL: The estimated jitter of PTP time packets. 0:direct connection to GM/BC, 1:light, 2:medium, 3:heavy -->
RRH_PTPV2_JITTER_LEVEL = 0
<!-- RRH_PTPV2_VLAN_ID: VLAN ID of PTPv2. 0/1: No VLAN of PTPv2; [2~4092]: valid VLAN of PTPv2; >4092: Invalid and no VLAN will be applied -->
RRH_PTPV2_VLAN_ID = 0
<!-- RRH_PTPV2_IP_MODE: 4: IPv4 (default); 6: IPv6. Note: this configuration must be defined before RRH_PTPV2_GRAND_MASTER_IP -->
RRH_PTPV2_IP_MODE = 4
<!-- RRH_PTPV2_GRAND_MASTER_IP: IP address of grand-master -->
RRH_PTPV2_GRAND_MASTER_IP = 192.167.27.150
<!-- RRH_PTPV2_SUB_DOMAIN_NUM: The sub-domain number -->
RRH_PTPV2_SUB_DOMAIN_NUM = 24
<!-- RRH_PTPV2_ACCEPTED_CLOCK_CLASS: The acceptable clockClass threshold [6~248] -->
RRH_PTPV2_ACCEPTED_CLOCK_CLASS = 248
<!-- RRH_TRACE_PERIOD: The period of trace log. It can be applied only when the start-trace-logs of M-plane is enabled -->
RRH_TRACE_PERIOD = 10
<!-- RRH_CLGC_EXPECTED_LOOP_GAIN: The clgc excepted loop gain: 0xFF: enable clgc after receive cu-plane data 300s, other value: use this value to enable clgc on dpd init -->
RRH_CLGC_EXPECTED_LOOP_GAIN = 0xFF, 0xFF, 0xFF, 0xFF
<!-- RRH_DL_IQ_SCALING: Set DL iq scaling value that 0x0: disable dl iq scaling-->
RRH_DL_IQ_SCALING = 0x10001
i<!-- RRH_CFR_PEAK_THRESHOLD: Set CFR Peak Threshold -->
RRH_CFR_PEAK_THRESHOLD = 0.5
Additional Information
[Any additional information, configuration or data that might be necessary to reproduce the issue]
Hi @Xinya25
By looking at your gnb config file, you are not enabling DPDK in the gnb, could you give it a try to see if it makes any difference?
@faluco Thank you for your suggestion!
However, I would like to ask, if the NIC (enp131s0f0) is bound to DPDK, how should the NIC executing PTP?
I previously tried configuring VF on the NIC and binding VF to DPDK to run srsRAN, while using the physical network card for PTP. However, srsRAN currently does not support the VF binding to DPDK feature.
Therefore, is it necessary to use the adjacent port network card (e.g., enp131s0f1) for PTP synchronization, or is there another method to perform PTP synchronization?
You can and should use the VF for the OFH traffic. Just specify correct one in the config and it'll work. Make sure to configure the VF without VLAN as we add the VLAN tag in the ethernet frame directly. Otherwise you end up with packets that carry the VLAN tag twice ;-)
@andrepuschmann Thank you for the reminder!
By the way, I followed the srsRAN gNB with DPDK guide and successfully ran srsRAN gNB with DPDK, as shown in the screenshot below.
However, this guide binds the physical NIC to DPDK and does not mention how the PTP implementation works.
Therefore, I wanted to ask if you have a guide for configuring VF and binding DPDK with VF, as well as how PTP is implemented.
Here is the method I am currently using to configure VF and bind DPDK with VF. I added a VLAN tag to the VF configuration file and removed the VLAN configuration in the srsRAN RU config file to ensure there is only one VLAN tag.
setup_enp131s0f0_vf.sh
#!/bin/bash
# Set CPU power management
sudo cpupower idle-set -D 0
sudo tuned-adm profile realtime
# Load SCTP module
sudo modprobe sctp
# Set maximum ring buffers for enp131s0f0
# sudo ethtool -G enp131s0f0 rx 4096
# sudo ethtool -G enp131s0f0 tx 4096
# Set the maximum MTU
sudo ifconfig enp131s0f0 mtu 9600
# Load the Intel Ethernet Adaptive Virtual Function driver
sudo modprobe iavf
# Set up SR-IOV virtual function
sudo sh -c 'echo 0 > /sys/class/net/enp131s0f0/device/sriov_numvfs'
sudo sh -c 'echo 1 > /sys/class/net/enp131s0f0/device/sriov_numvfs'
# Create virtual function with MAC address and VLAN
sudo ip link set enp131s0f0 vf 0 mac 00:11:22:33:44:66 vlan 1 qos 0 spoofchk off mtu 1500
sleep 1
# Get PCI address for the virtual function
VF0_PCI=$(sudo lspci | grep "Virtual Function" | head -n 1 | awk '{print $1}')
# Unbind virtual function
sudo /usr/local/bin/dpdk-devbind.py --unbind $VF0_PCI
# Load VFIO-PCI module
sudo modprobe vfio-pci
# Bind virtual function to VFIO-PCI
sudo /usr/local/bin/dpdk-devbind.py --bind=vfio-pci $VF0_PCI
# Check DPDK binding status
sudo /usr/local/bin/dpdk-devbind.py --status
# Check configuration
ip link show enp131s0f0
# Show network interface settings
sudo ethtool -g enp131s0f0
sudo ifconfig enp131s0f0
Setting log:
╭─srsran@srsran-SSG-6028R-E1CR12L ~/srsRAN_Project ‹main●›
╰─$ sudo ./setup_enp131s0f0_vf.sh
Network devices using DPDK-compatible driver
============================================
0000:83:01.0 'Ethernet Adaptive Virtual Function 1889' drv=vfio-pci unused=iavf
Network devices using kernel driver
===================================
0000:04:00.0 'Ethernet Controller X710 for 10GbE SFP+ 1572' if=enp4s0f0 drv=i40e unused=vfio-pci
0000:04:00.1 'Ethernet Controller X710 for 10GbE SFP+ 1572' if=enp4s0f1 drv=i40e unused=vfio-pci
0000:04:00.2 'Ethernet Controller X710 for 10GbE SFP+ 1572' if=enp4s0f2 drv=i40e unused=vfio-pci
0000:04:00.3 'Ethernet Controller X710 for 10GbE SFP+ 1572' if=enp4s0f3 drv=i40e unused=vfio-pci
0000:06:00.0 'Ethernet Controller 10-Gigabit X540-AT2 1528' if=enp6s0f0 drv=ixgbe unused=vfio-pci *Active*
0000:06:00.1 'Ethernet Controller 10-Gigabit X540-AT2 1528' if=enp6s0f1 drv=ixgbe unused=vfio-pci
0000:81:00.0 'Ethernet Controller X710 for 10GbE SFP+ 1572' if=enp129s0f0 drv=i40e unused=vfio-pci
0000:81:00.1 'Ethernet Controller X710 for 10GbE SFP+ 1572' if=enp129s0f1 drv=i40e unused=vfio-pci
0000:81:00.2 'Ethernet Controller X710 for 10GbE SFP+ 1572' if=enp129s0f2 drv=i40e unused=vfio-pci
0000:81:00.3 'Ethernet Controller X710 for 10GbE SFP+ 1572' if=enp129s0f3 drv=i40e unused=vfio-pci
0000:83:00.0 'Ethernet Controller E810-XXV for SFP 159b' if=enp131s0f0 drv=ice unused=vfio-pci
0000:83:00.1 'Ethernet Controller E810-XXV for SFP 159b' if=enp131s0f1 drv=ice unused=vfio-pci
No 'Baseband' devices detected
==============================
No 'Crypto' devices detected
============================
No 'Eventdev' devices detected
==============================
No 'Mempool' devices detected
=============================
No 'Compress' devices detected
9: enp131s0f0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 9600 qdisc mq state UP mode DEFAULT group default qlen 1000
link/ether 50:7c:6f:3c:35:1a brd ff:ff:ff:ff:ff:ff
vf 0 link/ether 00:11:22:33:44:66 brd ff:ff:ff:ff:ff:ff, vlan 1, spoof checking off, link-state auto, trust off
Ring parameters for enp131s0f0:
Pre-set maximums:
RX: 8160
RX Mini: n/a
RX Jumbo: n/a
TX: 8160
Current hardware settings:
RX: 2048
RX Mini: n/a
RX Jumbo: n/a
TX: 256
enp131s0f0: flags=4163<UP,BROADCAST,RUNNING,MULTICAST> mtu 9600
inet6 fe80::1757:eed1:dd50:a8cd prefixlen 64 scopeid 0x20<link>
ether 50:7c:6f:3c:35:1a txqueuelen 1000 (Ethernet)
RX packets 3 bytes 738 (738.0 B)
RX errors 0 dropped 0 overruns 0 frame 0
TX packets 2363 bytes 390885 (390.8 KB)
TX errors 0 dropped 0 overruns 0 carrier 0 collisions 0
However, when running srsRAN gNB, I encounter the following issue. Based on information I found online, it seems that this is due to the Intel E810-XXV NIC driver (ice). The ice driver in the kernel lacks support for the OP_SET_RSS_HENA operation. Source link error log:
iavf_execute_vf_cmd(): Cmd 26 not supported
iavf_set_hena(): Failed to execute command of OP_SET_RSS_HENA
iavf_default_rss_disable(): fail to disable default
I wanted to ask if you have encountered similar issues or if there are any methods to resolve this problem? Thanks!
Hey - It seems you're still creating the VF with VLAN tagging enabled. Please disable it:
sudo ip link set enp131s0f0 vf 0 mac 00:11:22:33:44:66 qos 0 spoofchk off mtu 1500.
We'll add an option to disable the tagging inside the OFH layer but right now that's not possible.
Hey - closing the issue as I am assuming it's working now. Please reopen if that's still not the case. Meanwhile we've also added the option to not pack the VLAN tag - simply remove the config values for vlan_tag_cp and vlan_tag_up from the file and they won't be present in the generated Ethernet frame.