srsRAN_4G
srsRAN_4G copied to clipboard
srsran
5G SA: Assertion failed in `./lib/src/rlc/rlc_am_nr.cc:1825`
Issue Description
Assertion failed in ./lib/src/rlc/rlc_am_nr.cc:1825.
Setup Details
5G SA connection between two USRP B210's (with a 30 dB attenuation on the line).
Expected Behavior
No assertion.
Actual Behaviour
srsLog error - The backend queue size is about to reach its maximum capacity of 8192 elements, new log entries will get discarded.
Consider increasing the queue capacity.
RF status: O=0, U=1, L=0
[INFO] [UHD RF] Tx while waiting for EOB, timed out... 1432.22 >= 1432.22. Starting new burst...
RF status: O=0, U=1, L=0
[INFO] [UHD RF] Tx while waiting for EOB, timed out... 1434.32 >= 1434.32. Starting new burst...
RF status: O=0, U=3, L=2
./lib/src/rlc/rlc_am_nr.cc:1825: Assertion Failure: Error: rx_highest_status assigned outside rx window
--- command='srsue ue-uhd.conf' version=22.04.0 signal=6 date='13/06/2022 15:09:16' ---
srsue(+0x273791) [0x55b3be1e5791]
/lib/x86_64-linux-gnu/libc.so.6(+0x3bd60) [0x7f19f6480d60]
/lib/x86_64-linux-gnu/libc.so.6(gsignal+0x141) [0x7f19f6480ce1]
/lib/x86_64-linux-gnu/libc.so.6(abort+0x123) [0x7f19f646a537]
srsue(+0xdea42) [0x55b3be050a42]
srsue(+0x3d5681) [0x55b3be347681]
srsue(+0xdee2b) [0x55b3be050e2b]
srsue(+0xd93db) [0x55b3be04b3db]
srsue(+0xd822d) [0x55b3be04a22d]
srsue(+0x7ce45) [0x55b3bdfeee45]
/lib/x86_64-linux-gnu/libpthread.so.0(+0x8ea7) [0x7f19f6c5cea7]
/lib/x86_64-linux-gnu/libc.so.6(clone+0x3f) [0x7f19f6542def]
srsRAN crashed. Please send this backtrace to the developers ...
--- exiting ---
--- command='srsue ue-uhd.conf' version=22.04.0 signal=11 date='13/06/2022 15:09:16' ---
srsue(+0x273791) [0x55b3be1e5791]
/lib/x86_64-linux-gnu/libc.so.6(+0x3bd60) [0x7f19f6480d60]
srsue(+0xa284b) [0x55b3be01484b]
srsue(+0xa0ec7) [0x55b3be012ec7]
srsue(+0xa10b4) [0x55b3be0130b4]
srsue(+0xa50a9) [0x55b3be0170a9]
srsue(+0x27af89) [0x55b3be1ecf89]
srsue(+0x7ce45) [0x558e57f76e45]
/lib/x86_64-linux-gnu/libpthread.so.0(+0x8ea7) [0x7f9105850ea7]
/lib/x86_64-linux-gnu/libc.so.6(clone+0x3f) [0x7f9105136def]
srsRAN crashed. Please send this backtrace to the developers ...
--- exiting ---
terminate called without an active exception
--- command='srsue ue-uhd.conf' version=22.04.0 signal=6 date='13/06/2022 14:30:46' ---
srsue(+0x273791) [0x558e5816d791]
/lib/x86_64-linux-gnu/libc.so.6(+0x3bd60) [0x7f9105074d60]
/lib/x86_64-linux-gnu/libc.so.6(gsignal+0x141) [0x7f9105074ce1]
/lib/x86_64-linux-gnu/libc.so.6(abort+0x123) [0x7f910505e537]
/lib/x86_64-linux-gnu/libstdc++.so.6(+0x9a7ec) [0x7f91053f67ec]
/lib/x86_64-linux-gnu/libstdc++.so.6(+0xa5966) [0x7f9105401966]
/lib/x86_64-linux-gnu/libstdc++.so.6(+0xa59d1) [0x7f91054019d1]
/lib/x86_64-linux-gnu/libsrsran_rf_uhd.so(+0x17e71) [0x7f9104eebe71]
/lib/x86_64-linux-gnu/libsrsran_rf_uhd.so(+0x1803a) [0x7f9104eec03a]
/lib/x86_64-linux-gnu/libc.so.6(+0x3e4d7) [0x7f91050774d7]
/lib/x86_64-linux-gnu/libc.so.6(+0x3e67a) [0x7f910507767a]
srsue(+0x2737c7) [0x558e5816d7c7]
/lib/x86_64-linux-gnu/libc.so.6(+0x3bd60) [0x7f9105074d60]
srsue(+0x20d235) [0x558e58107235]
srsue(+0xf2938) [0x558e57fec938]
srsue(+0xfbbcd) [0x558e57ff5bcd]
srsue(+0x7ce45) [0x558e57f76e45]
/lib/x86_64-linux-gnu/libpthread.so.0(+0x8ea7) [0x7f9105850ea7]
/lib/x86_64-linux-gnu/libc.so.6(clone+0x3f) [0x7f9105136def]
srsRAN crashed. Please send this backtrace to the developers ...
--- exiting ---
Steps to reproduce the problem
- Start Open5GS
- Start
srsenbwith the supplied configuration files - Start
srsuewith the supplied configuration files - Start
iperf3 -s -Zon the gNB machine - Start
iperf3 -c 10.45.0.1 -Z --bidir -u -b 30.5m -t 3600on the UE machine
Additional Information
enb-uhd.conf.txt rr.conf.txt ue-uhd.conf.txt
Thanks for reporting. Could you give https://github.com/srsran/srsRAN/tree/agpl_next a try please? We've fixed a few RLC issues there already
Upgraded to agpl_next, and got the same/similar assertion:
Random Access Transmission: prach_occasion=0, preamble_index=0, ra-rnti=0xf, tti=7691
Random Access Complete. c-rnti=0x4601, ta=1
RRC Connected
RRC NR reconfiguration successful.
PDU Session Establishment successful. IP: 10.45.0.2
RRC NR reconfiguration successful.
./lib/src/rlc/rlc_am_nr.cc:1854: Assertion Failure: Error: rx_highest_status assigned outside rx window
--- command='srsue ue-uhd.conf' version=22.04.0 signal=6 date='14/06/2022 08:02:08' ---
srsue(+0x272fd1) [0x560f2f35ffd1]
/lib/x86_64-linux-gnu/libc.so.6(+0x3bd60) [0x7fb358d15d60]
/lib/x86_64-linux-gnu/libc.so.6(gsignal+0x141) [0x7fb358d15ce1]
/lib/x86_64-linux-gnu/libc.so.6(abort+0x123) [0x7fb358cff537]
srsue(+0xdef12) [0x560f2f1cbf12]
srsue(+0x3d5f91) [0x560f2f4c2f91]
srsue(+0xdf2fb) [0x560f2f1cc2fb]
srsue(+0xd98ab) [0x560f2f1c68ab]
srsue(+0xd86fd) [0x560f2f1c56fd]
srsue(+0x7d315) [0x560f2f16a315]
/lib/x86_64-linux-gnu/libpthread.so.0(+0x8ea7) [0x7fb3594f1ea7]
/lib/x86_64-linux-gnu/libc.so.6(clone+0x3f) [0x7fb358dd7def]
srsRAN crashed. Please send this backtrace to the developers ...
--- exiting ---
--- command='srsue ue-uhd.conf' version=22.04.0 signal=11 date='14/06/2022 08:02:08' --- [4/1877]
srsue(+0x272fd1) [0x560f2f35ffd1]
/lib/x86_64-linux-gnu/libc.so.6(+0x3bd60) [0x7fb358d15d60]
srsue(+0xa2d1b) [0x560f2f18fd1b]
srsue(+0xa1397) [0x560f2f18e397]
srsue(+0xa1584) [0x560f2f18e584]
srsue(+0xa5579) [0x560f2f192579]
srsue(+0x27a7c9) [0x560f2f3677c9]
srsue(+0x7d315) [0x560f2f16a315]
/lib/x86_64-linux-gnu/libpthread.so.0(+0x8ea7) [0x7fb3594f1ea7]
/lib/x86_64-linux-gnu/libc.so.6(clone+0x3f) [0x7fb358dd7def]
srsRAN crashed. Please send this backtrace to the developers ...
--- exiting ---
--- command='srsue ue-uhd.conf' version=22.04.0 signal=11 date='14/06/2022 08:02:08' ---
srsue(+0x272fd1) [0x560f2f35ffd1]
/lib/x86_64-linux-gnu/libc.so.6(+0x3bd60) [0x7fb358d15d60]
srsue(+0x20c4b5) [0x560f2f2f94b5]
srsue(+0xf2e08) [0x560f2f1dfe08]
srsue(+0xfc09d) [0x560f2f1e909d]
srsue(+0x7d315) [0x560f2f16a315]
/lib/x86_64-linux-gnu/libpthread.so.0(+0x8ea7) [0x7fb3594f1ea7]
/lib/x86_64-linux-gnu/libc.so.6(clone+0x3f) [0x7fb358dd7def]
srsRAN crashed. Please send this backtrace to the developers ...
--- exiting ---
terminate called without an active exception
--- command='srsue ue-uhd.conf' version=22.04.0 signal=6 date='14/06/2022 08:02:08' ---
srsue(+0x272fd1) [0x560f2f35ffd1]
/lib/x86_64-linux-gnu/libc.so.6(+0x3bd60) [0x7fb358d15d60]
/lib/x86_64-linux-gnu/libc.so.6(gsignal+0x141) [0x7fb358d15ce1]
/lib/x86_64-linux-gnu/libc.so.6(abort+0x123) [0x7fb358cff537]
/lib/x86_64-linux-gnu/libstdc++.so.6(+0x9a7ec) [0x7fb3590977ec]
/lib/x86_64-linux-gnu/libstdc++.so.6(+0xa5966) [0x7fb3590a2966]
/lib/x86_64-linux-gnu/libstdc++.so.6(+0xa59d1) [0x7fb3590a29d1]
/lib/x86_64-linux-gnu/libsrsran_rf_uhd.so(+0x17e71) [0x7fb358b8ce71]
/lib/x86_64-linux-gnu/libsrsran_rf_uhd.so(+0x1803a) [0x7fb358b8d03a]
/lib/x86_64-linux-gnu/libc.so.6(+0x3e4d7) [0x7fb358d184d7]
/lib/x86_64-linux-gnu/libc.so.6(+0x3e67a) [0x7fb358d1867a]
srsue(+0x273007) [0x560f2f360007]
/lib/x86_64-linux-gnu/libc.so.6(+0x3bd60) [0x7fb358d15d60]
srsue(+0x20c4b5) [0x560f2f2f94b5]
srsue(+0xf2e08) [0x560f2f1dfe08]
srsue(+0xfc09d) [0x560f2f1e909d]
srsue(+0x7d315) [0x560f2f16a315]
/lib/x86_64-linux-gnu/libpthread.so.0(+0x8ea7) [0x7fb3594f1ea7]
/lib/x86_64-linux-gnu/libc.so.6(clone+0x3f) [0x7fb358dd7def]
srsRAN crashed. Please send this backtrace to the developers ...
--- exiting ---
--- command='srsue ue-uhd.conf' version=22.04.0 signal=11 date='14/06/2022 08:02:08' ---
The assertion came a lot faster though.
Machine 1: Start Open5GS and srsenb (22.04 without updates)
Machine 2: Start srsue (agpl_next)
Machine 1: start iperf3 -s
Machine 2: start iperf3 -c 10.45.0.1 -Z --bidir -u -b 30.5m
The assertion came in the first second.
Running agpl_next on both gNB and UE seems to fix the problem, but I'll keep testing for a while.
The assertion is gone for 5G SA, and the resilience of the LTE connection (2x2 MIMO with PRB=100) is a lot better. Before it would crap out after a while and not recover, now it recovers relatively quickly and goes back to 147 Mbps DL and 48 Mbps UL.
Thanks for confirming the fixes for NR. They were expected. However, for LTE I don't recall any significant changes to the code base. Are you sure you compared 22.04 and the recent agpl_next branch with each other?
It's mostly annecdotal, it seemed to be more stable. I've now had a couple runs where it would stop completely, one where I had an MME failure (reason 0x9 I think?), and some other things. When it works, it has a nice throughput of 147 Mbps DL and 48.9 Mbps UL. I'm not sure why the LTE connection fails, the CPUs are mostly idle(ish).
I'm still seeing this error if I use two x300's to connect. I haven't found the correct gains yet to use with the same bandwidth as the b210's. It seems to be occur when the bandwidth used for iperf3's client is bigger than the bandwidth it can transfer; it occurs more often when I use --bidir, and not so much when I use one direction at a time.
If you can reproduce it with the agpl_next branch with RLC logs in info that would be helpful for us. Thanks
This is a session with two X300's connected with 10 Gbps ethernet to two i9 12900K machines. 5G SA, the two X300's are connected from RXA1-> TXA2, TXA1 -> RXA2, RXB1 -> TXB2, TXB1 -> RXB2; there is a 60 dB attenuation on each line (2 x 30 dB connected serially). I get a throughput of about 6-7 Mbps upstream and 16.5 mbps downstream. On my B210's I get 30.5 Mbps downstream and 18.5 upstream with a similar setup (with 30 dB attenuation on the line, and a different gain setting for ue and enb).
Configuration files used for UE: ue-x300.conf.txt Configuration files used for gNB: enb-x300.conf.txt rb.conf.txt rr.conf.txt
PDU Session Establishment successful. IP: 10.45.0.2 RRC NR reconfiguration successful. RF status: O=0, U=1, L=0 RF status: O=0, U=1, L=0 srsLog error - The backend queue size is about to reach its maximum capacity of 8192 elements, new log entries will get discarded. Consider increasing the queue capacity. RF status: O=0, U=11, L=0 RF status: O=0, U=11, L=0 RF status: O=0, U=9, L=0 RF status: O=0, U=8, L=0 RF status: O=0, U=10, L=0 ./lib/src/rlc/rlc_am_nr.cc:1854: Assertion Failure: Error: rx_highest_status assigned outside rx window --- command='srsue ue-x300.conf' version=22.04.0 signal=6 date='30/06/2022 07:42:50' --- srsue(+0x272fd1) [0x5614adc6efd1] /lib/x86_64-linux-gnu/libc.so.6(+0x3bd60) [0x7fa72e6c5d60] /lib/x86_64-linux-gnu/libc.so.6(gsignal+0x141) [0x7fa72e6c5ce1] /lib/x86_64-linux-gnu/libc.so.6(abort+0x123) [0x7fa72e6af537] srsue(+0xdef12) [0x5614adadaf12] srsue(+0x3d5f91) [0x5614addd1f91] srsue(+0xdf2fb) [0x5614adadb2fb] srsue(+0xd98ab) [0x5614adad58ab] srsue(+0xd86fd) [0x5614adad46fd] srsue(+0x7d315) [0x5614ada79315] /lib/x86_64-linux-gnu/libpthread.so.0(+0x8ea7) [0x7fa72eea1ea7] /lib/x86_64-linux-gnu/libc.so.6(clone+0x3f) [0x7fa72e787def] srsRAN crashed. Please send this backtrace to the developers ... --- exiting --- --- command='srsue ue-x300.conf' version=22.04.0 signal=11 date='30/06/2022 07:42:50' --- srsue(+0x272fd1) [0x5614adc6efd1] /lib/x86_64-linux-gnu/libc.so.6(+0x3bd60) [0x7fa72e6c5d60] srsue(+0x20c4b5) [0x5614adc084b5] srsue(+0xf2e08) [0x5614adaeee08] srsue(+0xfc09d) [0x5614adaf809d] srsue(+0x7d315) [0x5614ada79315] /lib/x86_64-linux-gnu/libpthread.so.0(+0x8ea7) [0x7fa72eea1ea7] /lib/x86_64-linux-gnu/libc.so.6(clone+0x3f) [0x7fa72e787def] srsRAN crashed. Please send this backtrace to the developers ... --- exiting --- terminate called without an active exception --- command='srsue ue-x300.conf' version=22.04.0 signal=6 date='30/06/2022 07:42:50' --- srsue(+0x272fd1) [0x5614adc6efd1] /lib/x86_64-linux-gnu/libc.so.6(+0x3bd60) [0x7fa72e6c5d60] /lib/x86_64-linux-gnu/libc.so.6(gsignal+0x141) [0x7fa72e6c5ce1] /lib/x86_64-linux-gnu/libc.so.6(abort+0x123) [0x7fa72e6af537] /lib/x86_64-linux-gnu/libstdc++.so.6(+0x9a7ec) [0x7fa72ea477ec] /lib/x86_64-linux-gnu/libstdc++.so.6(+0xa5966) [0x7fa72ea52966] /lib/x86_64-linux-gnu/libstdc++.so.6(+0xa59d1) [0x7fa72ea529d1] /lib/x86_64-linux-gnu/libsrsran_rf_uhd.so(+0x15f11) [0x7fa72e55bf11] /lib/x86_64-linux-gnu/libsrsran_rf_uhd.so(+0x15f9a) [0x7fa72e55bf9a] /lib/x86_64-linux-gnu/libc.so.6(__cxa_finalize+0xc6) [0x7fa72e6c8ac6] /lib/x86_64-linux-gnu/libsrsran_rf_uhd.so(+0xc343) [0x7fa72e552343] srsRAN crashed. Please send this backtrace to the developers ... --- exiting --- corrupted double-linked list Segmentation fault
This is with the agpl_next branch from commit 6a3b9257e33948977bedab71ce75e6746d843de2
@andrepuschmann I have the logs available and saved, but they are too large to upload here; I have all_level = info, phy_lib_level = none.
Would you prefer all_level = none and rlc_level = info?
reproduced with all_level = none and rlc_level = info.
bvermeulen@gbodoue02:~$ sudo UHD_IMAGES_DIR=/usr/share/uhd/images srsue ue-x300.conf
Active RF plugins: libsrsran_rf_uhd.so libsrsran_rf_soapy.so libsrsran_rf_blade.so libsrsran_rf_zmq.so
Inactive RF plugins:
Reading configuration file ue-x300.conf...
Built in RelWithDebInfo mode using 22.04.0.
Opening 1 channels in RF device=UHD with args=type=x300,addr=192.168.40.2,sampling_rate=23.04e6,lo_freq_offset_hz=23.04e6
Supported RF device list: UHD soapy bladeRF zmq file
[INFO] [UHD] linux; GNU C++ version 10.2.1 20210110; Boost_107400; UHD_4.1.0.5-3molex1
[INFO] [LOGGING] Fastpath logging disabled at runtime.
Opening USRP channels=1, args: type=x300,addr=192.168.40.2,lo_freq_offset_hz=23.04e6,master_clock_rate=184.32e6
[INFO] [UHD RF] RF UHD Generic instance constructed
[INFO] [X300] X300 initialization sequence...
[INFO] [X300] Maximum frame size: 8000 bytes.
[INFO] [X300] Radio 1x clock: 184.32 MHz
Waiting PHY to initialize ... done!
Attaching UE...
Random Access Transmission: prach_occasion=0, preamble_index=0, ra-rnti=0xf, tti=651
Random Access Complete. c-rnti=0x4601, ta=1
RRC Connected
RRC NR reconfiguration successful.
PDU Session Establishment successful. IP: 10.45.0.2
RRC NR reconfiguration successful.
srsLog error - The backend queue size is about to reach its maximum capacity of 8192 elements, new log entries will get discarded.
Consider increasing the queue capacity.
RF status: O=0, U=7, L=0
RF status: O=0, U=5, L=16
RF status: O=0, U=5, L=0
RF status: O=0, U=5, L=16
RF status: O=0, U=5, L=16
RF status: O=0, U=3, L=32
RF status: O=0, U=6, L=16
./lib/src/rlc/rlc_am_nr.cc:1854: Assertion Failure: Error: rx_highest_status assigned outside rx window
--- command='srsue ue-x300.conf' version=22.04.0 signal=6 date='30/06/2022 08:57:15' ---
srsue(+0x272fd1) [0x55daa6ba7fd1]
/lib/x86_64-linux-gnu/libc.so.6(+0x3bd60) [0x7f3a2d804d60]
/lib/x86_64-linux-gnu/libc.so.6(gsignal+0x141) [0x7f3a2d804ce1]
/lib/x86_64-linux-gnu/libc.so.6(abort+0x123) [0x7f3a2d7ee537]
srsue(+0xdef12) [0x55daa6a13f12]
srsue(+0x3d5f91) [0x55daa6d0af91]
srsue(+0xdf2fb) [0x55daa6a142fb]
srsue(+0xd98ab) [0x55daa6a0e8ab]
srsue(+0xd86fd) [0x55daa6a0d6fd]
srsue(+0x7d315) [0x55daa69b2315]
/lib/x86_64-linux-gnu/libpthread.so.0(+0x8ea7) [0x7f3a2dfe0ea7]
/lib/x86_64-linux-gnu/libc.so.6(clone+0x3f) [0x7f3a2d8c6def]
srsRAN crashed. Please send this backtrace to the developers ...
--- exiting ---
--- command='srsue ue-x300.conf' version=22.04.0 signal=11 date='30/06/2022 08:57:15' --- [62/1769]
srsue(+0x272fd1) [0x55daa6ba7fd1]
/lib/x86_64-linux-gnu/libc.so.6(+0x3bd60) [0x7f3a2d804d60]
srsue(+0x2662d4) [0x55daa6b9b2d4]
srsue(+0x3a84eb) [0x55daa6cdd4eb]
srsue(+0x3daae7) [0x55daa6d0fae7]
srsue(+0x3bbbb1) [0x55daa6cf0bb1]
srsue(+0x3a3e79) [0x55daa6cd8e79]
srsue(+0x267b75) [0x55daa6b9cb75]
srsue(+0x26d5c3) [0x55daa6ba25c3]
srsue(+0x26daf7) [0x55daa6ba2af7]
srsue(+0x25cf1b) [0x55daa6b91f1b]
srsue(+0x9ecf5) [0x55daa69d3cf5]
srsue(+0xa5789) [0x55daa69da789]
srsue(+0x27a7c9) [0x55daa6baf7c9]
srsue(+0x7d315) [0x55daa69b2315]
/lib/x86_64-linux-gnu/libpthread.so.0(+0x8ea7) [0x7f3a2dfe0ea7]
/lib/x86_64-linux-gnu/libc.so.6(clone+0x3f) [0x7f3a2d8c6def]
srsRAN crashed. Please send this backtrace to the developers ...
--- exiting ---
terminate called without an active exception [42/1769]
--- command='srsue ue-x300.conf' version=22.04.0 signal=6 date='30/06/2022 08:57:15' ---
srsue(+0x272fd1) [0x55daa6ba7fd1]
/lib/x86_64-linux-gnu/libc.so.6(+0x3bd60) [0x7f3a2d804d60]
/lib/x86_64-linux-gnu/libc.so.6(gsignal+0x141) [0x7f3a2d804ce1]
/lib/x86_64-linux-gnu/libc.so.6(abort+0x123) [0x7f3a2d7ee537]
/lib/x86_64-linux-gnu/libstdc++.so.6(+0x9a7ec) [0x7f3a2db867ec]
/lib/x86_64-linux-gnu/libstdc++.so.6(+0xa5966) [0x7f3a2db91966]
/lib/x86_64-linux-gnu/libstdc++.so.6(+0xa59d1) [0x7f3a2db919d1]
/lib/x86_64-linux-gnu/libsrsran_rf_uhd.so(+0x15f11) [0x7f3a2d69af11]
/lib/x86_64-linux-gnu/libsrsran_rf_uhd.so(+0x15f9a) [0x7f3a2d69af9a]
/lib/x86_64-linux-gnu/libc.so.6(__cxa_finalize+0xc6) [0x7f3a2d807ac6]
/lib/x86_64-linux-gnu/libsrsran_rf_uhd.so(+0xc343) [0x7f3a2d691343]
srsRAN crashed. Please send this backtrace to the developers ...
--- exiting ---
Segmentation fault
ue-x300-rlc.1.log ue-x300-rlc.2.log ue-x300-rlc.log
I have the srsenb logs available as well with the same log settings if you need them.
I saw at least one instance of mac_sch_pdu_nr::to_string() writing one DL and loads of LCID=0 len=0 from what seems to be a pdu with lots of empty subpdu's. I'll see if I can catch it again and upload the logs (these logs were 100MB per file, which I can't upload here).
I saw a similar backtrace on the enb, but wasn't able to copy it. I'm including the log files here:
enb-uhd.6700.log enb-uhd.6701.log enb-uhd.6702.log enb-uhd.6703.log enb-uhd.6704.log enb-uhd.6705.log enb-uhd.6706.log enb-uhd.6707.log
bvermeulen@gbodoue02:~$ sudo UHD_LOG_FILE=/home/bvermeulen/uhd.log UHD_IMAGES_DIR=/usr/share/uhd/images srsue ue-b210.conf
[sudo] password for bvermeulen:
Active RF plugins: libsrsran_rf_uhd.so libsrsran_rf_soapy.so libsrsran_rf_blade.so libsrsran_rf_zmq.so
Inactive RF plugins:
Reading configuration file ue-b210.conf...
Built in RelWithDebInfo mode using 22.04.0.
Opening 1 channels in RF device=UHD with args=type=b200,serial=3166D07,sampling_rate=23.04e6,lo_freq_offset_hz=23.04e6
Supported RF device list: UHD soapy bladeRF zmq file
[INFO] [UHD] linux; GNU C++ version 10.2.1 20210110; Boost_107400; UHD_4.1.0.5-3molex1
[INFO] [LOGGING] Fastpath logging disabled at runtime.
Opening USRP channels=1, args: type=b200,serial=3166D07,lo_freq_offset_hz=23.04e6,master_clock_rate=23.04e6
[INFO] [UHD RF] RF UHD Generic instance constructed
[INFO] [B200] Detected Device: B210
[INFO] [B200] Operating over USB 3.
[INFO] [B200] Detecting internal GPSDO....
[INFO] [GPS] Found an internal GPSDO: GPSTCXO , Firmware Rev 0.929a
[INFO] [B200] Initialize CODEC control...
[INFO] [B200] Initialize Radio control...
[INFO] [B200] Performing register loopback test...
[INFO] [B200] Register loopback test passed
[INFO] [B200] Performing register loopback test...
[INFO] [B200] Register loopback test passed
[INFO] [B200] Asking for clock rate 23.040000 MHz...
[INFO] [B200] Actually got clock rate 23.040000 MHz.
Waiting PHY to initialize ... done!
Attaching UE...
Random Access Transmission: prach_occasion=0, preamble_index=0, ra-rnti=0xf, tti=2251
Random Access Complete. c-rnti=0x4601, ta=1
RRC Connected
RRC NR reconfiguration successful.
PDU Session Establishment successful. IP: 10.45.0.2
RRC NR reconfiguration successful.
srsLog error - The backend queue size is about to reach its maximum capacity of 8192 elements, new log entries will get discarded.
Consider increasing the queue capacity.
RF status: O=0, U=1, L=0
[INFO] [UHD RF] Tx while waiting for EOB, timed out... 2373.72 >= 2373.72. Starting new burst...
RF status: O=0, U=3, L=5
RF status: O=0, U=1, L=0
./lib/src/rlc/rlc_am_nr.cc:1854: Assertion Failure: Error: rx_highest_status assigned outside rx window
--- command='srsue ue-b210.conf' version=22.04.0 signal=6 date='01/07/2022 12:21:29' ---
srsue(+0x272fd1) [0x55e51ca07fd1]
/lib/x86_64-linux-gnu/libc.so.6(+0x3bd60) [0x7f67f289ad60]
/lib/x86_64-linux-gnu/libc.so.6(gsignal+0x141) [0x7f67f289ace1]
/lib/x86_64-linux-gnu/libc.so.6(abort+0x123) [0x7f67f2884537]
srsue(+0xdef12) [0x55e51c873f12]
srsue(+0x3d5f91) [0x55e51cb6af91]
srsue(+0xdf2fb) [0x55e51c8742fb]
srsue(+0xd98ab) [0x55e51c86e8ab]
srsue(+0xd86fd) [0x55e51c86d6fd]
srsue(+0x7d315) [0x55e51c812315]
/lib/x86_64-linux-gnu/libpthread.so.0(+0x8ea7) [0x7f67f3076ea7]
/lib/x86_64-linux-gnu/libc.so.6(clone+0x3f) [0x7f67f295cdef]
srsRAN crashed. Please send this backtrace to the developers ...
--- exiting ---
--- command='srsue ue-b210.conf' version=22.04.0 signal=11 date='01/07/2022 12:21:29' ---
srsue(+0x272fd1) [0x55e51ca07fd1]
/lib/x86_64-linux-gnu/libc.so.6(+0x3bd60) [0x7f67f289ad60]
srsue(+0x3a3e73) [0x55e51cb38e73]
srsue(+0x267b75) [0x55e51c9fcb75]
srsue(+0x26d5c3) [0x55e51ca025c3]
srsue(+0x26daf7) [0x55e51ca02af7]
srsue(+0x25cf1b) [0x55e51c9f1f1b]
srsue(+0x9ecf5) [0x55e51c833cf5]
srsue(+0xa5789) [0x55e51c83a789]
srsue(+0x27a7c9) [0x55e51ca0f7c9]
srsue(+0x7d315) [0x55e51c812315]
/lib/x86_64-linux-gnu/libpthread.so.0(+0x8ea7) [0x7f67f3076ea7]
/lib/x86_64-linux-gnu/libc.so.6(clone+0x3f) [0x7f67f295cdef]
srsRAN crashed. Please send this backtrace to the developers ...
--- exiting ---
--- command='srsue ue-b210.conf' version=22.04.0 signal=11 date='01/07/2022 12:21:29' ---
srsue(+0x272fd1) [0x55e51ca07fd1]
/lib/x86_64-linux-gnu/libc.so.6(+0x3bd60) [0x7f67f289ad60]
srsue(+0x99b2e0) [0x55e51d1302e0]
srsRAN crashed. Please send this backtrace to the developers ...
--- exiting ---
terminate called without an active exception
--- command='srsue ue-b210.conf' version=22.04.0 signal=6 date='01/07/2022 12:21:29' ---
srsue(+0x272fd1) [0x55e51ca07fd1]
/lib/x86_64-linux-gnu/libc.so.6(+0x3bd60) [0x7f67f289ad60]
/lib/x86_64-linux-gnu/libc.so.6(gsignal+0x141) [0x7f67f289ace1]
/lib/x86_64-linux-gnu/libc.so.6(abort+0x123) [0x7f67f2884537]
/lib/x86_64-linux-gnu/libstdc++.so.6(+0x9a7ec) [0x7f67f2c1c7ec]
/lib/x86_64-linux-gnu/libstdc++.so.6(+0xa5966) [0x7f67f2c27966]
/lib/x86_64-linux-gnu/libstdc++.so.6(+0xa59d1) [0x7f67f2c279d1]
/lib/x86_64-linux-gnu/libsrsran_rf_uhd.so(+0x15f11) [0x7f67f2730f11]
/lib/x86_64-linux-gnu/libsrsran_rf_uhd.so(+0x15f9a) [0x7f67f2730f9a]
/lib/x86_64-linux-gnu/libc.so.6(__cxa_finalize+0xc6) [0x7f67f289dac6]
/lib/x86_64-linux-gnu/libsrsran_rf_uhd.so(+0xc343) [0x7f67f2727343]
srsRAN crashed. Please send this backtrace to the developers ...
--- exiting ---
corrupted double-linked list
Segmentation fault
bvermeulen@gbodoue02:~$
The corresponding log files:
ue-b210.2730.log ue-b210.2731.log ue-b210.2732.log ue-b210.2733.log ue-b210.2734.log ue-b210.2735.log
ue-b210.2732.log contains a line with hundreds of LCID=0 len=0 elements (subpdu's with 0 size) that I'm fairly sure isn't correct.
I was running iperf3 -s on the gNB machine, and iperf3 -c 10.45.0.1 -Z -u -b 30.5m --bidir -t 3600 on the UE machine.
If you need more logs, I have the 2400 seconds available (from ue-b210.log to ue-b210.2735.log).
Hi @TallGuy74 --
Could you try to replicate the issue with the RLC logs at debug and everything else at none?
I'm trying to understand the issue with the logs you have sent already in the meantime.
@alvasMan Reproduced on the X300 to X300 setup (direct cable connection between RX/TX on both, with a 60 dB attenuation on the line; the configuration files are attached earlier in the issue). I ran iperf3 -s on the gNB machine, and iperf3 -c 10.45.0.1 -Z -u -b 20m --bidir on the UE machine. This crashed within 4 seconds (with this log level).
ue-x300-rlc.1.log ue-x300-rlc.2.log ue-x300-rlc.3.log ue-x300-rlc.4.log ue-x300-rlc.5.log ue-x300-rlc.6.log ue-x300-rlc.7.log ue-x300-rlc.8.log ue-x300-rlc.9.log ue-x300-rlc.10.log ue-x300-rlc.11.log ue-x300-rlc.12.log ue-x300-rlc.13.log ue-x300-rlc.14.log ue-x300-rlc.15.log ue-x300-rlc.16.log ue-x300-rlc.17.log ue-x300-rlc.18.log ue-x300-rlc.19.log ue-x300-rlc.20.log ue-x300-rlc.log
The issue is caused when t-Reassembly timer expires Last sequentially received SN was 407 (Rx_Next=407), see the following lines:
2022-07-04T07:47:31.939306 [RLC-NR ] [D] DRB1: Reassembly timer expired after 50ms
2022-07-04T07:47:31.939307 [RLC-NR ] [E] DRB1: Rx_Highest_Status not inside RX window
2022-07-04T07:47:31.939307 [RLC-NR ] [D] DRB1: RX entity state: Rx_Next=407, Rx_Next_Status_Trigger=2453, Rx_Highest_Status=2455, Rx_Next_Highest=2455
Looking back on the logs, I can see 407 was never received. The the first NACK we send for it seems to be here, on ue-x300-rlc.2.log:
2022-07-04T07:47:28.466705 [RLC-NR ] [D] DRB1: Adding NACK for full SDU. NACK SN=407
So a couple of questions here: 1 - Why did the UE not retx SN=407? 2 - Why did Rx Highest Status grew so big that it's outside of the RX window?
Let's leave 1) for now and focus on 2).
The code to update Rx_Highest_Status when t-Reassembly expires is the following:
/*
* 5.2.3.2.4 Actions when t-Reassembly expires:
* - update RX_Highest_Status to the SN of the first RLC SDU with SN >= RX_Next_Status_Trigger for which not
* all bytes have been received;
* - if RX_Next_Highest> RX_Highest_Status +1: or
* - if RX_Next_Highest = RX_Highest_Status + 1 and there is at least one missing byte segment of the SDU
* associated with SN = RX_Highest_Status before the last byte of all received segments of this SDU:
* - start t-Reassembly;
* - set RX_Next_Status_Trigger to RX_Next_Highest.
*/
uint32_t sn_upd = {};
for (sn_upd = st.rx_next_status_trigger; rx_mod_base_nr(sn_upd) < rx_mod_base_nr(st.rx_next_highest);
sn_upd = (sn_upd + 1) % mod_nr) {
if (not rx_window->has_sn(sn_upd) || (rx_window->has_sn(sn_upd) && not(*rx_window)[sn_upd].fully_received)) {
break;
}
}
st.rx_highest_status = sn_upd;
if (not inside_rx_window(st.rx_highest_status)) {
RlcError("Rx_Highest_Status not inside RX window");
debug_state();
}
srsran_assert(inside_rx_window(st.rx_highest_status), "Error: rx_highest_status assigned outside rx window");
Considering we are using 12 bits SN, thus a window of 2048, the end of the window should be 407+2048=2455. Which is exactly RX_Highest_Status.
This makes sense since we received PDUs up until the end of the RX widow and we don't check for the full window in that loop. I'm currently trying to think what is the best solution for this.
Do you have the the matching logs of the UE, so we can try to figure out the issue on the TX side?
@alvasMan I can reproduce it really easily, so if you want I can get you matching ue and enb logs for this. What log levels would you want to have for the enb?
Currently I'm just focusing on the RLC, those at debug. The rest can be at warning, at least for now. If you can reproduce this easily we can always increase the level of the lower layers later if we need it.
UE logs (multipart zip):
ue-x300.zip ue-x300.z01 ue-x300.z02
ENB logs (multipart zip): enb-x300.1.zip enb-x300.2.zip enb-x300.3.zip enb-x300.zip
both with rlc_level = debug, all_level = none
Could you please just have a quick try at this: rlc_rx_highest.txt
And let me know if the crash stops at least?
@alvasMan The crash stops, but the software also stops passing data.
UE logs: ue-x300-rlc-bugfix-log.z01 ue-x300-rlc-bugfix-log.z02 ue-x300-rlc-bugfix-log.z03 ue-x300-rlc-bugfix-log.z04 ue-x300-rlc-bugfix-log.z05 ue-x300-rlc-bugfix-log.z06 ue-x300-rlc-bugfix-log.z07 ue-x300-rlc-bugfix-log.z08 ue-x300-rlc-bugfix-log.z09 ue-x300-rlc-bugfix-log.z10 ue-x300-rlc-bugfix-log.z11 ue-x300-rlc-bugfix-log.z12 ue-x300-rlc-bugfix-log.z13 ue-x300-rlc-bugfix-log.z14 ue-x300-rlc-bugfix-log.z15 ue-x300-rlc-bugfix-log.z16 ue-x300-rlc-bugfix-log.z17 ue-x300-rlc-bugfix-log.z18 ue-x300-rlc-bugfix-log.z19 ue-x300-rlc-bugfix-log.z20 ue-x300-rlc-bugfix-log.z21 ue-x300-rlc-bugfix-log.z22 ue-x300-rlc-bugfix-log.z23 ue-x300-rlc-bugfix-log.z24 ue-x300-rlc-bugfix-log.z25 ue-x300-rlc-bugfix-log.z26 ue-x300-rlc-bugfix-log.z27 ue-x300-rlc-bugfix-log.z28 ue-x300-rlc-bugfix-log.zip
ENB logs: enb-x300-rlc-bugfix-log.z01 enb-x300-rlc-bugfix-log.z02 enb-x300-rlc-bugfix-log.z03 enb-x300-rlc-bugfix-log.zip
That is somewhat expected though. The RX window was getting full, so PDUs will drop when it is full. I can see in the logs you sent that that seems to be the issue:
./ue-x300-rlc.100.log:2022-07-05T07:00:51.377687 [RLC-NR ] [I] DRB1: SN=515 outside rx window [2337:4385] - discarding
./ue-x300-rlc.100.log:2022-07-05T07:00:51.377688 [RLC-NR ] [I] DRB1: SN=516 outside rx window [2337:4385] - discarding
./ue-x300-rlc.100.log:2022-07-05T07:00:51.377689 [RLC-NR ] [I] DRB1: SN=517 outside rx window [2337:4385] - discarding
./ue-x300-rlc.100.log:2022-07-05T07:00:51.377690 [RLC-NR ] [I] DRB1: SN=518 outside rx window [2337:4385] - discarding
./ue-x300-rlc.100.log:2022-07-05T07:00:51.399444 [RLC-NR ] [I] DRB1: SN=518 outside rx window [2337:4385] - discarding
./ue-x300-rlc.100.log:2022-07-05T07:00:51.404157 [RLC-NR ] [I] DRB1: SN=521 outside rx window [2337:4385] - discarding
./ue-x300-rlc.100.log:2022-07-05T07:00:51.406471 [RLC-NR ] [I] DRB1: SN=524 outside rx window [2337:4385] - discarding
./ue-x300-rlc.100.log:2022-07-05T07:00:51.406472 [RLC-NR ] [I] DRB1: SN=525 outside rx window [2337:4385] - discarding
./ue-x300-rlc.100.log:2022-07-05T07:00:51.406473 [RLC-NR ] [I] DRB1: SN=526 outside rx window [2337:4385] - discarding
./ue-x300-rlc.100.log:2022-07-05T07:00:51.406474 [RLC-NR ] [I] DRB1: SN=527 outside rx window [2337:4385] - discarding
./ue-x300-rlc.100.log:2022-07-05T07:00:51.408302 [RLC-NR ] [I] DRB1: SN=527 outside rx window [2337:4385] - discarding
./ue-x300-rlc.100.log:2022-07-05T07:00:51.408303 [RLC-NR ] [I] DRB1: SN=528 outside rx window [2337:4385] - discarding
./ue-x300-rlc.100.log:2022-07-05T07:00:51.408304 [RLC-NR ] [I] DRB1: SN=529 outside rx window [2337:4385] - discarding
./ue-x300-rlc.100.log:2022-07-05T07:00:51.428371 [RLC-NR ] [I] DRB1: SN=541 outside rx window [2337:4385] - discarding
./ue-x300-rlc.100.log:2022-07-05T07:00:51.428373 [RLC-NR ] [I] DRB1: SN=542 outside rx window [2337:4385] - discarding
Currently investigating what is causing this.
It even stops passing data if I stop iperf3 and start it up again after a little while. I'd expect the dropped PDUs to impact my throughput (as those will probably be retransmitted?), but not drop to 0.
It even stops passing data if I stop iperf3 and start it up again after a little while. I'd expect the dropped PDUs to impact my throughput (as those will probably be re-transmitted?), but not drop to 0.
Sorry, perhaps I was not clear.
I did not mean to say that there was no bug here. I meant to say that I expected such a data stall bug, since the RLC stopped re-transmitting packets in the previous logs you sent.
I had a look at the most recent logs you sent, and basically the TX window is getting overrun. See the following entry, where a NACK is not being processed, even though there was no ACK to remove that SN:
2022-07-05T07:00:41.371339 [RLC-NR ] [I] DRB2: TX window does not contain NACK_SN. SDU SN=2337, Tx_Next_Ack=2337, Tx_Next=312
There is currently no check for the TX window getting overrun in the agpl_next branch, can you try adding it by doing the following:
diff --git a/lib/src/rlc/rlc_am_nr.cc b/lib/src/rlc/rlc_am_nr.cc
index 99b0d5bcc..191d2fe08 100644
--- a/lib/src/rlc/rlc_am_nr.cc
+++ b/lib/src/rlc/rlc_am_nr.cc
@@ -184,6 +184,13 @@ uint32_t rlc_am_nr_tx::build_new_pdu(uint8_t* payload, uint32_t nof_bytes)
RlcInfo("Not enough bytes for payload plus header. nof_bytes=%d", nof_bytes);
return 0;
}
+
+ // do not build any more PDUs if window is already full
+ if (tx_window->full()) {
+ RlcInfo("Cannot build data PDU - Tx window full.");
+ return 0;
+ }
+
// Read new SDU from TX queue
unique_byte_buffer_t tx_sdu;
RlcDebug("Reading from RLC SDU queue. Queue size %d", tx_sdu_queue.size());
(END)
Let me know if that helps with the data stall.
@alvasMan: The data stall is still there:
UE logs: ue-x300-txwindow.z01 ue-x300-txwindow.zip
ENB logs: enb-x300-txwindow.zip
Some more information:
In this situation I have X300-1 connected to the ENB machine, and X300-2 connected to the UE machine. If I connect X300-1 to the UE machine, and X300-2 to the ENB machine, I have no problem and I get ~30 Mbps DL and ~19 Mbps UL. The physical connection stays the same, I only change the 10 Gbps ethernet connection from one PC to the other and vice-versa.
I really don't know why it does that, if you have any idea on what could cause this phenomenon, that would help immensely.
I am keeping the setup in the fault sensitive configuration so we can continue testing this for now.
The X300's don't have a GPSDO module installed, and according to Ettus this could cause the problem that I am seeing with the setup working if reversed.
The X300's don't have a GPSDO module installed, and according to Ettus this could cause the problem that I am seeing with the setup working if reversed.
Yes, that's quite possible. The X300 have quite good local oscillators already (compared to the B210). But having a GPSDO or external reference if definitely better.
I had a look at the logs that you sent me and I spotted two issues.
1 - When t-PollRetransmit expired, the re-transmitted PDU was being dropped because of a check if the PDU was inside the RX window at the RX. This was done before checking if p bit was set and thus it was not triggering the status report and causing the data stall.
2 - I saw at least one instance of a status report being dropped at the TX, because the ACK_SN > TX_NEXT. This is actually possible to happen, if we get a NACK for an SDU segment of an SDU that has not finished transmitting all its segments. Probably this was not causing you data stall, but nasty corner case none the less.
Could you try the following for issue 1):
diff --git a/lib/src/rlc/rlc_am_nr.cc b/lib/src/rlc/rlc_am_nr.cc
index 6acd14acc4..da27b25875 100644
--- a/lib/src/rlc/rlc_am_nr.cc
+++ b/lib/src/rlc/rlc_am_nr.cc
@@ -1425,21 +1425,25 @@ void rlc_am_nr_rx::handle_data_pdu(uint8_t* payload, uint32_t nof_bytes)
RlcHexInfo(payload, nof_bytes, "Rx data PDU SN=%d (%d B)", header.sn, nof_bytes);
log_rlc_am_nr_pdu_header_to_string(logger.debug, header, rb_name);
- // Check whether SDU is within Rx Window
- if (!inside_rx_window(header.sn)) {
- RlcInfo("SN=%d outside rx window [%d:%d] - discarding", header.sn, st.rx_next, st.rx_next + rx_window_size());
- return;
- }
-
// Trigger polling if poll bit is set.
- // We do this before discarding duplicate SDUs/SDU segments
+ // We do this before checking if the PDU is inside the RX window,
+ // as the RX window may have advanced without the TX having received the ACKs
+ // This can cause a data stall, whereby the TX keeps retransmiting
+ // a PDU outside of the Rx window.
+ // Also, we do this before discarding duplicate SDUs/SDU segments
// Because t-PollRetransmit may transmit a PDU that was already
// received.
- if (header.p) {
+ if (header.p != 0U) {
RlcInfo("status packet requested through polling bit");
do_status = true;
}
+ // Check whether SDU is within Rx Window
+ if (!inside_rx_window(header.sn)) {
+ RlcInfo("SN=%d outside rx window [%d:%d] - discarding", header.sn, st.rx_next, st.rx_next + rx_window_size());
+ return;
+ }
+
// Section 5.2.3.2.2, discard duplicate PDUs
if (rx_window->has_sn(header.sn) && (*rx_window)[header.sn].fully_received) {
RlcInfo("discarding duplicate SN=%d", header.sn);
And this for issue 2):
diff --git a/lib/src/rlc/rlc_am_nr.cc b/lib/src/rlc/rlc_am_nr.cc
index df58921572..6acd14acc4 100644
--- a/lib/src/rlc/rlc_am_nr.cc
+++ b/lib/src/rlc/rlc_am_nr.cc
@@ -794,7 +794,7 @@ void rlc_am_nr_tx::handle_control_pdu(uint8_t* payload, uint32_t nof_bytes)
info_state();
return;
}
- if (tx_mod_base_nr(status.ack_sn) > tx_mod_base_nr(st.tx_next)) {
+ if (tx_mod_base_nr(status.ack_sn) > tx_mod_base_nr(st.tx_next + 1)) {
RlcWarning("Received ACK with SN larger than TX_NEXT, ignoring status report. SN=%d, TX_NEXT_ACK=%d, TX_NEXT=%d",
status.ack_sn,
st.tx_next_ack,
@alvasMan Logs for this (rlc_level = debug, the rest is none):
UE: ue-x300-issue2-log.z01 ue-x300-issue2-log.z02 ue-x300-issue2-log.z03 ue-x300-issue2-log.z04 ue-x300-issue2-log.zip
ENB: enb-x300-issue-log.z01 enb-x300-issue-log.z02 enb-x300-issue-log.z03 enb-x300-issue-log.zip
Strange... I see the following loop at the end of the logs:
@UE
2022-07-07T15:34:49.980721 [RLC-NR ] [I] DRB1: status packet requested through polling bit
2022-07-07T15:34:49.980721 [RLC-NR ] [I] DRB1: SN=2013 outside rx window [2018:4066] - discarding
...
2022-07-07T15:34:49.981540 [RLC-NR ] [D] DRB1: buffer state - do_status=yes
2022-07-07T15:34:49.981541 [RLC-NR ] [D] DRB1: Generating status PDU
2022-07-07T15:34:49.981542 [RLC-NR ] [D] DRB1: Adding NACKs for segmented SDU. NACK SN=2018
2022-07-07T15:34:49.981542 [RLC-NR ] [D] DRB1: Final segment missing. NACK_SN=2018. SO_start=1351, SO_end=65535
@eNB
2022-07-07T15:34:48.615912 [RLC-NR ] [I] RX status PDU: ACK_SN = 4061, N_nack = 69, NACK_SN = [2018 1351:1236 r4][2050 594:486 r4][2176 1237:1022 r7][2185 916:808 r4][2191 702:588 r4][2197 482:831 r65][2287 1355:1140 r7][2296 1034:926 r4][2302 820:712 r4][2308 593:1470 r50][2363 1257:935 r10][2398 1459:1244 r7][2407 1138:1030 r4][2413 924:816 r4][2419 710:1146 r62][2486 933:718 r7][2501 398:290 r4][2510 79:1134 r12][2530 814:120 r54][2591 1030:922 r4][2603 602:494 r4][2617 1298:1190 r4][2623 1084:869 r7][2641 409:868 r47][2696 293:80 r7][2728 602:387 r7][2751 1232:382 r57][2813 170:1440 r6][2842 585:477 r4][2851 264:51 r7][2862 1275:746 r48][2918 372:52 r10][2929 1430:1322 r4][2941 1002:894 r4][2947 788:680 r4][2962 253:146 r4][2970 1418:1353 r83][3058 1140:925 r7][3079 391:1283 r56][3137 1177:855 r10][3193 631:97 r48][3248 1261:1046 r7][3278 192:85 r4][3301 821:66 r54][3359 1337:1122 r7][3397 1432:1217 r7][3415 790:239 r48][3470 1405:1190 r7][3494 549:441 r4][3503 229:347 r68][3576 136:29 r4][3584 1045:937 r4][3602 403:41 r4][3634 387:1356 r53][3692 1143:928 r7][3701 822:714 r4][3739 917:809 r4][3748 596:1405 r50][3803 1192:870 r10][3821 550:442 r4][3859 646:1087 r53][3914 726:618 r4][3920 512:297 r7][3934 1463:1355 r4][3964 393:1080 r53][4025 505:290 r7][4034 185:78 r4][4042 1349:986 r4][4048 880:772 r4]
2022-07-07T15:34:48.615912 [RLC-NR ] [I] DRB2: Received ACK with SN outside of TX_WINDOW, ignoring status report. ACK_SN=4061, TX_NEXT_ACK=2013.
Do you have the following snippet applied? On both the eNB and UE? Could you just double check if both the eNB and the UE have all the patches I've sent you please?
if (not valid_ack_sn(status.ack_sn)) {
RlcInfo("Received ACK with SN outside of TX_WINDOW, ignoring status report. ACK_SN=%d, TX_NEXT_ACK=%d.",
status.ack_sn,
st.tx_next_ack);
info_state();
return;
}
2013 + 2048 = 4061, so that ACK_SN should have been accepted.