srsRAN_4G icon indicating copy to clipboard operation
srsRAN_4G copied to clipboard

Published 20 hours ago • verified publisher icon srsran

Incorrect IMSI handling within SPGW

Open ECTPavloR opened this issue 5 years ago • 19 comments
trafficstars

Hello!

During the same setup as for issue #585 , found another unexpected behavior: SPGW allocating IP address (correct static address as per user_db.csv) to IMSI 000000000000000 instead of correct IMSI 255172020080007. This leads to further Failed to send S1AP PDU. Error: Broken pipe

It looks like an SPGW issue since at some point all the connections with other IMSIs turning into this behavior (see detailed log below).

Behavior close to the issue here: http://www.softwareradiosystems.com/pipermail/srslte-users/2020-May/003544.html (but I didn't find any traces of resolution)

Setup Details

srslte 20.04 25UEs (based on Quectel modems, LTE cat1) BladeRF frontend

Expected Behavior

assigning an address to a real IMSI instead of 000000000000000 and correct S1AP reply.

Actual Behaviour

Creating Session Response -- IMSI: 255172020080007 Creating Session Response -- MME control TEID: 493 Received GTP-C PDU. Message type: GTPC_MSG_TYPE_CREATE_SESSION_REQUEST SPGW: GTP-C context for IMSI 255172020080007 already exists. SPGW: Deleted previous context. SPGW: Allocated Ctrl TEID 493 SPGW: Allocated User TEID 493 SPGW: Allocate UE IP 172.16.0.107 Received Create Session Response Create Session Response -- SPGW control TEID 493 Create Session Response -- SPGW S1-U Address: 127.0.1.100 SPGW Allocated IP 172.16.0.107 to IMSI 000000000000000 Adding attach accept to Initial Context Setup Request Failed to send S1AP PDU. Error: Broken pipe

Steps to reproduce the problem

Running few UEs with connect/disconnect behavior for a long time. In our case 25-26 UEs for ~2-3h. So far didn't catch the exact point after which it turns to abnormal behavior.

Additional Information

Complete log for a couple of UEs/IMSIs

Sending Create Session Request. Creating Session Response -- IMSI: 255172020080007 Creating Session Response -- MME control TEID: 493 Received GTP-C PDU. Message type: GTPC_MSG_TYPE_CREATE_SESSION_REQUEST SPGW: GTP-C context for IMSI 255172020080007 already exists. SPGW: Deleted previous context. SPGW: Allocated Ctrl TEID 493 SPGW: Allocated User TEID 493 SPGW: Allocate UE IP 172.16.0.107 Received Create Session Response Create Session Response -- SPGW control TEID 493 Create Session Response -- SPGW S1-U Address: 127.0.1.100 SPGW Allocated IP 172.16.0.107 to IMSI 000000000000000 Adding attach accept to Initial Context Setup Request Failed to send S1AP PDU. Error: Broken pipe Received UE Context Release Request. MME-UE S1AP Id 1560 UE is not ECM connected. No need to release S1-U. MME UE S1AP Id 1560 Received UE Context Release Request. MME-UE S1AP Id 1561 UE is not ECM connected. No need to release S1-U. MME UE S1AP Id 1561 Initial UE message: LIBLTE_MME_MSG_TYPE_ATTACH_REQUEST Received Initial UE message -- Attach Request Attach request -- IMSI: 255172020080029 Attach request -- eNB-UE S1AP Id: 113 Attach request -- Attach type: 2 Attach Request -- UE Network Capabilities EEA: 11110000 Attach Request -- UE Network Capabilities EIA: 01110000 Attach Request -- MS Network Capabilities Present: true PDN Connectivity Request -- EPS Bearer Identity requested: 0 PDN Connectivity Request -- Procedure Transaction Id: 1 PDN Connectivity Request -- ESM Information Transfer requested: true Attach Request -- Found previously attach UE. Downlink NAS: Sending Authentication Request UL NAS: Received Authentication Response Authentication Response -- IMSI 255172020080029 UE Authentication Accepted. Generating KeNB with UL NAS COUNT: 0 Downlink NAS: Sending NAS Security Mode Command. UL NAS: Received Security Mode Complete Security Mode Command Complete -- IMSI: 255172020080029 Sending ESM information request UL NAS: Received ESM Information Response ESM Info: APN internet Getting subscription information -- QCI 7 Sending Create Session Request. Creating Session Response -- IMSI: 255172020080029 Creating Session Response -- MME control TEID: 494 Received GTP-C PDU. Message type: GTPC_MSG_TYPE_CREATE_SESSION_REQUEST SPGW: GTP-C context for IMSI 255172020080029 already exists. SPGW: Deleted previous context. SPGW: Allocated Ctrl TEID 494 SPGW: Allocated User TEID 494 SPGW: Allocate UE IP 172.16.0.129 Received Create Session Response Create Session Response -- SPGW control TEID 494 Create Session Response -- SPGW S1-U Address: 127.0.1.100 SPGW Allocated IP 172.16.0.129 to IMSI 000000000000000 Adding attach accept to Initial Context Setup Request Failed to send S1AP PDU. Error: Broken pipe Received UE Context Release Request. MME-UE S1AP Id 1562 UE is not ECM connected. No need to release S1-U. MME UE S1AP Id 1562 Initial UE message: LIBLTE_MME_MSG_TYPE_ATTACH_REQUEST Received Initial UE message -- Attach Request Attach request -- IMSI: 255172020080033 Attach request -- eNB-UE S1AP Id: 114 Attach request -- Attach type: 2 Attach Request -- UE Network Capabilities EEA: 11110000 Attach Request -- UE Network Capabilities EIA: 01110000 Attach Request -- MS Network Capabilities Present: true PDN Connectivity Request -- EPS Bearer Identity requested: 0 PDN Connectivity Request -- Procedure Transaction Id: 1 PDN Connectivity Request -- ESM Information Transfer requested: true Attach Request -- Found previously attach UE. Downlink NAS: Sending Authentication Request UL NAS: Received Authentication Response Authentication Response -- IMSI 255172020080033 UE Authentication Accepted. Generating KeNB with UL NAS COUNT: 0 Downlink NAS: Sending NAS Security Mode Command. UL NAS: Received Security Mode Complete Security Mode Command Complete -- IMSI: 255172020080033 Sending ESM information request UL NAS: Received ESM Information Response ESM Info: APN internet Getting subscription information -- QCI 7 Sending Create Session Request. Creating Session Response -- IMSI: 255172020080033 Creating Session Response -- MME control TEID: 495 Received GTP-C PDU. Message type: GTPC_MSG_TYPE_CREATE_SESSION_REQUEST SPGW: GTP-C context for IMSI 255172020080033 already exists. SPGW: Deleted previous context. SPGW: Allocated Ctrl TEID 495 SPGW: Allocated User TEID 495 SPGW: Allocate UE IP 172.16.0.133 Received Create Session Response Create Session Response -- SPGW control TEID 495 Create Session Response -- SPGW S1-U Address: 127.0.1.100 SPGW Allocated IP 172.16.0.133 to IMSI 000000000000000 Adding attach accept to Initial Context Setup Request Failed to send S1AP PDU. Error: Broken pipe Received UE Context Release Request. MME-UE S1AP Id 1563 UE is not ECM connected. No need to release S1-U. MME UE S1AP Id 1563 Initial UE message: LIBLTE_MME_MSG_TYPE_ATTACH_REQUEST Received Initial UE message -- Attach Request Attach request -- IMSI: 255172020080015 Attach request -- eNB-UE S1AP Id: 115 Attach request -- Attach type: 2 Attach Request -- UE Network Capabilities EEA: 11110000 Attach Request -- UE Network Capabilities EIA: 01110000 Attach Request -- MS Network Capabilities Present: true PDN Connectivity Request -- EPS Bearer Identity requested: 0 PDN Connectivity Request -- Procedure Transaction Id: 1 PDN Connectivity Request -- ESM Information Transfer requested: true Attach Request -- Found previously attach UE. Downlink NAS: Sending Authentication Request UL NAS: Received Authentication Response Authentication Response -- IMSI 255172020080015 UE Authentication Accepted. Generating KeNB with UL NAS COUNT: 0 Downlink NAS: Sending NAS Security Mode Command. UL NAS: Received Security Mode Complete Security Mode Command Complete -- IMSI: 255172020080015 Sending ESM information request UL NAS: Received ESM Information Response ESM Info: APN testapn ESM Info: 7 Protocol Configuration Options Getting subscription information -- QCI 7 Sending Create Session Request. Creating Session Response -- IMSI: 255172020080015 Creating Session Response -- MME control TEID: 496 Received GTP-C PDU. Message type: GTPC_MSG_TYPE_CREATE_SESSION_REQUEST SPGW: GTP-C context for IMSI 255172020080015 already exists. SPGW: Deleted previous context. SPGW: Allocated Ctrl TEID 496 SPGW: Allocated User TEID 496 SPGW: Allocate UE IP 172.16.0.115 Received Create Session Response Create Session Response -- SPGW control TEID 496 Create Session Response -- SPGW S1-U Address: 127.0.1.100 SPGW Allocated IP 172.16.0.115 to IMSI 000000000000000 Adding attach accept to Initial Context Setup Request Failed to send S1AP PDU. Error: Broken pipe Initial UE message: LIBLTE_MME_MSG_TYPE_ATTACH_REQUEST Received Initial UE message -- Attach Request Attach request -- IMSI: 255172020080007 Attach request -- eNB-UE S1AP Id: 116 Attach request -- Attach type: 2 Attach Request -- UE Network Capabilities EEA: 11110000 Attach Request -- UE Network Capabilities EIA: 01110000 Attach Request -- MS Network Capabilities Present: true PDN Connectivity Request -- EPS Bearer Identity requested: 0 PDN Connectivity Request -- Procedure Transaction Id: 1 PDN Connectivity Request -- ESM Information Transfer requested: true Attach Request -- Found previously attach UE. Downlink NAS: Sending Authentication Request UL NAS: Received Authentication Response Authentication Response -- IMSI 255172020080007 UE Authentication Accepted. Generating KeNB with UL NAS COUNT: 0 Downlink NAS: Sending NAS Security Mode Command. UL NAS: Received Security Mode Complete Security Mode Command Complete -- IMSI: 255172020080007 Sending ESM information request UL NAS: Received ESM Information Response ESM Info: APN internet Getting subscription information -- QCI 7 Sending Create Session Request. Creating Session Response -- IMSI: 255172020080007 Creating Session Response -- MME control TEID: 497 Received GTP-C PDU. Message type: GTPC_MSG_TYPE_CREATE_SESSION_REQUEST SPGW: GTP-C context for IMSI 255172020080007 already exists. SPGW: Deleted previous context. SPGW: Allocated Ctrl TEID 497 SPGW: Allocated User TEID 497 SPGW: Allocate UE IP 172.16.0.107 Received Create Session Response Create Session Response -- SPGW control TEID 497 Create Session Response -- SPGW S1-U Address: 127.0.1.100 SPGW Allocated IP 172.16.0.107 to IMSI 000000000000000 Adding attach accept to Initial Context Setup Request Failed to send S1AP PDU. Error: Broken pipe Received UE Context Release Request. MME-UE S1AP Id 1564 UE is not ECM connected. No need to release S1-U. MME UE S1AP Id 1564 Received UE Context Release Request. MME-UE S1AP Id 1565 UE is not ECM connected. No need to release S1-U. MME UE S1AP Id 1565 Received UE Context Release Request. MME-UE S1AP Id 1545 There are active E-RABs, send release access bearers request Received GTP-C PDU. Message type: GTPC_MSG_TYPE_RELEASE_ACCESS_BEARERS_REQUEST Received UE Context Release Complete. MME-UE S1AP Id 1545 There are active E-RABs, send release access bearers request UE Context Release Completed. Received GTP-C PDU. Message type: GTPC_MSG_TYPE_RELEASE_ACCESS_BEARERS_REQUEST Initial UE message: NAS Message Type Unknown Received Initial UE message -- Service Request Service request -- S-TMSI 0x22ae173a Service request -- eNB UE S1AP Id 117 Service Request -- Short MAC valid Service Request -- User is ECM DISCONNECTED UE previously assigned IP: 172.16.0.131 Generating KeNB with UL NAS COUNT: 7 UE Ctr TEID 0 Sent Initial Context Setup Request. E-RAB id 5 Received Initial Context Setup Response E-RAB Context Setup. E-RAB id 5 E-RAB Context -- eNB TEID 0xce0003; eNB GTP-U Address 127.0.1.1 Initial Context Setup Response triggered from Service Request. Sending Modify Bearer Request. Received GTP-C PDU. Message type: GTPC_MSG_TYPE_MODIFY_BEARER_REQUEST Initial UE message: LIBLTE_MME_MSG_TYPE_ATTACH_REQUEST Received Initial UE message -- Attach Request Attach request -- IMSI: 255172020080029 Attach request -- eNB-UE S1AP Id: 118 Attach request -- Attach type: 2 Attach Request -- UE Network Capabilities EEA: 11110000 Attach Request -- UE Network Capabilities EIA: 01110000 Attach Request -- MS Network Capabilities Present: true PDN Connectivity Request -- EPS Bearer Identity requested: 0 PDN Connectivity Request -- Procedure Transaction Id: 1 PDN Connectivity Request -- ESM Information Transfer requested: true Attach Request -- Found previously attach UE. Downlink NAS: Sending Authentication Request UL NAS: Received Authentication Response Authentication Response -- IMSI 255172020080029 UE Authentication Accepted. Generating KeNB with UL NAS COUNT: 0 Downlink NAS: Sending NAS Security Mode Command. UL NAS: Received Security Mode Complete Security Mode Command Complete -- IMSI: 255172020080029 Sending ESM information request UL NAS: Received ESM Information Response ESM Info: APN internet Getting subscription information -- QCI 7 Sending Create Session Request. Creating Session Response -- IMSI: 255172020080029 Creating Session Response -- MME control TEID: 498 Received GTP-C PDU. Message type: GTPC_MSG_TYPE_CREATE_SESSION_REQUEST SPGW: GTP-C context for IMSI 255172020080029 already exists. SPGW: Deleted previous context. SPGW: Allocated Ctrl TEID 498 SPGW: Allocated User TEID 498 SPGW: Allocate UE IP 172.16.0.129 Received Create Session Response Create Session Response -- SPGW control TEID 498 Create Session Response -- SPGW S1-U Address: 127.0.1.100 SPGW Allocated IP 172.16.0.129 to IMSI 000000000000000 Adding attach accept to Initial Context Setup Request Failed to send S1AP PDU. Error: Broken pipe Received UE Context Release Request. MME-UE S1AP Id 1567 UE is not ECM connected. No need to release S1-U. MME UE S1AP Id 1567

ECTPavloR avatar Nov 09 '20 18:11 ECTPavloR

Hey, could you please try with the current version, i.e. 20.10.1? The issue should be fixed by now.

andrepuschmann avatar Nov 22 '20 13:11 andrepuschmann

Hey @andrepuschmann ! Thanks for the prompt reaction and for adding fixes to the minor .1 release! Will try to check early this week. Any tips for the best upgrade strategy(mainly worried if I need to reload pack for RF FE)? Can I reuse the same configs? Sorry I didn't find detailed release notes(

ECTPavloR avatar Nov 22 '20 21:11 ECTPavloR

Yes, the EPC should work with the 20.04 configs. I'll close the issue, please reopen if you still see the issue.

andrepuschmann avatar Nov 23 '20 08:11 andrepuschmann

Hello @andrepuschmann I seem to still see this issue...

I am running

~/srsLTE/srsepc$ srsenb --version --- Software Radio Systems LTE eNodeB --- Version 20.10.1

But strangely, can't get the version of srsepc?

~/srsLTE/srsepc$ srsepc --version Built in Release mode using commit 45486b6e2 on branch master. --- Software Radio Systems EPC --- Reading configuration file /home/bts/.config/srslte/epc.conf...

I have several UE attached, and for one I faced the issue. Even after trying to re-attach the UE the issue persists.

The one below are a section of the messages.

Restarting srsepc and srsenb cures the issue.

Thanks and regards

UE Authentication Accepted. Generating KeNB with UL NAS COUNT: 0 Downlink NAS: Sending NAS Security Mode Command. UL NAS: Received Security Mode Complete Security Mode Command Complete -- IMSI: 901700000044980 Sending ESM information request UL NAS: Received ESM Information Response ESM Info: APN srsapn Getting subscription information -- QCI 7 Sending Create Session Request. Creating Session Response -- IMSI: 901700000044980 Creating Session Response -- MME control TEID: 11 Received GTP-C PDU. Message type: GTPC_MSG_TYPE_CREATE_SESSION_REQUEST SPGW: GTP-C context for IMSI 901700000044980 already exists. SPGW: Deleted previous context. SPGW: Allocated Ctrl TEID 11 SPGW: Allocated User TEID 11 SPGW: Allocate UE IP 172.16.0.12 Received Create Session Response Create Session Response -- SPGW control TEID 11 Create Session Response -- SPGW S1-U Address: 127.0.1.100 SPGW Allocated IP 172.16.0.12 to IMSI 000000000000000 Adding attach accept to Initial Context Setup Request Sent Initial Context Setup Request. E-RAB id 0 Received UE Context Release Request. MME-UE S1AP Id 40 No UE context to release found. MME-UE S1AP Id: 40 Received UE Context Release Request. MME-UE S1AP Id 41 Initial UE message: LIBLTE_MME_MSG_TYPE_ATTACH_REQUEST Received Initial UE message -- Attach Request Attach request -- IMSI: 901700000044980 Attach request -- eNB-UE S1AP Id: 42 Attach request -- Attach type: 2 Attach Request -- UE Network Capabilities EEA: 11100000 Attach Request -- UE Network Capabilities EIA: 01100000 Attach Request -- MS Network Capabilities Present: true PDN Connectivity Request -- EPS Bearer Identity requested: 0 PDN Connectivity Request -- Procedure Transaction Id: 4 PDN Connectivity Request -- ESM Information Transfer requested: true Attach Request -- Found previously attach UE. Downlink NAS: Sending Authentication Request UL NAS: Received Authentication Response Authentication Response -- IMSI 901700000044980 UE Authentication Accepted. Generating KeNB with UL NAS COUNT: 0 Downlink NAS: Sending NAS Security Mode Command. UL NAS: Received Security Mode Complete Security Mode Command Complete -- IMSI: 901700000044980 Sending ESM information request UL NAS: Received ESM Information Response ESM Info: APN srsapn Getting subscription information -- QCI 7 Sending Create Session Request. Creating Session Response -- IMSI: 901700000044980 Creating Session Response -- MME control TEID: 12 Received GTP-C PDU. Message type: GTPC_MSG_TYPE_CREATE_SESSION_REQUEST SPGW: GTP-C context for IMSI 901700000044980 already exists. SPGW: Deleted previous context. SPGW: Allocated Ctrl TEID 12 SPGW: Allocated User TEID 12 SPGW: Allocate UE IP 172.16.0.13 Received Create Session Response Create Session Response -- SPGW control TEID 12 Create Session Response -- SPGW S1-U Address: 127.0.1.100 SPGW Allocated IP 172.16.0.13 to IMSI 000000000000000 Adding attach accept to Initial Context Setup Request Sent Initial Context Setup Request. E-RAB id 0 Received UE Context Release Request. MME-UE S1AP Id 44 There are active E-RABs, send release access bearers request Received GTP-C PDU. Message type: GTPC_MSG_TYPE_RELEASE_ACCESS_BEARERS_REQUEST Received UE Context Release Complete. MME-UE S1AP Id 44 UE Context Release Completed. Received UE Context Release Request. MME-UE S1AP Id 45 Received UE Context Release Request. MME-UE S1AP Id 42 There are active E-RABs, send release access bearers request Received GTP-C PDU. Message type: GTPC_MSG_TYPE_RELEASE_ACCESS_BEARERS_REQUEST Received UE Context Release Complete. MME-UE S1AP Id 42 UE Context Release Completed. Initial UE message: NAS Message Type Unknown Received Initial UE message -- Service Request Service request -- S-TMSI 0x7d38f6d0 Service request -- eNB UE S1AP Id 43 Service Request -- Short MAC valid Service Request -- User is ECM DISCONNECTED UE previously assigned IP: 172.16.0.11 Generating KeNB with UL NAS COUNT: 3 UE Ctr TEID 0 Sent Initial Context Setup Request. E-RAB id 5 Received Initial Context Setup Response E-RAB Context Setup. E-RAB id 5 E-RAB Context -- eNB TEID 0x21; eNB GTP-U Address 127.0.1.1 Initial Context Setup Response triggered from Service Request. Sending Modify Bearer Request. Received GTP-C PDU. Message type: GTPC_MSG_TYPE_MODIFY_BEARER_REQUEST Initial UE message: LIBLTE_MME_MSG_TYPE_ATTACH_REQUEST Received Initial UE message -- Attach Request Attach request -- IMSI: 901700000044980 Attach request -- eNB-UE S1AP Id: 44 Attach request -- Attach type: 2 Attach Request -- UE Network Capabilities EEA: 11100000 Attach Request -- UE Network Capabilities EIA: 01100000 Attach Request -- MS Network Capabilities Present: true PDN Connectivity Request -- EPS Bearer Identity requested: 0 PDN Connectivity Request -- Procedure Transaction Id: 4 PDN Connectivity Request -- ESM Information Transfer requested: true Attach Request -- Found previously attach UE. Downlink NAS: Sending Authentication Request UL NAS: Received Authentication Response Authentication Response -- IMSI 901700000044980 UE Authentication Accepted. Generating KeNB with UL NAS COUNT: 0 Downlink NAS: Sending NAS Security Mode Command. UL NAS: Received Security Mode Complete Security Mode Command Complete -- IMSI: 901700000044980 Sending ESM information request UL NAS: Received ESM Information Response ESM Info: APN srsapn Getting subscription information -- QCI 7 Sending Create Session Request. Creating Session Response -- IMSI: 901700000044980 Creating Session Response -- MME control TEID: 13 Received GTP-C PDU. Message type: GTPC_MSG_TYPE_CREATE_SESSION_REQUEST SPGW: GTP-C context for IMSI 901700000044980 already exists. SPGW: Deleted previous context. SPGW: Allocated Ctrl TEID 13 SPGW: Allocated User TEID 13 SPGW: Allocate UE IP 172.16.0.14 Received Create Session Response Create Session Response -- SPGW control TEID 13 Create Session Response -- SPGW S1-U Address: 127.0.1.100 SPGW Allocated IP 172.16.0.14 to IMSI 000000000000000 Adding attach accept to Initial Context Setup Request Sent Initial Context Setup Request. E-RAB id 0 Received UE Context Release Request. MME-UE S1AP Id 46 Initial UE message: LIBLTE_MME_MSG_TYPE_ATTACH_REQUEST Received Initial UE message -- Attach Request Attach request -- IMSI: 901700000044980 Attach request -- eNB-UE S1AP Id: 45 Attach request -- Attach type: 2 Attach Request -- UE Network Capabilities EEA: 11100000 Attach Request -- UE Network Capabilities EIA: 01100000 Attach Request -- MS Network Capabilities Present: true PDN Connectivity Request -- EPS Bearer Identity requested: 0 PDN Connectivity Request -- Procedure Transaction Id: 4 PDN Connectivity Request -- ESM Information Transfer requested: true Attach Request -- Found previously attach UE. Downlink NAS: Sending Authentication Request UL NAS: Received Authentication Response Authentication Response -- IMSI 901700000044980 UE Authentication Accepted. Generating KeNB with UL NAS COUNT: 0 Downlink NAS: Sending NAS Security Mode Command. UL NAS: Received Security Mode Complete Security Mode Command Complete -- IMSI: 901700000044980 Sending ESM information request UL NAS: Received ESM Information Response ESM Info: APN srsapn Getting subscription information -- QCI 7 Sending Create Session Request. Creating Session Response -- IMSI: 901700000044980 Creating Session Response -- MME control TEID: 14 Received GTP-C PDU. Message type: GTPC_MSG_TYPE_CREATE_SESSION_REQUEST SPGW: GTP-C context for IMSI 901700000044980 already exists. SPGW: Deleted previous context. SPGW: Allocated Ctrl TEID 14 SPGW: Allocated User TEID 14 SPGW: Allocate UE IP 172.16.0.15 Received Create Session Response Create Session Response -- SPGW control TEID 14 Create Session Response -- SPGW S1-U Address: 127.0.1.100 SPGW Allocated IP 172.16.0.15 to IMSI 000000000000000 Adding attach accept to Initial Context Setup Request Sent Initial Context Setup Request. E-RAB id 0 Received UE Context Release Request. MME-UE S1AP Id 48 Received UE Context Release Request. MME-UE S1AP Id 49 Initial UE message: NAS Message Type Unknown Received Initial UE message -- Service Request Service request -- S-TMSI 0x7d38f6c9 Service request -- eNB UE S1AP Id 46 Service Request -- Short MAC valid Service Request -- User is ECM DISCONNECTED UE previously assigned IP: 172.16.0.4 Generating KeNB with UL NAS COUNT: 7 UE Ctr TEID 0 Sent Initial Context Setup Request. E-RAB id 5 Received Initial Context Setup Response E-RAB Context Setup. E-RAB id 5 E-RAB Context -- eNB TEID 0x22; eNB GTP-U Address 127.0.1.1 Initial Context Setup Response triggered from Service Request. Sending Modify Bearer Request. Received GTP-C PDU. Message type: GTPC_MSG_TYPE_MODIFY_BEARER_REQUEST Received UE Context Release Request. MME-UE S1AP Id 47 There are active E-RABs, send release access bearers request Received GTP-C PDU. Message type: GTPC_MSG_TYPE_RELEASE_ACCESS_BEARERS_REQUEST Received UE Context Release Complete. MME-UE S1AP Id 47 UE Context Release Completed. Found UE for Downlink Notification MME Ctr TEID 0xa, IMSI: 901700000044982 T3413 expired -- Could not page the ue. Received GTP-C PDU. Message type: GTPC_MSG_TYPE_DOWNLINK_DATA_NOTIFICATION_FAILURE_INDICATION Found UE for Downlink Notification MME Ctr TEID 0xa, IMSI: 901700000044982 T3413 expired -- Could not page the ue. Received GTP-C PDU. Message type: GTPC_MSG_TYPE_DOWNLINK_DATA_NOTIFICATION_FAILURE_INDICATION Found UE for Downlink Notification

avalori1 avatar Feb 03 '21 08:02 avalori1

If the issue persists with 20.10.1, please provide full logs and PCAP for eNB and EPC. EPC logs in debug, please.

Thanks Andre

andrepuschmann avatar Feb 03 '21 09:02 andrepuschmann

Hello @andrepuschmann , thanks for the quick reply. I forgot to activate the pcap... I have done it now for the future. Here are the .log if they are useful anyway... I had to upload them externally since one is HUGE https://drive.google.com/drive/folders/1g_31-enZ8kE0URc4EEUlQUD-z6wl1nkE?usp=sharing

I hope this works. Thanks Andrea

avalori1 avatar Feb 03 '21 11:02 avalori1

Dear @andrepuschmann do you need some more tests? I am happy to provide. No issues seems happening until there is a single UE connected...

Cheers

avalori1 avatar Feb 05 '21 07:02 avalori1

Not for the moment, we'll investigate. Thanks

andrepuschmann avatar Feb 05 '21 08:02 andrepuschmann

Getting back to the thread - yes, time-to-time keep facing this issue with multiple UEs. In my environment, it's hard to grab a cap since it's randomly happening during kind system performance testing with a high load, pcaps will be huge. Can share more consol logs if needed.

ECTPavloR avatar Feb 05 '21 13:02 ECTPavloR

Hi all, I arrived at this bug report while scratching my head in Google searching for "IP allocations to IMSI 000000000000000". Some extra info, while I collect logs:

  • I can reliably reproduce this with one of my test UEs, a Cat S60 phone, while a OnePlus 6 does not experience this issue.
  • The issue appears once the S60's screen is turned off, at which point the EPC shows "could not page" notifications.
  • Turning airplane mode off/on or rebooting UE does not fix the issue.
  • Restarting srsepc while keeping srsenb running does fix the issue.
  • I have tried to see if the order of attachment makes a difference, and it does not - no matter what UE jumps onto the network first, the S60 exhibits the issue.

I'm on 20.10.1, built on commit 45486b6e2 from master, and using sysmocom SIM cards. RF runs on a USRP B200mini.

puchol avatar Mar 03 '21 13:03 puchol

I have the same problem. E-RAB Context Setup. E-RAB id 5 E-RAB Context -- eNB TEID 0x2; eNB GTP-U Address 127.0.1.1 UL NAS: Received Attach Complete Unpacked Attached Complete Message. IMSI 1010000000002 Unpacked Activate Default EPS Bearer message. EPS Bearer id 5 Received GTP-C PDU. Message type: GTPC_MSG_TYPE_MODIFY_BEARER_REQUEST Sending EMM Information Unhandled NAS integrity protected message LIBLTE_MME_MSG_TYPE_PDN_CONNECTIVITY_REQUEST Unhandled NAS integrity protected message LIBLTE_MME_MSG_TYPE_PDN_CONNECTIVITY_REQUEST Received UE Context Release Request. MME-UE S1AP Id 2 There are active E-RABs, send release access bearers request Received GTP-C PDU. Message type: GTPC_MSG_TYPE_RELEASE_ACCESS_BEARERS_REQUEST Received UE Context Release Complete. MME-UE S1AP Id 2 UE Context Release Completed. Found UE for Downlink Notification MME Ctr TEID 0x2, IMSI: 001010000000002 T3413 expired -- Could not page the ue. Received GTP-C PDU. Message type: GTPC_MSG_TYPE_DOWNLINK_DATA_NOTIFICATION_FAILURE_INDICATION Found UE for Downlink Notification MME Ctr TEID 0x2, IMSI: 001010000000002 T3413 expired -- Could not page the ue. Received GTP-C PDU. Message type: GTPC_MSG_TYPE_DOWNLINK_DATA_NOTIFICATION_FAILURE_INDICATION Found UE for Downlink Notification MME Ctr TEID 0x2, IMSI: 001010000000002 T3413 expired -- Could not page the ue. Received GTP-C PDU. Message type: GTPC_MSG_TYPE_DOWNLINK_DATA_NOTIFICATION_FAILURE_INDICATION Found UE for Downlink Notification MME Ctr TEID 0x2, IMSI: 001010000000002 T3413 expired -- Could not page the ue. Received GTP-C PDU. Message type: GTPC_MSG_TYPE_DOWNLINK_DATA_NOTIFICATION_FAILURE_INDICATION Found UE for Downlink Notification MME Ctr TEID 0x2, IMSI: 001010000000002 T3413 expired -- Could not page the ue. Received GTP-C PDU. Message type: GTPC_MSG_TYPE_DOWNLINK_DATA_NOTIFICATION_FAILURE_INDICATION Found UE for Downlink Notification MME Ctr TEID 0x2, IMSI: 001010000000002 T3413 expired -- Could not page the ue. Received GTP-C PDU. Message type: GTPC_MSG_TYPE_DOWNLINK_DATA_NOTIFICATION_FAILURE_INDICATION Found UE for Downlink Notification MME Ctr TEID 0x2, IMSI: 001010000000002 T3413 expired -- Could not page the ue. Received GTP-C PDU. Message type: GTPC_MSG_TYPE_DOWNLINK_DATA_NOTIFICATION_FAILURE_INDICATION Found UE for Downlink Notification MME Ctr TEID 0x2, IMSI: 001010000000002 T3413 expired -- Could not page the ue. Received GTP-C PDU. Message type: GTPC_MSG_TYPE_DOWNLINK_DATA_NOTIFICATION_FAILURE_INDICATION Found UE for Downlink Notification MME Ctr TEID 0x2, IMSI: 001010000000002 T3413 expired -- Could not page the ue. Received GTP-C PDU. Message type: GTPC_MSG_TYPE_DOWNLINK_DATA_NOTIFICATION_FAILURE_INDICATION Found UE for Downlink Notification MME Ctr TEID 0x2, IMSI: 001010000000002 T3413 expired -- Could not page the ue. Received GTP-C PDU. Message type: GTPC_MSG_TYPE_DOWNLINK_DATA_NOTIFICATION_FAILURE_INDICATION Initial UE message: LIBLTE_MME_MSG_TYPE_ATTACH_REQUEST Received Initial UE message -- Attach Request Attach request -- M-TMSI: 0x11878869 Attach request -- eNB-UE S1AP Id: 3 Attach request -- Attach type: 2 Attach Request -- UE Network Capabilities EEA: 11110000 Attach Request -- UE Network Capabilities EIA: 01110000 Attach Request -- MS Network Capabilities Present: true PDN Connectivity Request -- EPS Bearer Identity requested: 0 PDN Connectivity Request -- Procedure Transaction Id: 33 PDN Connectivity Request -- ESM Information Transfer requested: true Attach Request -- Found previously attach UE. Found UE context. IMSI: 001010000000002, old eNB UE S1ap Id 0, old MME UE S1AP Id 0 Received GUTI-Attach Request from attached user. GUTI Attach request NAS integrity failed. RE-starting authentication procedure. Downlink NAS: Sent Authentication Request Received GTP-C PDU. Message type: GTPC_MSG_TYPE_DELETE_SESSION_REQUEST UL NAS: Received Authentication Response Authentication Response -- IMSI 001010000000002 UE Authentication Accepted. Generating KeNB with UL NAS COUNT: 0 Downlink NAS: Sending NAS Security Mode Command. UL NAS: Received Security Mode Complete Security Mode Command Complete -- IMSI: 001010000000002 Sending ESM information request

quzard avatar Apr 09 '21 02:04 quzard

Disclaimer: all the comments below come empirical observations and (almost) blind attempts, may not be technically correct, and I will be happy to be corrected if I am wrong (and even happier to learn from the correction)!

I noticed that, when the PC starts "struggling", VERY weird things happen, and, at least for the non expert, is DEFINITELY NOT clear that the PC is suffering.

I suggest:

  1. If you have a Desktop installation, remove the graphics, it is obvious that they add weight, but I was shocked by the shear of the difference. I followed this https://linuxconfig.org/how-to-disable-enable-gui-on-boot-in-ubuntu-20-04-focal-fossa-linux-desktop to remove them without re-installing the server version. Then I used this https://askubuntu.com/questions/472829/how-to-open-multi-terminal-on-ubuntu-server to be able to open multiple terminals and run srsenb and srsepc on different terminals.
  2. Reduce the n_prb parameter. This will (of course) decrease the bit rate, but it will be much less demanding on the PC, and it will allow you to check if my hypothesis above makes sense.
  3. I have the impression that this effect of "forgetting the IMSI" happens only with multiple UE (which, intuitively, would make sense). However, it may also be that the probability of happening is only massively decreased and I never saw it happening with a single UE just for statistical reasons.
  4. To test "worse case scenario" I am using Iperf to stimulate the network and push it to the limits. When n_prb is low enough to allow running (for relatively long time) Iperf in a stable manner, the network seems stable in "all" (better say "most", since I did not test extensively all the parameters) other scenarios.

I hope this helps.

avalori1 avatar Apr 09 '21 06:04 avalori1

I have the same problem. E-RAB Context Setup. E-RAB id 5 E-RAB Context -- eNB TEID 0x2; eNB GTP-U Address 127.0.1.1 UL NAS: Received Attach Complete Unpacked Attached Complete Message. IMSI 1010000000002 Unpacked Activate Default EPS Bearer message. EPS Bearer id 5 Received GTP-C PDU. Message type: GTPC_MSG_TYPE_MODIFY_BEARER_REQUEST Sending EMM Information Unhandled NAS integrity protected message LIBLTE_MME_MSG_TYPE_PDN_CONNECTIVITY_REQUEST Unhandled NAS integrity protected message LIBLTE_MME_MSG_TYPE_PDN_CONNECTIVITY_REQUEST Received UE Context Release Request. MME-UE S1AP Id 2 There are active E-RABs, send release access bearers request Received GTP-C PDU. Message type: GTPC_MSG_TYPE_RELEASE_ACCESS_BEARERS_REQUEST Received UE Context Release Complete. MME-UE S1AP Id 2 UE Context Release Completed. Found UE for Downlink Notification MME Ctr TEID 0x2, IMSI: 001010000000002 T3413 expired -- Could not page the ue. Received GTP-C PDU. Message type: GTPC_MSG_TYPE_DOWNLINK_DATA_NOTIFICATION_FAILURE_INDICATION Found UE for Downlink Notification MME Ctr TEID 0x2, IMSI: 001010000000002 T3413 expired -- Could not page the ue. Received GTP-C PDU. Message type: GTPC_MSG_TYPE_DOWNLINK_DATA_NOTIFICATION_FAILURE_INDICATION Found UE for Downlink Notification MME Ctr TEID 0x2, IMSI: 001010000000002 T3413 expired -- Could not page the ue. Received GTP-C PDU. Message type: GTPC_MSG_TYPE_DOWNLINK_DATA_NOTIFICATION_FAILURE_INDICATION Found UE for Downlink Notification MME Ctr TEID 0x2, IMSI: 001010000000002 T3413 expired -- Could not page the ue. Received GTP-C PDU. Message type: GTPC_MSG_TYPE_DOWNLINK_DATA_NOTIFICATION_FAILURE_INDICATION Found UE for Downlink Notification MME Ctr TEID 0x2, IMSI: 001010000000002 T3413 expired -- Could not page the ue. Received GTP-C PDU. Message type: GTPC_MSG_TYPE_DOWNLINK_DATA_NOTIFICATION_FAILURE_INDICATION Found UE for Downlink Notification MME Ctr TEID 0x2, IMSI: 001010000000002 T3413 expired -- Could not page the ue. Received GTP-C PDU. Message type: GTPC_MSG_TYPE_DOWNLINK_DATA_NOTIFICATION_FAILURE_INDICATION Found UE for Downlink Notification MME Ctr TEID 0x2, IMSI: 001010000000002 T3413 expired -- Could not page the ue. Received GTP-C PDU. Message type: GTPC_MSG_TYPE_DOWNLINK_DATA_NOTIFICATION_FAILURE_INDICATION Found UE for Downlink Notification MME Ctr TEID 0x2, IMSI: 001010000000002 T3413 expired -- Could not page the ue. Received GTP-C PDU. Message type: GTPC_MSG_TYPE_DOWNLINK_DATA_NOTIFICATION_FAILURE_INDICATION Found UE for Downlink Notification MME Ctr TEID 0x2, IMSI: 001010000000002 T3413 expired -- Could not page the ue. Received GTP-C PDU. Message type: GTPC_MSG_TYPE_DOWNLINK_DATA_NOTIFICATION_FAILURE_INDICATION Found UE for Downlink Notification MME Ctr TEID 0x2, IMSI: 001010000000002 T3413 expired -- Could not page the ue. Received GTP-C PDU. Message type: GTPC_MSG_TYPE_DOWNLINK_DATA_NOTIFICATION_FAILURE_INDICATION Initial UE message: LIBLTE_MME_MSG_TYPE_ATTACH_REQUEST Received Initial UE message -- Attach Request Attach request -- M-TMSI: 0x11878869 Attach request -- eNB-UE S1AP Id: 3 Attach request -- Attach type: 2 Attach Request -- UE Network Capabilities EEA: 11110000 Attach Request -- UE Network Capabilities EIA: 01110000 Attach Request -- MS Network Capabilities Present: true PDN Connectivity Request -- EPS Bearer Identity requested: 0 PDN Connectivity Request -- Procedure Transaction Id: 33 PDN Connectivity Request -- ESM Information Transfer requested: true Attach Request -- Found previously attach UE. Found UE context. IMSI: 001010000000002, old eNB UE S1ap Id 0, old MME UE S1AP Id 0 Received GUTI-Attach Request from attached user. GUTI Attach request NAS integrity failed. RE-starting authentication procedure. Downlink NAS: Sent Authentication Request Received GTP-C PDU. Message type: GTPC_MSG_TYPE_DELETE_SESSION_REQUEST UL NAS: Received Authentication Response Authentication Response -- IMSI 001010000000002 UE Authentication Accepted. Generating KeNB with UL NAS COUNT: 0 Downlink NAS: Sending NAS Security Mode Command. UL NAS: Received Security Mode Complete Security Mode Command Complete -- IMSI: 001010000000002 Sending ESM information request

This is caused by insufficient performance of my computer

quzard avatar Apr 16 '21 14:04 quzard

Having this issue at the moment too.

I am only using one UE, and I am running without GUI with low PRB (6).

In my experience this happens somewhat randomly, or alternatively when I try access web pages with mobile data.

Has anyone managed to find a solution to this?

knbchse avatar Jul 27 '21 02:07 knbchse

@knbchse, which computer are you using? I realized the hard way that srsRAN is extremely computationally intensive... sometime "insufficient performance" comes very early. When you say "without GUI" do you mean without the GUI part of SRS, or do yo mean the entire Linux installation without UI? If you have the OS UI, I suggest you try to disable it (I followed this guide https://linuxconfig.org/how-to-disable-enable-gui-on-boot-in-ubuntu-20-04-focal-fossa-linux-desktop). Without the OS GUI the performance of my setup increased massively.

avalori1 avatar Jul 27 '21 06:07 avalori1

Thanks for your reply @avalori1!

which computer are you using? I realized the hard way that srsRAN is extremely computationally intensive... sometime "insufficient performance" comes very early.

Ah yes, this makes a lot of sense. I am indeed using a low performance computer, a Raspberry Pi 4 8GB model to be exact. I have enabled the performance CPU governance, however this might still be insufficient. By checking htop, I can see that none of the 4 cores ever reach more than 70% utilisation. However htop may not give us the whole story.

When you say "without GUI" do you mean without the GUI part of SRS, or do yo mean the entire Linux installation without UI?

Both, I have disabled SRS GUI, and I have disabled GUI for the Ubuntu Server installation on the Raspberry Pi.

I have been considering running srsEPC on a separate (more powerful) machine, do you think this would allow better performance of srsRAN? Have you found that srsENB is quite computationally intensive as well?

Are you still having the IMSI issue?

knbchse avatar Jul 27 '21 06:07 knbchse

@knbchse Even if Raspberry is in one of the suggested hardware setups, I suggest you start with a much more powerful computer (a fairly recent i5 at least) and work your way down (if you need) when you have your system working there. For my experience, lack of resources makes you waste a lot of time, since it manifests with different and unclear phenomenon and definitely NOT with a clear error message.

Again, for my (not expert) experience, htop does not give you the entire picture... looks that the CPU gets a spike and then something crashes (and CPU utilization decreases). It is very easy to miss the spike of CPU utilization.

I never tried using different machines for the two components, but had the feeling that the hard part is the ENB. Anyway, if you have a more powerful machine, I would start with both components on that one (and, as you did, keep the PRB low).

avalori1 avatar Jul 27 '21 08:07 avalori1

Hey @avalori1

Thanks for your reply!

For my experience, lack of resources makes you waste a lot of time, since it manifests with different and unclear phenomenon and definitely NOT with a clear error message.

Definitely would agree with this now. I loaded this onto my workstation and weird errors are occurring much less frequently.

Cheers for the advice. Have you been running into any other stability issues out of interest?

knbchse avatar Aug 06 '21 04:08 knbchse

  • UEContextReleaseCommand [NAS-cause=normal-release] protocolIEs: 2 items Item 0: id-UE-S1AP-IDs ProtocolIE-Field id: id-UE-S1AP-IDs (99) criticality: reject (0) value UE-S1AP-IDs: uE-S1AP-ID-pair (0) uE-S1AP-ID-pair mME-UE-S1AP-ID: 7 eNB-UE-S1AP-ID: 10 Item 1: id-Cause ProtocolIE-Field id: id-Cause (2) criticality: ignore (1) value Cause: nas (2) nas: normal-release (0)

  • initiatingMessage procedureCode: id-ErrorIndication (15) criticality: ignore (1) value ErrorIndication protocolIEs: 1 item Item 0: id-Cause ProtocolIE-Field id: id-Cause (2) criticality: ignore (1) value Cause: radioNetwork (0) radioNetwork: unknown-enb-ue-s1ap-id (14) .......

  • UplinkNASTransport ....... Item 1: id-eNB-UE-S1AP-ID ProtocolIE-Field id: id-eNB-UE-S1AP-ID (8) criticality: reject (0) value ENB-UE-S1AP-ID: 13 ......

  • InitialContextSetupRequest, Attach accept, Activate default EPS bearer context request Item 1: id-eNB-UE-S1AP-ID ProtocolIE-Field id: id-eNB-UE-S1AP-ID (8) criticality: reject (0) value ENB-UE-S1AP-ID: 10

  • S1 Application Protocol

    S1AP-PDU: initiatingMessage (0) initiatingMessage procedureCode: id-ErrorIndication (15) criticality: ignore (1) value ErrorIndication protocolIEs: 1 item Item 0: id-Cause ProtocolIE-Field id: id-Cause (2) criticality: ignore (1) value Cause: radioNetwork (0) radioNetwork: unknown-enb-ue-s1ap-id (14) `

reprocedure when UE reattach after idle context release,the EPC seems still using the last ENB-UE-S1AP-ID for the current new context.which cause the eNB refuse to pass,definitely something wrong with the EPC context management function

copslock avatar Feb 18 '22 21:02 copslock

Hello, it SEEMS that this changes decreased the frequency of the issue (I am not sure IF, WHICH and HOW):

  • putting static IPs to the UE in the user_db.csv
  • decreasing the number of UEs
  • increasing the PC performances (and decreasing the requests by putting n_prb=6)

HOWEVER this still happened with plenty of computing power, very little load (one UE at n_prb=6 not even at full DL) and with version 22.4.0

avalori1 avatar Mar 15 '24 07:03 avalori1