UE failed to ping UPF after separating RAN and 5GC

[ljysdfz]

@herlesupreeth

Hello, Herle and every other lovely guy. I have tried to externalize ueransim and deploy it in a separate VM that resides outside the 5GC SA networks. ueransim part was left as much the way as it used to be. My version of open5gs is v2.4.7, ueransim version is v3.2.6. This is the specific topology:

Both of the gNB and UE were up, but I found that the UE failed to ping 192.168.100.1, that is the data plane of UPF.

gNB log:

ip route table in gNB

NIC list in gNB

UE log:

ip route table in UE

NIC list in UE

What's weird was that UE was able to ping 192.168.100.1 through the NIC eth0.

And if you look at the proper examples in the integrated version set up before, you can find that the name of eth0 should be eth0@uesimtun0 instead of eth0@if14. In that way, the traffic enters eth0 will finally pass through the GTP-U tunnel.

the proper one:

From here, I will show what have been configured in the VMs. I've done these things in the VM-5GC, aka 5GC SA networks:

1. In the .env file, change UPF_ADVERTISE_IP -> DOCKER_HOST_IP
2. In the sa-deploy.yaml file, uncomment these 2 code blocks in order to enforce port mappings.

    # ports:
    #   - "38412:38412/sctp"

    # ports:
    #   - "2152:2152/udp"

I've made a new folder ran/ for the individual RAN. Here is a breakdown of the file system.

ran/
  gnb/
    Dockerfile
    open5gs-gnb.yaml
  ue/
    Dockerfile
    open5gs-ue.yaml
  nr-gnb.yaml
  nr-ue.yaml

I will list a series of aforementioned configuration out here:

ran/gnb/Dockerfile

# These are some built images used as prerequisites
FROM docker_ueransim:builder AS ueransim_builder
FROM docker_ueransim:ueransim_provenance

# I just put every single environmental constants in place, 
# then extract the original shell codes and put them into a static Dockerfile
COPY --from=ueransim_builder /UERANSIM/build/nr-gnb /ueransim/nr-gnb
COPY open5gs-gnb.yaml /ueransim
WORKDIR /ueransim
CMD     export IP_ADDR=$(awk 'END{print $1}' /etc/hosts) && \
        ./nr-gnb -c open5gs-gnb.yaml & bash

ran/gnb/open5gs-gnb.yaml

......
......
linkIp: 172.23.0.2  # gNB's local IP address for Radio Link Simulation (Usually same with local IP)
ngapIp: 172.23.0.2   # gNB's local IP address for N2 Interface (Usually same with local IP)
gtpIp: 172.23.0.2   # gNB's local IP address for N3 Interface (Usually same with local IP)

# List of AMF address information
amfConfigs:
  - address: 192.168.12.6
    port: 38412
......
......

ran/ue/Dockerfile

FROM docker_ueransim:builder AS ueransim_builder
FROM docker_ueransim:ueransim_provenance

COPY --from=ueransim_builder /UERANSIM/build/nr-ue /ueransim/nr-ue
COPY open5gs-ue.yaml /ueransim
WORKDIR /ueransim
CMD     export IP_ADDR=$(awk 'END{print $1}' /etc/hosts) && \
        ./nr-ue -c open5gs-ue.yaml & bash

ran/ue/open5gs-ue.yaml

......
......
# List of gNB IP addresses for Radio Link Simulation
gnbSearchList:
  - 172.23.0.2
......
......

ran/nr-gnb.yaml

version: '3'
services:
  nr_gnb:
    image: docker_ueransim_gnb
    container_name: nr_gnb
    stdin_open: true
    tty: true
    volumes:
      - /etc/timezone:/etc/timezone:ro
      - /etc/localtime:/etc/localtime:ro
    expose:
      - "38412/sctp"
      - "2152/udp"
      - "4997/udp"
    cap_add:
      - NET_ADMIN
    devices:
      - "/dev/net/tun"
    privileged: true
    networks:
      default:
        ipv4_address: 172.23.0.2
networks:
  default:
    ipam:
      config:
        - subnet: 172.23.0.0/24
          gateway: 172.23.0.1

ran/nr-ue.yaml

version: '3'
services:
  nr_ue:
    image: docker_ueransim_ue
    container_name: nr_ue
    stdin_open: true
    tty: true
    volumes:
      - /etc/timezone:/etc/timezone:ro
      - /etc/localtime:/etc/localtime:ro
    expose:
      - "4997/udp"
    cap_add:
      - NET_ADMIN
    privileged: true
    networks:
      default:
        ipv4_address: 172.23.0.3
networks:
  default:
    name: ran_default

In light of the fact that there is only ONE IP allocated per VM, so that there is no more host IP space for a vlan (ipvlan or macvlan) driver, though I've made it through by using a ipvlan for the target scenario. I have to use the bridge driver to save IP addresses. I hope whoever is looking at this would come up with a favorable advice or solution. Thank you in advance.

[herlesupreeth]

Can you please attach a trace for the issue taken on both the VMs?

My suspicion is that since both (gNB + UE) and 5GC are running in two VMs behind a NATed IPs (192.168.100.X) there maybe a need for static route in 5GC host machine to reach gNB

[ljysdfz]

@herlesupreeth

I have been weighing up the possibility of the circumstances you have suggested. But in fact, if there is any problem with the visibility for 5GC to reach gNB, then it can't be done for UE to ping UPF through eth0. I have shown that a UE is capable of reaching UPF and vice versa in one of above clips. So I guess the nature of the problem is that the uesimtun0 failed to bind the eth0, so there is not any traffic that have been sent through uesimtun0. But unfortunately, the UERANSIM indicated that the tunnel has been established.

I've used tcpdump to capture the ICMP packets of ping. Actually, when we used the uesimtun0 as the NIC ingress, the tcpdump didn't receive any packets. On the contrary, when we used eth0 as the NIC ingress, ICMP traffic could be seen on the screen.

From the perspective of RAN as pinging through eth0

From the perspective of 5GC as pinging through eth0

If I were to have more time to establish an IDE for stepping in the code, I should find the bug.

[RaulB16]

Hello! I am in the same situation with srsRAN. I can't ping the upf using the ue tunnel. Any update on this topic? @ljysdfz Thanks.

[ljysdfz]

@RaulB16 Sorry, just recognized your comment. I've adopted a new docker network driver named IPVLAN. It acts like the UE/gNB uses a IP address in the same subnet as the host of 5GC. For example, UE (192.168.12.2) gNB (192.168.12.3) 5GC host (192.168.12.4). The 5GC NFs lurk behind the 5GC host as before under a bridge network driver.