srsRAN_Project Network connectivity issues with docker setup

Network connectivity issues with docker setup

Open dominikheinz opened this issue 7 months ago • 12 comments

Issue Description

I am using srsran and open5gs in the docker compose setup. 1 system runs gnb+core and the other system runs the UE. I get the UE to connect, and from what I can tell also establish a PDN session, but I can't perform a speedtest with iperf3 because of connectivity problems. I also can't reach the librespeed instance that is running on the gnb system under 192.168.42.79:80. I assume my network routing is configured incorrectly. I hope someone can help me.

Setup Details

Ubuntu 22.04
USRP B210
Quectel RM520N-GL

Expected Behavior

I would assume that I can perform a speedtest of the connection with iperf3.

Actual Behaviour

I can't reach the gnb (i think?). I am not entirely sure what IP relates to what.

Steps to reproduce the problem

Run docker-compose -f docker-compose-custom.yml up
Observe the UE establishing a connection
Configure ip on UE system like so:

sudo ip link set dev wwan0 up
sudo ip a add 10.45.0.3/16 dev wwan0

Try to ping 10.53.1.3 -I wwan0 (I think thats the core?). Can't reach.
Try to ping 192.168.42.79 -I wwan0 (gnb host system where speedtest is running). Can't reach.

Additional Information

Routing table gnb+core system:

╰─❯  route -n
Kernel IP routing table
Destination     Gateway         Genmask         Flags Metric Ref    Use Iface
0.0.0.0         192.168.42.1    0.0.0.0         UG    101    0        0 enp0s31f6
10.53.1.0       0.0.0.0         255.255.255.0   U     0      0        0 br-bd331f1276aa
169.254.0.0     0.0.0.0         255.255.0.0     U     1000   0        0 enx00e02265f824
172.17.0.0      0.0.0.0         255.255.0.0     U     0      0        0 docker0
172.18.0.0      0.0.0.0         255.255.0.0     U     0      0        0 br-e89b9b16d236
172.19.1.0      0.0.0.0         255.255.255.0   U     0      0        0 br-dc96ae153b4a
192.168.2.0     0.0.0.0         255.255.255.0   U     100    0        0 enx00e02265f824
192.168.42.0    0.0.0.0         255.255.255.0   U     101    0        0 enp0s31f6

Routing table gnb system:

1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000
    link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
    inet 127.0.0.1/8 scope host lo
       valid_lft forever preferred_lft forever
    inet6 ::1/128 scope host 
       valid_lft forever preferred_lft forever
2: enp0s31f6: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc fq_codel state UP group default qlen 1000
    link/ether 4c:d7:17:84:e2:e9 brd ff:ff:ff:ff:ff:ff
    inet 192.168.42.79/24 brd 192.168.42.255 scope global noprefixroute enp0s31f6
       valid_lft forever preferred_lft forever
    inet6 fe80::8416:f39b:5897:ee54/64 scope link noprefixroute 
       valid_lft forever preferred_lft forever
3: enx00e02265f824: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc fq_codel state UNKNOWN group default qlen 1000
    link/ether 00:e0:22:65:f8:24 brd ff:ff:ff:ff:ff:ff
    inet 192.168.2.10/24 brd 192.168.2.255 scope global dynamic noprefixroute enx00e02265f824
       valid_lft 860128sec preferred_lft 860128sec
    inet6 fe80::c840:b501:6201:10f2/64 scope link noprefixroute 
       valid_lft forever preferred_lft forever
4: ogstun: <NO-CARRIER,POINTOPOINT,MULTICAST,NOARP,UP> mtu 1400 qdisc fq_codel state DOWN group default qlen 500
    link/none 
    inet6 fe80::175:c16d:3c3a:5f1f/64 scope link stable-privacy 
       valid_lft forever preferred_lft forever
5: docker0: <NO-CARRIER,BROADCAST,MULTICAST,UP> mtu 1500 qdisc noqueue state DOWN group default 
    link/ether 02:42:bd:34:d0:05 brd ff:ff:ff:ff:ff:ff
    inet 172.17.0.1/16 brd 172.17.255.255 scope global docker0
       valid_lft forever preferred_lft forever
8: br-e89b9b16d236: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP group default 
    link/ether 02:42:44:c5:b9:a5 brd ff:ff:ff:ff:ff:ff
    inet 172.18.0.1/16 brd 172.18.255.255 scope global br-e89b9b16d236
       valid_lft forever preferred_lft forever
    inet6 fe80::42:44ff:fec5:b9a5/64 scope link 
       valid_lft forever preferred_lft forever
12: veth6b1ae66@if11: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue master br-e89b9b16d236 state UP group default 
    link/ether 0a:c3:05:25:f0:87 brd ff:ff:ff:ff:ff:ff link-netnsid 0
    inet6 fe80::8c3:5ff:fe25:f087/64 scope link 
       valid_lft forever preferred_lft forever
31: br-bd331f1276aa: <NO-CARRIER,BROADCAST,MULTICAST,UP> mtu 1500 qdisc noqueue state DOWN group default 
    link/ether 02:42:2b:ff:45:1a brd ff:ff:ff:ff:ff:ff
    inet 10.53.1.1/24 brd 10.53.1.255 scope global br-bd331f1276aa
       valid_lft forever preferred_lft forever
    inet6 fe80::42:2bff:feff:451a/64 scope link 
       valid_lft forever preferred_lft forever
32: br-dc96ae153b4a: <NO-CARRIER,BROADCAST,MULTICAST,UP> mtu 1500 qdisc noqueue state DOWN group default 
    link/ether 02:42:3c:35:c0:8f brd ff:ff:ff:ff:ff:ff
    inet 172.19.1.1/24 brd 172.19.1.255 scope global br-dc96ae153b4a
       valid_lft forever preferred_lft forever
    inet6 fe80::42:3cff:fe35:c08f/64 scope link 
       valid_lft forever preferred_lft forever

Routing table UE system:

Destination     Gateway         Genmask         Flags Metric Ref    Use Iface
0.0.0.0         0.0.0.0    0.0.0.0         255.255.0.0    U    0        0 wwan0

Interfaces of UE system:

1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000
    link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
    inet 127.0.0.1/8 scope host lo
       valid_lft forever preferred_lft forever
    inet6 ::1/128 scope host 
       valid_lft forever preferred_lft forever
2: enp0s31f6: <NO-CARRIER,BROADCAST,MULTICAST,UP> mtu 1500 qdisc fq_codel state DOWN group default qlen 1000
    link/ether ac:91:a1:52:e7:f5 brd ff:ff:ff:ff:ff:ff
4: wlp0s20f3: <BROADCAST,MULTICAST> mtu 1500 qdisc noop state DOWN group default qlen 1000
    link/ether e0:2e:0b:9f:fe:c0 brd ff:ff:ff:ff:ff:ff
5: wwan0: <POINTOPOINT,MULTICAST,NOARP,UP,LOWER_UP> mtu 1500 qdisc fq_codel state UNKNOWN group default qlen 1000
    link/none 
    inet 10.45.0.3/16 scope global wwan0
       valid_lft forever preferred_lft forever
    inet6 fe80::9228:a5ef:9441:ee85/64 scope link stable-privacy 
       valid_lft forever preferred_lft forever

docker-compose-custom.yml:

services:
  5gc:
    container_name: open5gs_5gc
    build:
      context: open5gs
      target: open5gs
      args:
        OS_VERSION: "22.04"
        OPEN5GS_VERSION: "v2.7.0"
    env_file:
      - ${OPEN_5GS_ENV_FILE:-open5gs/open5gs.env}
    privileged: true
    ports:
      - "9999:9999/tcp"
    # Uncomment port to use the 5gc from outside the docker network
      # - "38412:38412/sctp"
      # - "2152:2152/udp"
    command: 5gc -c open5gs-5gc.yml
    healthcheck:
      test: [ "CMD-SHELL", "nc -z 127.0.0.20 7777" ]
      interval: 3s
      timeout: 1s
      retries: 60
    networks:
      ran:
        ipv4_address: ${OPEN5GS_IP:-10.53.1.2}

  gnb:
    container_name: srsran_gnb
    # Build info
    image: srsran/gnb
    build:
      context: ..
      dockerfile: docker/Dockerfile
      args:
        OS_VERSION: "24.04"
    # privileged mode is requred only for accessing usb devices
    privileged: true
    # Extra capabilities always required
    cap_add:
      - SYS_NICE
      - CAP_SYS_PTRACE
    volumes:
      # Access USB to use some SDRs
      - /dev/bus/usb/:/dev/bus/usb/
      # Sharing images between the host and the pod.
      # It's also possible to download the images inside the pod
      - /usr/share/uhd/images:/usr/share/uhd/images
      # Save logs and more into gnb-storage
      - gnb-storage:/tmp
      - ../configs/gnb_custom.yml:/gnb_config.yml:ro
    # It creates a file/folder into /config_name inside the container
    # Its content would be the value of the file used to create the config
    configs:
      - gnb_config.yml
    # Customize your desired network mode.
    # current netowrk configuration creastes a private netwoek with both containers attached
    # An alterantive would be `network: host"`. That would expose your host network into the container. It's the easiest to use if the 5gc is not in your PC
    networks:
      ran:
        ipv4_address: ${GNB_IP:-10.53.1.3}
      metrics:
        ipv4_address: 172.19.1.3
    # Start GNB container after 5gc is up and running
    depends_on:
      5gc:
        condition: service_healthy
    # Command to run into the final container
    command: gnb -c /gnb_config.yml amf --addr ${OPEN5GS_IP:-10.53.1.2} --bind_addr ${GNB_IP:-10.53.1.3}

  metrics-server:
    container_name: metrics_server
    image: srsran/metrics_server
    build:
      context: metrics_server
    environment:
      - PORT=${METRICS_SERVER_PORT}
      - BUCKET=${DOCKER_INFLUXDB_INIT_BUCKET}
      - TESTBED=default
      - URL=http://${DOCKER_INFLUXDB_INIT_HOST}:${DOCKER_INFLUXDB_INIT_PORT}
      - ORG=${DOCKER_INFLUXDB_INIT_ORG}
      - TOKEN=${DOCKER_INFLUXDB_INIT_ADMIN_TOKEN}
    ports:
      - 55555:${METRICS_SERVER_PORT}/udp
    networks:
      metrics:
        ipv4_address: 172.19.1.4

  influxdb:
    container_name: influxdb
    image: influxdb:${DOCKER_INFLUXDB_VERSION}
    volumes:
      - influxdb-storage:/var/lib/influxdb2:rw
    env_file:
      - .env
    restart: on-failure:10
    # Uncomment port section to access InfluxDB from outside the docker network
    # ports:
    #   - 8086:${DOCKER_INFLUXDB_INIT_PORT}
    networks:
      metrics:
        ipv4_address: 172.19.1.5

  grafana:
    container_name: grafana
    image: srsran/grafana
    build:
      context: grafana
    volumes:
      - grafana-storage:/var/lib/grafana:rw
    env_file:
      - .env
    depends_on:
      - influxdb
      - metrics-server
    ports:
      - 3300:${GRAFANA_PORT}
    networks:
      metrics:
        ipv4_address: 172.19.1.6

configs:
  gnb_config.yml:
    # file: ${GNB_CONFIG_PATH:-../configs/gnb_rf_b200_tdd_n78_20mhz.yml} # Path to your desired config file
    file: ${GNB_CONFIG_PATH:-../configs/gnb_custom.yml} # Path to your desired config file

volumes:
  gnb-storage:
  grafana-storage:
  influxdb-storage:

networks:
  ran:
    ipam:
      driver: default
      config:
        - subnet: 10.53.1.0/24
  metrics:
    ipam:
      driver: default
      config:
        - subnet: 172.19.1.0/24

docker logs:

open5gs_5gc       | 07/23 14:24:14.360: [amf] INFO: gNB-N2 accepted[10.53.1.3]:43886 in ng-path module (../src/amf/ngap-sctp.c:113)
open5gs_5gc       | 07/23 14:24:14.360: [amf] INFO: gNB-N2 accepted[10.53.1.3] in master_sm module (../src/amf/amf-sm.c:741)
open5gs_5gc       | 07/23 14:24:14.363: [amf] INFO: [Added] Number of gNBs is now 1 (../src/amf/context.c:1231)
open5gs_5gc       | 07/23 14:24:14.363: [amf] INFO: gNB-N2[10.53.1.3] max_num_of_ostreams : 30 (../src/amf/amf-sm.c:780)
srsran_gnb        | [INFO] [UHD] linux; GNU C++ version 13.2.0; Boost_108300; UHD_4.6.0.0-0-unknown
srsran_gnb        | [INFO] [LOGGING] Fastpath logging disabled at runtime.
srsran_gnb        | [INFO] [B200] Detected Device: B210
srsran_gnb        | [INFO] [B200] Operating over USB 3.
srsran_gnb        | [INFO] [B200] Initialize CODEC control...
srsran_gnb        | [INFO] [B200] Initialize Radio control...
srsran_gnb        | [INFO] [B200] Performing register loopback test... 
srsran_gnb        | [INFO] [B200] Register loopback test passed
srsran_gnb        | [INFO] [B200] Performing register loopback test... 
srsran_gnb        | [INFO] [B200] Register loopback test passed
srsran_gnb        | [INFO] [B200] Setting master clock rate selection to 'automatic'.
srsran_gnb        | [INFO] [B200] Asking for clock rate 16.000000 MHz... 
srsran_gnb        | [INFO] [B200] Actually got clock rate 16.000000 MHz.
srsran_gnb        | [INFO] [MULTI_USRP] Setting master clock rate selection to 'manual'.
srsran_gnb        | [INFO] [B200] Asking for clock rate 23.040000 MHz... 
srsran_gnb        | [INFO] [B200] Actually got clock rate 23.040000 MHz.
open5gs_5gc       | 07/23 14:24:56.868: [amf] INFO: InitialUEMessage (../src/amf/ngap-handler.c:401)
open5gs_5gc       | 07/23 14:24:56.868: [amf] INFO: [Added] Number of gNB-UEs is now 1 (../src/amf/context.c:2550)
open5gs_5gc       | 07/23 14:24:56.868: [amf] INFO: Unknown UE by 5G-S_TMSI[AMF_ID:0x20000,M_TMSI:0xc00003fb] (../src/amf/ngap-handler.c:479)
open5gs_5gc       | 07/23 14:24:56.868: [amf] INFO:     RAN_UE_NGAP_ID[0] AMF_UE_NGAP_ID[1] TAC[7] CellID[0x66c000] (../src/amf/ngap-handler.c:562)
open5gs_5gc       | 07/23 14:24:56.868: [amf] INFO: Unknown UE by 5G-S_TMSI[AMF_ID:0x20040,M_TMSI:0xc00003fb] (../src/amf/context.c:1853)
open5gs_5gc       | 07/23 14:24:56.868: [amf] INFO: [Added] Number of AMF-UEs is now 1 (../src/amf/context.c:1616)
open5gs_5gc       | 07/23 14:24:56.868: [gmm] INFO: Registration request (../src/amf/gmm-sm.c:1165)
open5gs_5gc       | 07/23 14:24:56.868: [gmm] INFO: [Unknown ID]    5G-S_GUTI[AMF_ID:0x20040,M_TMSI:0xc00003fb] (../src/amf/gmm-handler.c:179)
open5gs_5gc       | 07/23 14:24:56.888: [gmm] INFO: Identity response (../src/amf/gmm-sm.c:1346)
open5gs_5gc       | 07/23 14:24:56.888: [gmm] INFO: [suci-0-999-70-0-0-0-0000066491]    SUCI (../src/amf/gmm-handler.c:921)
open5gs_5gc       | 07/23 14:24:56.948: [gmm] WARNING: Authentication failure(Synch failure) (../src/amf/gmm-sm.c:1567)
open5gs_5gc       | 07/23 14:24:57.271: [gmm] INFO: [imsi-999700000066491] Registration complete (../src/amf/gmm-sm.c:2146)
open5gs_5gc       | 07/23 14:24:57.271: [amf] INFO: [imsi-999700000066491] Configuration update command (../src/amf/nas-path.c:612)
open5gs_5gc       | 07/23 14:24:57.271: [gmm] INFO:     UTC [2024-07-23T12:24:57] Timezone[0]/DST[0] (../src/amf/gmm-build.c:559)
open5gs_5gc       | 07/23 14:24:57.271: [gmm] INFO:     LOCAL [2024-07-23T14:24:57] Timezone[7200]/DST[1] (../src/amf/gmm-build.c:564)
open5gs_5gc       | 07/23 14:25:00.005: [amf] INFO: UE Context Release [Action:2] (../src/amf/ngap-handler.c:1698)
open5gs_5gc       | 07/23 14:25:00.005: [amf] INFO:     RAN_UE_NGAP_ID[0] AMF_UE_NGAP_ID[1] (../src/amf/ngap-handler.c:1699)
open5gs_5gc       | 07/23 14:25:00.005: [amf] INFO:     SUCI[suci-0-999-70-0-0-0-0000066491] (../src/amf/ngap-handler.c:1702)
open5gs_5gc       | 07/23 14:25:00.005: [amf] INFO: [Removed] Number of gNB-UEs is now 0 (../src/amf/context.c:2557)
open5gs_5gc       | 07/23 14:25:00.207: [amf] INFO: InitialUEMessage (../src/amf/ngap-handler.c:401)
open5gs_5gc       | 07/23 14:25:00.207: [amf] INFO: [Added] Number of gNB-UEs is now 1 (../src/amf/context.c:2550)
open5gs_5gc       | 07/23 14:25:00.207: [amf] INFO: Unknown UE by 5G-S_TMSI[AMF_ID:0x20000,M_TMSI:0xc00003ca] (../src/amf/ngap-handler.c:479)
open5gs_5gc       | 07/23 14:25:00.207: [amf] INFO:     RAN_UE_NGAP_ID[1] AMF_UE_NGAP_ID[2] TAC[7] CellID[0x66c000] (../src/amf/ngap-handler.c:562)
open5gs_5gc       | 07/23 14:25:00.207: [amf] INFO: [suci-0-999-70-0-0-0-0000066491] Known UE by 5G-S_TMSI[AMF_ID:0x20040,M_TMSI:0xc00003ca] (../src/amf/context.c:1896)
open5gs_5gc       | 07/23 14:25:00.207: [gmm] INFO: Service request (../src/amf/gmm-sm.c:1283)
open5gs_5gc       | 07/23 14:25:00.207: [gmm] INFO: [suci-0-999-70-0-0-0-0000066491]    5G-S_GUTI[AMF_ID:0x20040,M_TMSI:0xc00003ca] (../src/amf/gmm-handler.c:642)
open5gs_5gc       | 07/23 14:25:00.431: [amf] INFO: [Added] Number of AMF-Sessions is now 1 (../src/amf/context.c:2571)
open5gs_5gc       | 07/23 14:25:00.431: [gmm] INFO: UE SUPI[imsi-999700000066491] DNN[srsapn] S_NSSAI[SST:1 SD:0xffffff] smContextRef [NULL] (../src/amf/gmm-handler.c:1241)
open5gs_5gc       | 07/23 14:25:00.431: [gmm] INFO: No SMF Instance (../src/amf/gmm-handler.c:1278)
open5gs_5gc       | 07/23 14:25:00.432: [sbi] WARNING: [SMF] (NRF-discover) NF has already been added [76b06046-48ee-41ef-9bbb-719bdd4c4934:1] (../lib/sbi/nnrf-handler.c:1057)
open5gs_5gc       | 07/23 14:25:00.432: [sbi] WARNING: NF EndPoint(addr) updated [127.0.0.4:80] (../lib/sbi/context.c:2174)
open5gs_5gc       | 07/23 14:25:00.432: [sbi] WARNING: NF EndPoint(addr) updated [127.0.0.4:7777] (../lib/sbi/context.c:1917)
open5gs_5gc       | 07/23 14:25:00.432: [sbi] INFO: [SMF] (NF-discover) NF Profile updated [76b06046-48ee-41ef-9bbb-719bdd4c4934:1] (../lib/sbi/nnrf-handler.c:1095)
open5gs_5gc       | 07/23 14:25:00.432: [smf] INFO: [Added] Number of SMF-UEs is now 1 (../src/smf/context.c:1019)
open5gs_5gc       | 07/23 14:25:00.432: [smf] INFO: [Added] Number of SMF-Sessions is now 1 (../src/smf/context.c:3068)
open5gs_5gc       | 07/23 14:25:00.434: [sbi] INFO: [SMF] (SCP-discover) NF registered [76b06046-48ee-41ef-9bbb-719bdd4c4934:1] (../lib/sbi/path.c:211)
open5gs_5gc       | 07/23 14:25:00.435: [smf] INFO: UE SUPI[imsi-999700000066491] DNN[srsapn] IPv4[10.45.0.3] IPv6[] (../src/smf/npcf-handler.c:539)
open5gs_5gc       | 07/23 14:25:00.435: [upf] INFO: [Added] Number of UPF-Sessions is now 1 (../src/upf/context.c:208)
open5gs_5gc       | 07/23 14:25:00.435: [gtp] INFO: gtp_connect() [127.0.0.4]:2152 (../lib/gtp/path.c:60)
open5gs_5gc       | 07/23 14:25:00.435: [upf] INFO: UE F-SEID[UP:0xb86 CP:0x3d2] APN[srsapn] PDN-Type[1] IPv4[10.45.0.3] IPv6[] (../src/upf/context.c:485)
open5gs_5gc       | 07/23 14:25:00.435: [upf] INFO: UE F-SEID[UP:0xb86 CP:0x3d2] APN[srsapn] PDN-Type[1] IPv4[10.45.0.3] IPv6[] (../src/upf/context.c:485)
open5gs_5gc       | 07/23 14:25:00.435: [gtp] INFO: gtp_connect() [10.53.1.2]:2152 (../lib/gtp/path.c:60)
open5gs_5gc       | 07/23 14:25:00.435: [smf] WARNING: Unknown PCO ID:(0x17) (../src/smf/context.c:3011)
open5gs_5gc       | 07/23 14:25:00.435: [smf] WARNING: Unknown PCO ID:(0x23) (../src/smf/context.c:3011)
open5gs_5gc       | 07/23 14:25:00.435: [smf] WARNING: Unknown PCO ID:(0x24) (../src/smf/context.c:3011)
open5gs_5gc       | 07/23 14:25:00.509: [gtp] INFO: gtp_connect() [10.53.1.3]:2152 (../lib/gtp/path.c:60)
open5gs_5gc       | 07/23 14:25:00.510: [amf] INFO: [imsi-999700000066491:1:11][0:0:NULL] /nsmf-pdusession/v1/sm-contexts/{smContextRef}/modify (../src/amf/nsmf-handler.c:837)
^CGracefully stopping... (press Ctrl+C again to force)
Stopping srsran_gnb     ... done
Stopping grafana        ... done
Stopping metrics_server ... done
Stopping influxdb       ... done
Stopping open5gs_5gc    ... done

Jul 23 '24 12:07 dominikheinz

srsRAN_Project srsRAN_Project copied to clipboard

Network connectivity issues with docker setup

Issue Description

Setup Details

Expected Behavior

Actual Behaviour

Steps to reproduce the problem

Additional Information

srsRAN_Project
srsRAN_Project copied to clipboard