raspberry-pi-pcie-devices icon indicating copy to clipboard operation
raspberry-pi-pcie-devices copied to clipboard
verified publisher icon geerlingguy

Switchberry - CM4 7-port Managed Switch for PTP

Open geerlingguy opened this issue 1 year ago • 11 comments

The Switchberry is an OCP-TAP open source hardware project to build an open 1 Gbps managed Ethernet switch with PTP capabilities:

The PCB integrates a managed 7-port Gigabit Ethernet switch, exposing various interfaces and peripherals for a versatile and cost-effective 1G network switch with precision time protocol (PTP) capabilities.

This design combines powerful features of the Raspberry Pi CM4 with a managed switch to create an accessible, open, and versatile platform for learning, prototyping, and deploying time-synchronized networks.

Image

At its heart is a Microchip KSZ9567STXI 7-port Gigabit Ethernet switch with IEEE 1588, SGMII/RGMII/MII/RMII.

I have one on hand and will be testing it soon, now that I have the Time HAT up and running on a Pi 5:

  • #674

geerlingguy avatar Mar 20 '25 16:03 geerlingguy

The board is on the site here: https://pipci.jeffgeerling.com/boards_cm/ocp-tap-switchberry.html

Copying over the Getting Started guide from the Switchberry README:

Getting Started

  1. Hardware Setup:

    • Insert the Compute Module 4 (CM4) into the PCB.
    • Connect peripherals such as HDMI, Ethernet, and USB as needed.
    • Connect network cables to the RJ45 connectors and SFP slot for the desired network configuration.

  2. Software Setup:

    • Flash the Raspberry Pi CM4 with a compatible OS (such as Raspberry Pi OS).
    • Set up the SPI interface to manage the KSZ9567 switch via CM4.
    • Configure PTP settings as required for your application.

  3. PTP Configuration:

    • Configure the switch’s Transparent Clock or Boundary Clock mode to enable precise time synchronization.
    • Use PTP software tools on the CM4 to manage and monitor network timing.

My plan is to go from that to either a full Ansible playbook to configure everything, or at least to have the shell commands so you could set it up by hand!

geerlingguy avatar Mar 20 '25 17:03 geerlingguy

Setup steps I've taken so far:

  1. Flash Pi OS to a CM5 with eMMC (haven't tested alternative boot methods yet)
  2. Boot the CM5
  3. Enable SPI: sudo raspi-config nonint do_spi 0 (yes, 0 enables SPI, by adding dtparam=spi=on in /boot/firmware/config.txt)
  4. Reboot the CM5

At this point, I see SPI loaded on the CM5:

$ lsmod | grep spi
spidev                 49152  0
spi_bcm2835            49152  0
spi_dw_mmio            49152  0
spi_dw                 49152  1 spi_dw_mmio

$ ls -l /dev/spi*
crw-rw---- 1 root spi 153, 0 May 22 13:27 /dev/spidev0.0
crw-rw---- 1 root spi 153, 1 May 22 13:27 /dev/spidev0.1
crw-rw---- 1 root spi 153, 2 May 22 13:27 /dev/spidev10.0

I'm thinking a Python program using spidev would be necessary to configure things, but that seems like riding a bicycle with your hands.

Microchip has an Application Note for SPI configuration of their switch chips, but the use is a little obtuse to this SPI noob.

And they refer to a web GUI for one of their evaluation boards that runs on... Ubuntu 14.04 :D — see https://github.com/Microchip-Ethernet/EVB-KSZ9897/releases/tag/LAN7801_KSZ9897_V_1.1

I also found this newer repository Microchip-Ethernet/EVB-KSZ9477 with a Web GUI, but it seems that, too, is meant to build a relatively ancient custom Linux ISO for an evaluation board...

geerlingguy avatar May 22 '25 18:05 geerlingguy

Very very cool looking board !

I've been working on the subject of having my own little switch for a while now, with a board similar to this one.

During my research I learned about switchdev which is a solution built-in the Kernel module to allow the communication with a switch IC just as the one present on this board more easily. It's a really cool way to do it, as it then allow the port to be managed your Linux commands such as ifconfig etc..

Unfortunately it seems like the switch IC used on this board is not supported, as no driver exist.. 😔

If such driver existed (like for the KSZ9477) then the board would be perfect, I think!

If someone knows how to write such drivers, it's their time to shine !

EDIT : Maybe the people working on this board are also working on such driver.

EDIT2 : I might have spoke too soon. On their website, Microchip seems to say that the development board for both the KSZ9477 and KSZ9567 are somewhat similar. They're both 7 Port Managed Gigabit Switch. Could be worth a try.

The following document might help setting up something. And they are mentioning switchdev !

  • https://microchip.my.site.com/s/article/EVB-KSZ9477--How-to-Use-the-Linux-DSA-Driver

  • https://ww1.microchip.com/downloads/en/Appnotes/AN3761-KSZ-DSA-Driver-Utilization-00003761A.pdf

redat00 avatar May 23 '25 15:05 redat00

@redat00 Come join us and let's make this board your dream board too. Julian already wrote the DSA kernel driver for it.

ahmadexp avatar May 23 '25 19:05 ahmadexp

Trying to get the switch chip drivers working on my Pi OS 12 Desktop install:

Step 1 - Kernel Build 1

  1. Make sure SPI is enabled: sudo raspi-config nonint do_spi 0
  2. Install prerequisites: sudo apt install -y bc bison flex libssl-dev make libncurses-dev
  3. Clone Raspberry Pi Linux source: cd Downloads && git clone --depth=1 https://github.com/raspberrypi/linux && cd linux
  4. Copy custom overlay (build1-ksz9567-overlay.dts) into place: scp ksz9567-overlay.dts [email protected]:~/Downloads/linux/arch/arm/boot/dts/overlays/ksz9567-overlay.dts
  5. Add ksz9567.dtbo inside the overlay makefile (nano arch/arm/boot/dts/overlays/Makefile
  6. Edit /boot/firmware/config.txt and add dtoverlay=ksz9567,spi0-0,dev="microchip,ksz9567",speed=50000 under [all]
  7. Copy custom kernel build config (build-config) into place: scp build-config [email protected]:~/Downloads/linux/.config

Attachments:

Rebuild the kernel (these instructions assume a CM4; CM5 will need a different KERNEL and .config):

KERNEL=kernel8
make -j6 Image.gz modules dtbs
sudo make -j6 modules_install
sudo cp /boot/firmware/$KERNEL.img /boot/firmware/$KERNEL-backup.img
sudo cp arch/arm64/boot/Image.gz /boot/firmware/$KERNEL.img
sudo cp arch/arm64/boot/dts/broadcom/*.dtb /boot/firmware/
sudo cp arch/arm64/boot/dts/overlays/*.dtb* /boot/firmware/overlays/
sudo cp arch/arm64/boot/dts/overlays/README /boot/firmware/overlays/

For the build config, it looks like the changes include enabling CONFIG_NET_DSA and associated options, as well as CONFIG_NET_SWITCHDEV, along with many CONFIG_B53 and CONFIG_NET_DSA_* options, along with a few other things... the .config that Julian provided seems to be built for kernel8 / CM4, whereas I'm doing my testing on a CM5 right now.

After the rebuild is complete, reboot and check to make sure the kernel module and ports are there:

$ uname -a
Linux switchberry 6.12.30-v8-DSA+ #1 SMP PREEMPT Fri May 23 15:22:51 CDT 2025 aarch64 GNU/Linux

$ dmesg | grep ksz
[    4.273665] ksz-switch spi0.0: found switch: KSZ9567, rev 0
[    4.273935] ksz-switch spi0.0: Port 6 interpreting RGMII delay settings based on "phy-mode" property, please update device tree to specify "rx-internal-delay-ps" and "tx-internal-delay-ps"
[    4.288183] ksz-switch spi0.0: Port5: using phy mode rgmii-txid
...

$ ip a
1: lo...
2: eth0...
3: wlan0...
4: lan1@eth0...
5: lan2@eth0...
6: lan3@eth0...
7: lan4@eth0...
8: lan5@eth0...
9: lan6-sfp@eth0...

Step 2 - Set up the switch chip as a dumb layer 2 switch

Create the following file (name it like configure-switch.sh):

#!/usr/bin/env bash
set -e

# Disable all ports.
sudo ip link set lan1 down
sudo ip link set lan2 down
sudo ip link set lan3 down
sudo ip link set lan4 down
sudo ip link set lan5 down

# Delete bridge if it exists.
if $(ip a | grep br0); then
    sudo ip link del br0
fi

# Create bridge.
sudo ip link add name br0 type bridge
sudo ip link set br0 type bridge vlan_filtering 1
sudo ip link set br0 up

# Enable all ports.
sudo ip link set lan1 up
sudo ip link set lan2 up
sudo ip link set lan3 up
sudo ip link set lan4 up
sudo ip link set lan5 up

# Set all ports under br0.
sudo ip link set lan1 master br0
sudo ip link set lan2 master br0
sudo ip link set lan3 master br0
sudo ip link set lan4 master br0
sudo ip link set lan5 master br0

# Pass all traffic on all ports.
sudo bridge vlan add dev lan1 vid 1 pvid untagged
sudo bridge vlan add dev lan2 vid 1 pvid untagged
sudo bridge vlan add dev lan3 vid 1 pvid untagged
sudo bridge vlan add dev lan4 vid 1 pvid untagged
sudo bridge vlan add dev lan5 vid 1 pvid untagged

Then make the script executable and run it:

chmod +x configure-switch.sh
./configure-switch.sh

Step 3 - Kernel Build 2

  1. Make sure you still have the updated .config and ksz9567-overlay.dts files in place in your Linux kernel source tree.
  2. Copy the provided microchip driver files[^driver] into linux/drivers/net/dsa/, and overwrite the existing driver:
    1. Download microchip_julian.zip
    2. Move it into place inside the source tree: mv microchip_julian.zip linux/drivers/net/dsa/
    3. Go into the dsa directory: cd linux/drivers/net/dsa/
    4. Unzip the new driver over the existing source files: unzip -o microchip_julian.zip
  3. Rebuild the kernel again (inside the linux directory, so cd ../../../ first):
KERNEL=kernel8
make -j6 Image.gz modules dtbs
sudo make -j6 modules_install
sudo cp /boot/firmware/$KERNEL.img /boot/firmware/$KERNEL-backup.img
sudo cp arch/arm64/boot/Image.gz /boot/firmware/$KERNEL.img
sudo cp arch/arm64/boot/dts/broadcom/*.dtb /boot/firmware/
sudo cp arch/arm64/boot/dts/overlays/*.dtb* /boot/firmware/overlays/
sudo cp arch/arm64/boot/dts/overlays/README /boot/firmware/overlays/

Reboot.

Step 4 - Configure switch

Create the file ksz_reg.sh, meant for setting registers on the switch chip:

#!/usr/bin/env bash
set -e

# Usage: ./ksz_reg.sh <size> <address> [value]
# Example read:  ./ksz_reg.sh 8 0x1432
# Example write: ./ksz_reg.sh 16 0x1432 0xABCD

SYSFS_BASE="/sys/bus/spi/devices/spi0.0"

# Validate args
if [[ "$#" -lt 2 ]]; then
    echo "Usage: $0 <size:8|16|32> <reg_addr> [reg_val]"
    exit 1
fi

SIZE="$1"
ADDR="$2"
VAL="$3"

# Validate size
if [[ "$SIZE" != "8" && "$SIZE" != "16" && "$SIZE" != "32" ]]; then
    echo "Invalid size: $SIZE (must be 8, 16, or 32)"
    exit 1
fi

# Set register size
echo "$SIZE" | sudo tee "$SYSFS_BASE/reg_size" > /dev/null

# Set register address
echo "$ADDR" | sudo tee "$SYSFS_BASE/reg_addr" > /dev/null

if [[ -n "$VAL" ]]; then
    # Write value
    echo "$VAL" | sudo tee "$SYSFS_BASE/reg_val" > /dev/null
    echo "Wrote $VAL to $ADDR (size ${SIZE}-bit)"
else
    # Read value
    VALUE=$(cat "$SYSFS_BASE/reg_val")
    echo "Read from $ADDR (size ${SIZE}-bit): $VALUE"
fi

Make the file executable (chmod +x ksz_reg.sh), and then use it to configure the switch. For example, you can create a script to set up the switch:

#!/usr/bin/env bash
set -e

# Path to the register access script
KSZ_REG="./ksz_reg.sh"

# Enable port VLAN membership
$KSZ_REG 32 0x1a04 0x5f
$KSZ_REG 32 0x2a04 0x5f
$KSZ_REG 32 0x3a04 0x5f
$KSZ_REG 32 0x4a04 0x5f
$KSZ_REG 32 0x5a04 0x5f
$KSZ_REG 32 0x6a04 0x5f
$KSZ_REG 32 0x7a04 0x5f

# ... do other stuff ...

# Enable transparent clock
$KSZ_REG 16 0x514 0x79

Here's a script used to configure the switch for hardware forwarding (currently being tested...), provided for reference: enable_hardware_forwarding.sh.txt (remove the .txt and make it executable to run it).

And the reference for all the registers is in the KSQ9567S Datasheet from Microchip. See especially 5.1.6.11 Global PTP Message Config 1 Register

[^driver]: Note: This driver is a little bit of a hack to allow register access. In the longer term, it would be nice to find a way to get access through the official driver, or upstream changes in a cleaner manner.

geerlingguy avatar May 23 '25 19:05 geerlingguy

Full diff of the bcm2711 defconfig defaults from the latest kernel to the .config provided by Julian for kernel build step 1 above:

Click to show .config differences 
< # Linux/arm64 6.12.30 Kernel Configuration
---
> # Linux/arm64 6.12.22 Kernel Configuration
21d20
< CONFIG_LD_CAN_USE_KEEP_IN_OVERLAY=y
33c32
< CONFIG_LOCALVERSION="-v8"
---
> CONFIG_LOCALVERSION="-v8-DSA"
1516c1515,1539
< # CONFIG_NET_DSA is not set
---
> CONFIG_NET_DSA=m
> CONFIG_NET_DSA_TAG_NONE=m
> CONFIG_NET_DSA_TAG_AR9331=m
> CONFIG_NET_DSA_TAG_BRCM_COMMON=m
> CONFIG_NET_DSA_TAG_BRCM=m
> CONFIG_NET_DSA_TAG_BRCM_LEGACY=m
> CONFIG_NET_DSA_TAG_BRCM_PREPEND=m
> CONFIG_NET_DSA_TAG_HELLCREEK=m
> CONFIG_NET_DSA_TAG_GSWIP=m
> CONFIG_NET_DSA_TAG_DSA_COMMON=m
> CONFIG_NET_DSA_TAG_DSA=m
> CONFIG_NET_DSA_TAG_EDSA=m
> CONFIG_NET_DSA_TAG_MTK=m
> CONFIG_NET_DSA_TAG_KSZ=m
> CONFIG_NET_DSA_TAG_OCELOT=m
> CONFIG_NET_DSA_TAG_OCELOT_8021Q=m
> CONFIG_NET_DSA_TAG_QCA=m
> CONFIG_NET_DSA_TAG_RTL4_A=m
> CONFIG_NET_DSA_TAG_RTL8_4=m
> CONFIG_NET_DSA_TAG_RZN1_A5PSW=m
> CONFIG_NET_DSA_TAG_LAN9303=m
> CONFIG_NET_DSA_TAG_SJA1105=m
> CONFIG_NET_DSA_TAG_TRAILER=m
> CONFIG_NET_DSA_TAG_VSC73XX_8021Q=m
> CONFIG_NET_DSA_TAG_XRS700X=m
1632c1655
< # CONFIG_DCB is not set
---
> CONFIG_DCB=y
1655c1678
< # CONFIG_NET_SWITCHDEV is not set
---
> CONFIG_NET_SWITCHDEV=y
1731d1753
< CONFIG_BT_MTK=m
1736c1758
< CONFIG_BT_HCIBTUSB_MTK=y
---
> # CONFIG_BT_HCIBTUSB_MTK is not set
1829a1852
> CONFIG_NET_IEEE8021Q_HELPERS=y
2587a2611,2659
> 
> #
> # Distributed Switch Architecture drivers
> #
> CONFIG_B53=m
> CONFIG_B53_SPI_DRIVER=m
> CONFIG_B53_MDIO_DRIVER=m
> CONFIG_B53_MMAP_DRIVER=m
> CONFIG_B53_SRAB_DRIVER=m
> CONFIG_B53_SERDES=m
> CONFIG_NET_DSA_BCM_SF2=m
> CONFIG_NET_DSA_LOOP=m
> CONFIG_NET_DSA_LANTIQ_GSWIP=m
> CONFIG_NET_DSA_MT7530=m
> CONFIG_NET_DSA_MT7530_MDIO=m
> CONFIG_NET_DSA_MT7530_MMIO=m
> CONFIG_NET_DSA_MV88E6060=m
> CONFIG_NET_DSA_MICROCHIP_KSZ_COMMON=m
> CONFIG_NET_DSA_MICROCHIP_KSZ9477_I2C=m
> CONFIG_NET_DSA_MICROCHIP_KSZ_SPI=m
> CONFIG_NET_DSA_MICROCHIP_KSZ_PTP=y
> CONFIG_NET_DSA_MICROCHIP_KSZ8863_SMI=m
> CONFIG_NET_DSA_MV88E6XXX=m
> CONFIG_NET_DSA_MV88E6XXX_PTP=y
> CONFIG_NET_DSA_MSCC_FELIX_DSA_LIB=m
> CONFIG_NET_DSA_MSCC_OCELOT_EXT=m
> CONFIG_NET_DSA_MSCC_SEVILLE=m
> CONFIG_NET_DSA_AR9331=m
> CONFIG_NET_DSA_QCA8K=m
> CONFIG_NET_DSA_QCA8K_LEDS_SUPPORT=y
> CONFIG_NET_DSA_SJA1105=m
> CONFIG_NET_DSA_SJA1105_PTP=y
> CONFIG_NET_DSA_XRS700X=m
> CONFIG_NET_DSA_XRS700X_I2C=m
> CONFIG_NET_DSA_XRS700X_MDIO=m
> CONFIG_NET_DSA_REALTEK=m
> CONFIG_NET_DSA_REALTEK_MDIO=y
> CONFIG_NET_DSA_REALTEK_SMI=y
> CONFIG_NET_DSA_REALTEK_RTL8365MB=m
> CONFIG_NET_DSA_REALTEK_RTL8366RB=m
> CONFIG_NET_DSA_REALTEK_RTL8366RB_LEDS=y
> CONFIG_NET_DSA_SMSC_LAN9303=m
> CONFIG_NET_DSA_SMSC_LAN9303_I2C=m
> CONFIG_NET_DSA_SMSC_LAN9303_MDIO=m
> CONFIG_NET_DSA_VITESSE_VSC73XX=m
> CONFIG_NET_DSA_VITESSE_VSC73XX_SPI=m
> CONFIG_NET_DSA_VITESSE_VSC73XX_PLATFORM=m
> # end of Distributed Switch Architecture drivers
> 
2689a2762
> CONFIG_IXGBE_DCB=y
2692a2766
> CONFIG_I40E_DCB=y
2700a2775
> CONFIG_ADIN1110=m
2710a2786,2787
> CONFIG_PRESTERA=m
> CONFIG_PRESTERA_PCI=m
2728c2805,2808
< # CONFIG_VCAP is not set
---
> CONFIG_LAN966X_SWITCH=m
> CONFIG_LAN966X_DCB=y
> CONFIG_VCAP=y
> CONFIG_FDMA=y
2729a2810,2811
> CONFIG_MSCC_OCELOT_SWITCH_LIB=m
> CONFIG_MSCC_OCELOT_SWITCH=m
2777a2860
> CONFIG_ROCKER=m
2867c2950,2951
< # CONFIG_MEDIATEK_GE_PHY is not set
---
> CONFIG_MEDIATEK_GE_PHY=m
> # CONFIG_MEDIATEK_GE_SOC_PHY is not set
2897c2981
< # CONFIG_VITESSE_PHY is not set
---
> CONFIG_VITESSE_PHY=m
2956c3040
< # CONFIG_MDIO_GPIO is not set
---
> CONFIG_MDIO_GPIO=m
2959c3043
< # CONFIG_MDIO_MSCC_MIIM is not set
---
> CONFIG_MDIO_MSCC_MIIM=m
2975c3059,3061
< # CONFIG_PCS_XPCS is not set
---
> CONFIG_PCS_XPCS=m
> CONFIG_PCS_LYNX=m
> CONFIG_PCS_MTK_LYNXI=m
3189d3274
< CONFIG_MT7925_COMMON=m
3191c3276
< CONFIG_MT7925U=m
---
> # CONFIG_MT7925U is not set
4144c4229
< CONFIG_SENSORS_POWERZ=m
---
> # CONFIG_SENSORS_POWERZ is not set
4236c4321
< CONFIG_SENSORS_ADS7828=m
---
> # CONFIG_SENSORS_ADS7828 is not set
4419c4504
< # CONFIG_MFD_OCELOT is not set
---
> CONFIG_MFD_OCELOT=m
4562d4646
< CONFIG_REGULATOR_WAVESHARE_TOUCHSCREEN=m
5102c5186
< CONFIG_VIDEO_IMX335=m
---
> # CONFIG_VIDEO_IMX335 is not set
5554,5555d5637
< CONFIG_DRM_CLIENT_SELECTION=y
< CONFIG_DRM_CLIENT_SETUP=y
5706d5787
< CONFIG_DRM_PANEL_WAVESHARE_TOUCHSCREEN_V2=m
6492d6572
< # CONFIG_HID_UNIVERSAL_PIDFF is not set
7950c8030
< CONFIG_TCS3472=m
---
> # CONFIG_TCS3472 is not set
7960c8040
< CONFIG_VEML6040=m
---
> # CONFIG_VEML6040 is not set
8509,8520c8589
< CONFIG_EROFS_FS=m
< # CONFIG_EROFS_FS_DEBUG is not set
< CONFIG_EROFS_FS_XATTR=y
< CONFIG_EROFS_FS_POSIX_ACL=y
< CONFIG_EROFS_FS_SECURITY=y
< CONFIG_EROFS_FS_BACKED_BY_FILE=y
< CONFIG_EROFS_FS_ZIP=y
< # CONFIG_EROFS_FS_ZIP_LZMA is not set
< # CONFIG_EROFS_FS_ZIP_DEFLATE is not set
< # CONFIG_EROFS_FS_ZIP_ZSTD is not set
< # CONFIG_EROFS_FS_ONDEMAND is not set
< # CONFIG_EROFS_FS_PCPU_KTHREAD is not set
---
> # CONFIG_EROFS_FS is not set
8649,8650c8718
< CONFIG_UNICODE=m
< # CONFIG_UNICODE_NORMALIZATION_SELFTEST is not set
---
> # CONFIG_UNICODE is not set
8973c9041
< # CONFIG_PACKING is not set
---
> CONFIG_PACKING=y
8996c9064
< CONFIG_CRYPTO_ARCH_HAVE_LIB_CHACHA=y
---
> CONFIG_CRYPTO_ARCH_HAVE_LIB_CHACHA=m
8998d9065
< CONFIG_CRYPTO_LIB_CHACHA_INTERNAL=m
9001d9067
< CONFIG_CRYPTO_LIB_CURVE25519_INTERNAL=m
9005c9071
< CONFIG_CRYPTO_ARCH_HAVE_LIB_POLY1305=y
---
> CONFIG_CRYPTO_ARCH_HAVE_LIB_POLY1305=m
9007d9072
< CONFIG_CRYPTO_LIB_POLY1305_INTERNAL=m
9384c9449
< CONFIG_FTRACE_SYSCALLS=y
---
> # CONFIG_FTRACE_SYSCALLS is not set

And here are my updated changes for a bcm2711 config (these work, just updated against a later kernel build):

Click to show updated config diff ```patch CONFIG_LOCALVERSION="-v8-DSA" 1516c1516,1540 CONFIG_NET_DSA=m > CONFIG_NET_DSA_TAG_NONE=m > CONFIG_NET_DSA_TAG_AR9331=m > CONFIG_NET_DSA_TAG_BRCM_COMMON=m > CONFIG_NET_DSA_TAG_BRCM=m > CONFIG_NET_DSA_TAG_BRCM_LEGACY=m > CONFIG_NET_DSA_TAG_BRCM_PREPEND=m > CONFIG_NET_DSA_TAG_HELLCREEK=m > CONFIG_NET_DSA_TAG_GSWIP=m > CONFIG_NET_DSA_TAG_DSA_COMMON=m > CONFIG_NET_DSA_TAG_DSA=m > CONFIG_NET_DSA_TAG_EDSA=m > CONFIG_NET_DSA_TAG_MTK=m > CONFIG_NET_DSA_TAG_KSZ=m > CONFIG_NET_DSA_TAG_OCELOT=m > CONFIG_NET_DSA_TAG_OCELOT_8021Q=m > CONFIG_NET_DSA_TAG_QCA=m > CONFIG_NET_DSA_TAG_RTL4_A=m > CONFIG_NET_DSA_TAG_RTL8_4=m > CONFIG_NET_DSA_TAG_RZN1_A5PSW=m > CONFIG_NET_DSA_TAG_LAN9303=m > CONFIG_NET_DSA_TAG_SJA1105=m > CONFIG_NET_DSA_TAG_TRAILER=m > CONFIG_NET_DSA_TAG_VSC73XX_8021Q=m > CONFIG_NET_DSA_TAG_XRS700X=m 1632c1656 CONFIG_DCB=y 1655c1679 CONFIG_NET_SWITCHDEV=y 1731d1754 # CONFIG_BT_HCIBTUSB_MTK is not set 1829a1853 > CONFIG_NET_IEEE8021Q_HELPERS=y 2587a2612,2660 > > # > # Distributed Switch Architecture drivers > # > CONFIG_B53=m > CONFIG_B53_SPI_DRIVER=m > CONFIG_B53_MDIO_DRIVER=m > CONFIG_B53_MMAP_DRIVER=m > CONFIG_B53_SRAB_DRIVER=m > CONFIG_B53_SERDES=m > CONFIG_NET_DSA_BCM_SF2=m > CONFIG_NET_DSA_LOOP=m > CONFIG_NET_DSA_LANTIQ_GSWIP=m > CONFIG_NET_DSA_MT7530=m > CONFIG_NET_DSA_MT7530_MDIO=m > CONFIG_NET_DSA_MT7530_MMIO=m > CONFIG_NET_DSA_MV88E6060=m > CONFIG_NET_DSA_MICROCHIP_KSZ_COMMON=m > CONFIG_NET_DSA_MICROCHIP_KSZ9477_I2C=m > CONFIG_NET_DSA_MICROCHIP_KSZ_SPI=m > CONFIG_NET_DSA_MICROCHIP_KSZ_PTP=y > CONFIG_NET_DSA_MICROCHIP_KSZ8863_SMI=m > CONFIG_NET_DSA_MV88E6XXX=m > CONFIG_NET_DSA_MV88E6XXX_PTP=y > CONFIG_NET_DSA_MSCC_FELIX_DSA_LIB=m > CONFIG_NET_DSA_MSCC_OCELOT_EXT=m > CONFIG_NET_DSA_MSCC_SEVILLE=m > CONFIG_NET_DSA_AR9331=m > CONFIG_NET_DSA_QCA8K=m > CONFIG_NET_DSA_QCA8K_LEDS_SUPPORT=y > CONFIG_NET_DSA_SJA1105=m > CONFIG_NET_DSA_SJA1105_PTP=y > CONFIG_NET_DSA_XRS700X=m > CONFIG_NET_DSA_XRS700X_I2C=m > CONFIG_NET_DSA_XRS700X_MDIO=m > CONFIG_NET_DSA_REALTEK=m > CONFIG_NET_DSA_REALTEK_MDIO=y > CONFIG_NET_DSA_REALTEK_SMI=y > CONFIG_NET_DSA_REALTEK_RTL8365MB=m > CONFIG_NET_DSA_REALTEK_RTL8366RB=m > CONFIG_NET_DSA_REALTEK_RTL8366RB_LEDS=y > CONFIG_NET_DSA_SMSC_LAN9303=m > CONFIG_NET_DSA_SMSC_LAN9303_I2C=m > CONFIG_NET_DSA_SMSC_LAN9303_MDIO=m > CONFIG_NET_DSA_VITESSE_VSC73XX=m > CONFIG_NET_DSA_VITESSE_VSC73XX_SPI=m > CONFIG_NET_DSA_VITESSE_VSC73XX_PLATFORM=m > # end of Distributed Switch Architecture drivers > 2689a2763 > CONFIG_IXGBE_DCB=y 2692a2767 > CONFIG_I40E_DCB=y 2700a2776 > CONFIG_ADIN1110=m 2710a2787,2788 > CONFIG_PRESTERA=m > CONFIG_PRESTERA_PCI=m 2728c2806,2809 CONFIG_LAN966X_SWITCH=m > CONFIG_LAN966X_DCB=y > CONFIG_VCAP=y > CONFIG_FDMA=y 2729a2811,2812 > CONFIG_MSCC_OCELOT_SWITCH_LIB=m > CONFIG_MSCC_OCELOT_SWITCH=m 2777a2861 > CONFIG_ROCKER=m 2867c2951,2952 CONFIG_MEDIATEK_GE_PHY=m > # CONFIG_MEDIATEK_GE_SOC_PHY is not set 2897c2982 CONFIG_VITESSE_PHY=m 2956c3041 CONFIG_MDIO_GPIO=m 2959c3044 CONFIG_MDIO_MSCC_MIIM=m 2975c3060,3062 CONFIG_PCS_XPCS=m > CONFIG_PCS_LYNX=m > CONFIG_PCS_MTK_LYNXI=m 3189d3275 # CONFIG_MT7925U is not set 4144c4230 # CONFIG_SENSORS_POWERZ is not set 4236c4322 # CONFIG_SENSORS_ADS7828 is not set 4419c4505 CONFIG_MFD_OCELOT=m 5102c5188 # CONFIG_VIDEO_IMX335 is not set 6492c6578 CONFIG_HID_UNIVERSAL_PIDFF=m 7950c8036 # CONFIG_TCS3472 is not set 7960c8046 # CONFIG_VEML6040 is not set 8509,8520c8595 # CONFIG_EROFS_FS is not set 8649,8650c8724 # CONFIG_UNICODE is not set 8973c9047 CONFIG_PACKING=y 9384c9458 # CONFIG_FTRACE_SYSCALLS is not set ```

And here are my changes for a bcm2712 config (didn't work, just posting for the fun of it):

Click to expand my bcm2712 config diff 
< CONFIG_LOCALVERSION="-v8-16k"
---
> CONFIG_LOCALVERSION="-v8-DSA"
1012a1013,1113
> 
> #
> # DSA and Switch options
> #
> CONFIG_NET_DSA=m
> CONFIG_NET_DSA_TAG_NONE=m
> CONFIG_NET_DSA_TAG_AR9331=m
> CONFIG_NET_DSA_TAG_BRCM_COMMON=m
> CONFIG_NET_DSA_TAG_BRCM=m
> CONFIG_NET_DSA_TAG_BRCM_LEGACY=m
> CONFIG_NET_DSA_TAG_BRCM_PREPEND=m
> CONFIG_NET_DSA_TAG_HELLCREEK=m
> CONFIG_NET_DSA_TAG_GSWIP=m
> CONFIG_NET_DSA_TAG_DSA_COMMON=m
> CONFIG_NET_DSA_TAG_DSA=m
> CONFIG_NET_DSA_TAG_EDSA=m
> CONFIG_NET_DSA_TAG_MTK=m
> CONFIG_NET_DSA_TAG_KSZ=m
> CONFIG_NET_DSA_TAG_OCELOT=m
> CONFIG_NET_DSA_TAG_OCELOT_8021Q=m
> CONFIG_NET_DSA_TAG_QCA=m
> CONFIG_NET_DSA_TAG_RTL4_A=m
> CONFIG_NET_DSA_TAG_RTL8_4=m
> CONFIG_NET_DSA_TAG_RZN1_A5PSW=m
> CONFIG_NET_DSA_TAG_LAN9303=m
> CONFIG_NET_DSA_TAG_SJA1105=m
> CONFIG_NET_DSA_TAG_TRAILER=m
> CONFIG_NET_DSA_TAG_VSC73XX_8021Q=m
> CONFIG_NET_DSA_TAG_XRS700X=m
> CONFIG_DCB=y
> CONFIG_NET_SWITCHDEV=y
> CONFIG_NET_IEEE8021Q_HELPERS=y
> CONFIG_IXGBE_DCB=y
> CONFIG_I40E_DCB=y
> CONFIG_ADIN1110=m
> CONFIG_PRESTERA=m
> CONFIG_PRESTERA_PCI=m
> CONFIG_LAN966X_SWITCH=m
> CONFIG_LAN966X_DCB=y
> CONFIG_VCAP=y
> CONFIG_FDMA=y
> CONFIG_MSCC_OCELOT_SWITCH_LIB=m
> CONFIG_MSCC_OCELOT_SWITCH=m
> CONFIG_ROCKER=m
> CONFIG_MEDIATEK_GE_PHY=m
> # CONFIG_MEDIATEK_GE_SOC_PHY is not set
> CONFIG_VITESSE_PHY=m
> CONFIG_MDIO_GPIO=m
> CONFIG_MDIO_MSCC_MIIM=m
> CONFIG_PCS_XPCS=m
> CONFIG_PCS_LYNX=m
> CONFIG_PCS_MTK_LYNXI=m
> CONFIG_MFD_OCELOT=m
> 
> #
> # Distributed Switch Architecture drivers
> #
> CONFIG_B53=m
> CONFIG_B53_SPI_DRIVER=m
> CONFIG_B53_MDIO_DRIVER=m
> CONFIG_B53_MMAP_DRIVER=m
> CONFIG_B53_SRAB_DRIVER=m
> CONFIG_B53_SERDES=m
> CONFIG_NET_DSA_BCM_SF2=m
> CONFIG_NET_DSA_LOOP=m
> CONFIG_NET_DSA_LANTIQ_GSWIP=m
> CONFIG_NET_DSA_MT7530=m
> CONFIG_NET_DSA_MT7530_MDIO=m
> CONFIG_NET_DSA_MT7530_MMIO=m
> CONFIG_NET_DSA_MV88E6060=m
> CONFIG_NET_DSA_MICROCHIP_KSZ_COMMON=m
> CONFIG_NET_DSA_MICROCHIP_KSZ9477_I2C=m
> CONFIG_NET_DSA_MICROCHIP_KSZ_SPI=m
> CONFIG_NET_DSA_MICROCHIP_KSZ_PTP=y
> CONFIG_NET_DSA_MICROCHIP_KSZ8863_SMI=m
> CONFIG_NET_DSA_MV88E6XXX=m
> CONFIG_NET_DSA_MV88E6XXX_PTP=y
> CONFIG_NET_DSA_MSCC_FELIX_DSA_LIB=m
> CONFIG_NET_DSA_MSCC_OCELOT_EXT=m
> CONFIG_NET_DSA_MSCC_SEVILLE=m
> CONFIG_NET_DSA_AR9331=m
> CONFIG_NET_DSA_QCA8K=m
> CONFIG_NET_DSA_QCA8K_LEDS_SUPPORT=y
> CONFIG_NET_DSA_SJA1105=m
> CONFIG_NET_DSA_SJA1105_PTP=y
> CONFIG_NET_DSA_XRS700X=m
> CONFIG_NET_DSA_XRS700X_I2C=m
> CONFIG_NET_DSA_XRS700X_MDIO=m
> CONFIG_NET_DSA_REALTEK=m
> CONFIG_NET_DSA_REALTEK_MDIO=y
> CONFIG_NET_DSA_REALTEK_SMI=y
> CONFIG_NET_DSA_REALTEK_RTL8365MB=m
> CONFIG_NET_DSA_REALTEK_RTL8366RB=m
> CONFIG_NET_DSA_REALTEK_RTL8366RB_LEDS=y
> CONFIG_NET_DSA_SMSC_LAN9303=m
> CONFIG_NET_DSA_SMSC_LAN9303_I2C=m
> CONFIG_NET_DSA_SMSC_LAN9303_MDIO=m
> CONFIG_NET_DSA_VITESSE_VSC73XX=m
> CONFIG_NET_DSA_VITESSE_VSC73XX_SPI=m
> CONFIG_NET_DSA_VITESSE_VSC73XX_PLATFORM=m
> # end of Distributed Switch Architecture drivers

Unfortunately I couldn't get a CM5 going with these options selected... I'll switch gears and try a CM4 instead.

I tried an eMMC and Lite CM4, neither would boot. The LED D1 just briefly flashes, then nothing. No HDMI output, no boot indication. Power usage is 1.4W, so the CM4 doesn't even seem to be drawing power (the board draws 1.4W with no module installed, too). Will debug later. Maybe EEPROM needs updating on the CM4?

Edit: I tried updating the EEPROM, but that didn't work either. No boot with CM4, always boots with CM5. Odd!

Edit 2: Craziest thing I've ever seen. CM4 would not boot cold. I was at my wit's end so I pulled the CM4 out completely while it was powered on, then just gently pushed in the size nearest the PMIC (so the 'left' side closest to the HDMI port area of the Switchberry). And... it started booting!

Then I pushed it fully in so all pins made contact since without that, I couldn't get Ethernet. And Ethernet worked!

I did a sudo reboot and it rebooted just fine. But then I did a sudo shutdown now and then pulled power. When I re-applied power, it would not cold boot. Same thing as earlier, D1 would just briefly flash, but the Pi wouldn't boot.

So doing the 'rock the Pi CM4' thing, it worked again, to get it booting.

Highly inconvenient, but highly entertaining!

geerlingguy avatar May 23 '25 20:05 geerlingguy

After doing that, recompiling the kernel, etc. I have the extra interfaces showing up — though the Pi's built in eth0 interface doesn't seem to get an address through the port on the rear of the Seaberry—only through ports on the front side. Here's what it looks like plugged into port "4" (top right of the group of the tower ports):

$ ip a
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000
    link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
    inet 127.0.0.1/8 scope host lo
       valid_lft forever preferred_lft forever
    inet6 ::1/128 scope host noprefixroute 
       valid_lft forever preferred_lft forever
2: eth0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc mq state UP group default qlen 1000
    link/ether e4:5f:01:36:5b:71 brd ff:ff:ff:ff:ff:ff
    inet6 fe80::77ec:4fd8:b959:3506/64 scope link noprefixroute 
       valid_lft forever preferred_lft forever
3: wlan0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UP group default qlen 1000
    link/ether e4:5f:01:36:5b:72 brd ff:ff:ff:ff:ff:ff
    inet 10.0.2.215/24 brd 10.0.2.255 scope global dynamic noprefixroute wlan0
       valid_lft 7189sec preferred_lft 7189sec
    inet6 fe80::d8b5:5205:c32c:d7cd/64 scope link noprefixroute 
       valid_lft forever preferred_lft forever
4: lan1@eth0: <NO-CARRIER,BROADCAST,MULTICAST,UP> mtu 1500 qdisc noqueue state LOWERLAYERDOWN group default qlen 1000
    link/ether e4:5f:01:36:5b:71 brd ff:ff:ff:ff:ff:ff
5: lan2@eth0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP group default qlen 1000
    link/ether e4:5f:01:36:5b:71 brd ff:ff:ff:ff:ff:ff
    inet6 fe80::7b6a:3d4f:7d1b:d759/64 scope link noprefixroute 
       valid_lft forever preferred_lft forever
6: lan3@eth0: <NO-CARRIER,BROADCAST,MULTICAST,UP> mtu 1500 qdisc noqueue state LOWERLAYERDOWN group default qlen 1000
    link/ether e4:5f:01:36:5b:71 brd ff:ff:ff:ff:ff:ff
7: lan4@eth0: <NO-CARRIER,BROADCAST,MULTICAST,UP> mtu 1500 qdisc noqueue state LOWERLAYERDOWN group default qlen 1000
    link/ether e4:5f:01:36:5b:71 brd ff:ff:ff:ff:ff:ff
8: lan5@eth0: <NO-CARRIER,BROADCAST,MULTICAST,UP> mtu 1500 qdisc noqueue state LOWERLAYERDOWN group default qlen 1000
    link/ether e4:5f:01:36:5b:71 brd ff:ff:ff:ff:ff:ff
9: lan6-sfp@eth0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP group default qlen 1000
    link/ether e4:5f:01:36:5b:71 brd ff:ff:ff:ff:ff:ff
    inet6 fe80::9497:f2ef:1abc:a933/64 scope link noprefixroute 
       valid_lft forever preferred_lft forever

geerlingguy avatar May 24 '25 22:05 geerlingguy

However, when I try running the commands to set up the switch as a dumb layer 2 switch, I get:

$ sudo ip link del br0
Cannot find device "br0"

I don't see any bridge anywhere...

Edit: I'm guessing that was meant to delete a bridge if it had already been set up previously. I added a conditional in the configure-switch.sh script above. That seems to fix things, and now a bridge br0 is set up:

$ ifconfig
br0: flags=4163<UP,BROADCAST,RUNNING,MULTICAST>  mtu 1500
        inet6 fe80::1ca9:1fff:fed5:b235  prefixlen 64  scopeid 0x20<link>
        ether e4:5f:01:36:5b:71  txqueuelen 1000  (Ethernet)
        RX packets 0  bytes 0 (0.0 B)
        RX errors 0  dropped 0  overruns 0  frame 0
        TX packets 0  bytes 0 (0.0 B)
        TX errors 0  dropped 0 overruns 0  carrier 0  collisions 0
...

geerlingguy avatar May 26 '25 14:05 geerlingguy

After going through all the instructions from @julianstj1 (listed a few comments earlier), I believe I have the switch set up as a transparent clock finally, with hardware forwarding.

Next up... how to confirm the settings? (And for future use: how to make it easy to switch between boundary / transparent clock, or change other common settings for the switch?)

geerlingguy avatar May 26 '25 15:05 geerlingguy

Fun aside, and probably just some strange quirk on this V2 prototype board I'm testing...

Every time I cold boot (have it powered off, and then power it on) with a CM4 installed, the CM4 will not begin powering up.

A CM5 powers up from power off just fine. But a CM4 will not.

However, after trying everything under the sun, I figured out if I unplug it and slowly plug it in, pushing down from the 'back' (the Pi's power/WiFi side) towards the 'front' (the Pi's RAM side), it will power up just fine, and all interfaces work.

It'll even power up and run without the front fully pushed down, but then Ethernet and some other high speed interfaces seem to not work (but they come up fine as I rock the whole thing into place.

Strangest behavior from the Hirose B2B connectors I've ever seen!

geerlingguy avatar May 26 '25 15:05 geerlingguy

@geerlingguy Thanks so much for your detailed comments above they’ve been incredibly helpful!

I’ve built a similar PCB based on the SwitchBerry design, but I’m still having trouble getting it to work as anything other than a basic L2 switch. Just wondering if you’ve made any further progress since your last update?

hayzamjs avatar Jun 20 '25 13:06 hayzamjs