geerlingguy
Homelabs Pi Storage, a CM5 NAS board
CM5 Expansion Board - NAS/Server Platform
Introduction
As member of the HomeLabs Club community i really like all of the Raspberry Pi ecosystem, so i decided to work on an expansion board for the Raspberry Pi CM5 that turns it into a proper NAS/server platform with low power consumption. After months of development, I think it's ready to share with the community.
While this started as a personal project inspired by the Raspberry Pi ecosystem and my passion for servers and homelab setups, the non-profit association Homelabs Club is planning to make this available commercially in a near future.
Hardware Features
CM5 Compatibility: Works with both SD card and eMMC versions of the CM5 Built-in SD card slot accessible from inside the board Internal USB-C port allows direct programming of CM5's eMMC or SD card
Sensors
- Current Sensor for the whole board, report directly by linux command "sensors"
Networking:
- 2.5G Ethernet port (Note: This NIC doesn't have a pre-burned MAC address, so the system will generate a random one at boot. I'm working on a solution to program a proper MAC address into the NIC)
- 1G Ethernet port
Connectivity:
- HDMI output
- 2x USB 3.0 ports
- 40-pin GPIO header (standard Pi layout)
Storage:
- 2x NVMe M.2 slots
- 6x SATA ports for HDDs
Power & Control:
- Hardware power button
- 2x 4-pin fan headers for case cooling
- Dedicated CM5 cooling connector
- CR2035 battery holder for RTC power
..... And a lot of LEDs, of course.
Power Requirements
The board needs a 16-21V DC power supply with at least 150W capacity. I recommend going with 180W to be safe, especially if you're planning to run multiple drives.
It must be a DC barrel jack with a 5.5x2.5mm conector, it's really easy and cheap to find. The best option its fo find a cheap laptop DC power supply adapter.
For example, i am using my old ASUS laptop DC power supply, and it work like a rock¡¡
Thermal Considerations
This thing gets hot. The CM5 absolutely needs active cooling - I'm using a heatsink with a fan. The board itself has heatsinks on the main power components, but you'll want good airflow through your case. The fan headers help with this.
Block Diagram
The CM5 has its limitations - I've tried to balance the available resources as best as possible, but there will definitely be bottlenecks. That's part of the fun though, figuring out what those limits are and how to work around them :)
Photos
It's a sleek black PCB - 6 layers of pure awesomeness! The board quality turned out really nice.
Community Project
I've released the STEP files for anyone who wants to design cases. We're 3D printing and building complete NAS units with community designs - it's been pretty cool to see what people come up with.
Notes
A few things I learned during development:
- Don't skimp on the power supply - drives spinning up can cause voltage drops
- Case airflow is critical, especially around the CM5 area
This has been a challenging and super fun project to work on. But honestly, the most exciting part will be seeing what the community creates around this board!
The project in real life is amazing!! @geerlingguy if you want more information to be added here or anything else, just let us know. Thanks @kelroy1990 for all your work, from the beginning of the project to the end.
Very cool board! I'm excited to finally see a nice looking NAS solution with the CM5; some of the other ones require a lot of extra work to get something looking halfway decent.
Another two options that could be considered in a future version:
- Using a PCIe Gen 3 switch chip so the 2.5 Gbps NIC and HDDs don't have to share 5 Gbps max bandwidth
- Maybe having holes in the PCB for airflow, so you can have the drives oriented horizontally with the PCB in the back, like a 'traditional' NAS (though toaster style is fun!).
Hi geerlingguy, the PCI packet Switch is a Gen 3 capable one, retrocompatible with Gen 2. So we have Gen 3 capability if you configure it on the CM5, already tested and the bandwith increase notably.
There is the chip information. https://www.asmedia.com.tw/product/788Yqe9SPAqi5XG0/7c5YQ79xz8urEGr1
You are right about the holes, but the problem its the complexity of this board, its a 6 layer and we need every inch of copper :D, there is a lot of High speed lanes.
[photo removed because somehow the link was broken]
This is only the top hahaha.
We have some extra boards if you want to test it, i am already trying to push it to the limit with the 3D design of another member of the community.
@kelroy1990 - Ah I missed that it was Gen 3!
I would love to test it out — if you ship a board, I would be able to print up to the size of my Bambu P1S print bed (not sure if the case design fits in that volume).
Do you have a website or GitHub repo for the board that I could link to? Also do you have plans to sell the board on Crowd Supply or elsewhere?
Regarding the holes in the board—it would just make all the high speed signals that much more fun hahaha
@geerlingguy Sure, how should we share the shipping information?, mail, discord, telegram...
Right now I am printing it on the H2D but yes, I think that this should fit on a P1S. In the following days some of the community members will meet at an event in Spain, so I have already printed some extra parts if you want, but... of course, you will lose the fun of printing it :) hahaha.
We will upload the 3D files from the community on the github on the following days, but i can send you in advance.
I opened a GitHub project https://github.com/kelroy1990/Homelabs-Pi-Storage where we can share more information as a public site. We will prepare a website for sure, as there are very talented people in the community.
The idea is to create a Crowd Supply campaign as a starting point, and then the Homelabs association will study whether to sell through our own website or with the support of a platform like Tindie, for example. Any advice is welcome of course¡, feedback is gold always.
Hi @geerlingguy where should I send the board? 🙂
We're receiving more designs from the community this week, and next week we'll update the Git repo with the new designs and additional photos. It's amazing how the community gets involved and creates different designs... especially since I'm not great at 3D design myself haha.
@kelroy1990 - you can email me at [email protected] (my public email) and go from there. I'm also somewhat of a noob at 3D printed design, but getting better. So far I haven't designed any PCBs, want to do that at least once by the end of this year (something really simple!).
I have them in hand, they got a quick mention today, before I get to testing them... maybe even with TrueNAS haha: https://www.youtube.com/watch?v=GryCOpcglyI
Just noting this project is on the site now: https://pipci.jeffgeerling.com/boards_cm/homelabs-pi-storage.html
One strange device I haven't seen before on USB is ASMedia KD-123:
pi@pinas:~ $ lsusb
Bus 001 Device 001: ID 1d6b:0002 Linux Foundation 2.0 root hub
Bus 005 Device 002: ID 174c:2362 ASMedia Technology Inc. KD-123
Bus 005 Device 001: ID 1d6b:0003 Linux Foundation 3.0 root hub
Bus 004 Device 001: ID 1d6b:0002 Linux Foundation 2.0 root hub
Bus 003 Device 003: ID 174c:2362 ASMedia Technology Inc. KD-123
Bus 003 Device 002: ID 04b4:6500 Cypress Semiconductor Corp.
Bus 003 Device 001: ID 1d6b:0003 Linux Foundation 3.0 root hub
Bus 002 Device 002: ID 04b4:6502 Cypress Semiconductor Corp. CY4609
Bus 002 Device 001: ID 1d6b:0002 Linux Foundation 2.0 root hub
pi@pinas:~ $ lspci
0001:00:00.0 PCI bridge: Broadcom Inc. and subsidiaries BCM2712 PCIe Bridge (rev 30)
0001:01:00.0 PCI bridge: ASMedia Technology Inc. Device 806a (rev 02)
0001:02:00.0 PCI bridge: ASMedia Technology Inc. Device 806a (rev 02)
0001:02:02.0 PCI bridge: ASMedia Technology Inc. Device 806a (rev 02)
0001:02:03.0 PCI bridge: ASMedia Technology Inc. Device 806a (rev 02)
0001:02:04.0 PCI bridge: ASMedia Technology Inc. Device 806a (rev 02)
0001:02:08.0 PCI bridge: ASMedia Technology Inc. Device 806a (rev 02)
0001:02:0c.0 PCI bridge: ASMedia Technology Inc. Device 806a (rev 02)
0001:02:0e.0 PCI bridge: ASMedia Technology Inc. Device 806a (rev 02)
0001:02:0f.0 PCI bridge: ASMedia Technology Inc. Device 806a (rev 02)
0001:03:00.0 SATA controller: ASMedia Technology Inc. ASM1166 Serial ATA Controller (rev 02)
0001:08:00.0 Ethernet controller: Realtek Semiconductor Co., Ltd. RTL8125 2.5GbE Controller (rev 05)
0002:00:00.0 PCI bridge: Broadcom Inc. and subsidiaries BCM2712 PCIe Bridge (rev 30)
0002:01:00.0 Ethernet controller: Raspberry Pi Ltd RP1 PCIe 2.0 South Bridge
Tested NIC at 2.2 Gbps.
Update: It looks like KD-123 are two NVMe to USB adapters on the USB bus for the two NVMe slots. I'm installing TrueNAS to an NVMe drive in one of the slots :)
A few notes about microSD card usage:
- The fan header is just above the microSD card slot on the design revision I'm testing. This means to install an SD card I've had to pull the cover off that slot, mount the card in it manually, and affix it with Kapton tape. It might've been easier to just remove the fan header on there since this is using a larger fan.
- Maybe due to the mounting, when using full speed I am getting
SDHCI REGISTER DUMPevery now and then, with ADMA Errors, that print to the Linux kernel log. I may try slower SD speeds to see if it cleans up that issue. - It would be a bit easier to have the microSD card slot mounted on an edge so I could get at it without removing the back of the case, but either way is fine... just need to find a way to position it so you don't have to break it to install a card ;)
Hi @geerlingguy maybe i should change the microSD connector, its a connector that you should open before put the microSD inside. If you pull the cover off... i think that you maybe broke it hahahhha :), maybe you can put it again on place.
There is a video example.
The Molex 47219-2001 is a hinge-type microSD card connector that uses a simple three-step installation process:
1.-Unlock the hinge cover: Use your fingernail to gently push the small hinge cover backwards until you hear/feel a click. This releases the locking mechanism and the cover becomes loose. 2.-Open and insert the microSD card: Lift the cover to a 90-degree angle, fully opening the connector. Place the microSD card into the connector slot, ensuring proper orientation. Card polarization features ensure correct card orientation preventing misinsertion. 3.-Close and lock the hinge cover: Lower the hinged cover back down over the microSD card, then push it forward in the opposite direction until you hear/feel another click. This locks the card securely into position.
PD: The KD-123 its the ASM2362 (USB3.2 Device Controller 10G).
If you pull the cover off... i think that you maybe broke it hahahhha
I thought it was one of those pull out types, and I was trying to get it to budge... but it must've been a little stuck. Maybe even shipping / vibration damage, as both units were a little shaken up (some screws here and there inside, and one of the power supply connectors was even smashed flat (that takes some force!).
So I'll be a little more methodical on the other unit haha.
Testing with TrueNAS, a few notes:
- TrueNAS doesn't seem to recognize the PCIe attached SATA drives or the PCIe attached NIC, as I presume they're using a PCIe bridge/switch that is not supported under UEFI :(
- The fans are quite loud once you put the covers on, as there is a lot of airflow restriction. The fan grills should maybe have more openings, and from my experience with my HL15, if you add a little spacer between the fan blades and the grill, that should also help a lot with sound.
- When I plug in power, there are no fans or power on the system. When I press the power button, it boots up as it should. But when I shut down the operating system, the fans and hard drives stay running even after the Pi is shut down. I can boot it again using the power button, but it would be nice if the fans would power off too.
- The front fans don't seem to have anywhere to exhaust, so they are both blowing air straight at the drive bays... with the air having nowhere to go. Removing the fan cover, I found that a lot of air would just blow right back through the fan blades. Having some way to exhaust that air (maybe through the bottom of the case?) would be helpful keeping the drives cooler.
I'm going to look into the UEFI firmware to see if there's a way to get it to work behind a PCIe switch, but I'm thinking maybe not yet :( See: https://github.com/NumberOneGit/rpi5-uefi/issues/21
I set up a RAIDZ1 array with four Seagate IronWolf NAS drives (2x 6 TB, 2x 8 TB), and used the built-in 2.5 Gbps network port, and got between 230-250 MB/sec write from my Mac, and 240-250 MB/sec read in Windows 11.
Hi @geerlingguy a few more information about.
# Current Sensor
dtoverlay=i2c-sensor,ina238,i2c_csi_dsi0,shunt_resistor=20000
# Fan PWM controller IC EMC2302 on the overlay i2c-fan
dtoverlay=i2c-fan,emc2301,addr=0x2e,i2c_csi_dsi0,minpwm=65,maxpwm=255,midtemp=45000,maxtemp=65000,minpwm=0,maxpwm=255
But currently now i control the FAN with a python, i am creating a service to make it easier to control :).
Looking at the case a little more closely, it looks like there are two exhaust vents that go below the CM5 board from the front drive bay area, but I think they may be sized a bit too small for the volume of air needed to keep the drives nice and cool. There is a lot of backpressure, at least with the fans going full blast.
The intake to exhaust ratio for the drive bay area is quite high!
Please don't take any of this feedback negatively—I love the overall case design, it's a very cool setup, and goes together quite nicely. But if there's a design revision, I think a few tweaks for cooling performance and noise levels may take it from good to great :)
Just want to add, this looks like an amazing project, and I for one am extremely excited at the prospect of this becoming commercially available at some point in the future! Would certainly make my current Pi5 ZFS-based NAS setup a lot cleaner and tidier. Great job to all those involved 👍🥇
Do you know, @geerlingguy , if the NVMe's being bridged via the USB controllers could be a hindrance to using those for ZIL or ZFS Special Devices on a pool?
Just want to add, this looks like an amazing project, and I for one am extremely excited at the prospect of this becoming commercially available at some point in the future! Would certainly make my current Pi5 ZFS-based NAS setup a lot cleaner and tidier. Great job to all those involved 👍🥇
Do you know, @geerlingguy , if the NVMe's being bridged via the USB controllers could be a hindrance to using those for ZIL or ZFS Special Devices on a pool?
Hi @eccgecko thanks for your words :). I already tested the NVME as SLOG and ARC cache working great with a hit test. We have the web HomePiNas where we will wrote the information about the product.
Looking at the case a little more closely, it looks like there are two exhaust vents that go below the CM5 board from the front drive bay area, but I think they may be sized a bit too small for the volume of air needed to keep the drives nice and cool. There is a lot of backpressure, at least with the fans going full blast.
The intake to exhaust ratio for the drive bay area is quite high!
Please don't take any of this feedback negatively—I love the overall case design, it's a very cool setup, and goes together quite nicely. But if there's a design revision, I think a few tweaks for cooling performance and noise levels may take it from good to great :)
Of course the feedback its gold always jeff, the community member who design the 3D print case its improving it.