Test Intel SASUC8I / LSI SAS3081E-R LSI 1068e RAID controller

[geerlingguy]

trafficstars

I received an Intel SASUC8I and LSI SAS3081E-R RAID controller based on the LSI 1068e controller from @kientran — it is similar to the SAS3082E-R I've been testing (see https://github.com/geerlingguy/raspberry-pi-pcie-devices/issues/18), but there are some differences. Here's a pretty picture:

And see this article from servethehome: Intel SASUC8I and LSI SAS3081E-R LSI 1068e Based RAID Controller Review.

[geerlingguy]

Just noting I'm seeing in some reviews this card only supports disks up to 2.2 TB ('in the latest firmware') FWIW. How could people in pre-2010 not design cards with 10PB drives in mind!?

[kientran]

This card had a previous life in a FreeBSD ZFS system and was flashed to the IT firmware for JBOD disk access. I could have sworn I used it with my 4TB drives, but that was at least 2 file servers ago so who knows ¯_(ツ)_/¯

[geerlingguy]

@kientran - Thanks for the extra info. If it's already flashed to IT, I am hopeful I might be able to have more luck getting it to start up in the Pi than the other two. I now have a separate Intel PC I can use to re-flash cards but I am lacking the time to get that done as I'm trying to focus on a couple other things before end-of-year.

But I do want to get at least one HBA working before 2021! This could be the one...

[geerlingguy]

Heh... looks like I won't get it tested before 2021, but soon after the year turns over ;)

[BryanKadzban]

I have one of these on the way here, along with an rpi 5, so I should be able to test things out once both of those arrive. I plan to use it in a NAS with an Intel RES2SV240 SAS expander, and about ten or so SAS drives (300G each, 15k RPM). Assuming it works of course.

Shipping is supposed to take like a week. But I wanted to write down what I was planning.

[BryanKadzban]

So, it registers in the PCIe subsystem, but in a standard Raspbian environment (as flashed onto an SD card by pishop.us) has no driver. :-/

$ sudo lspci -vvv
0000:01:00.0 SCSI storage controller: Broadcom / LSI SAS1068E PCI-Express Fusion-MPT SAS (rev 08)
        Subsystem: Broadcom / LSI SAS3081E-R 8-Port SAS/SATA Host Bus Adapter
        Control: I/O- Mem- BusMaster- SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR- FastB2B- DisINTx-
        Status: Cap+ 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx-
        Interrupt: pin A routed to IRQ 0
        Region 1: Memory at 1b00210000 (64-bit, non-prefetchable) [disabled] [size=16K]
        Region 3: Memory at 1b00200000 (64-bit, non-prefetchable) [disabled] [size=64K]
        Expansion ROM at 1b00000000 [virtual] [disabled] [size=2M]
        Capabilities: [50] Power Management version 2
                Flags: PMEClk- DSI- D1+ D2+ AuxCurrent=0mA PME(D0-,D1-,D2-,D3hot-,D3cold-)
                Status: D0 NoSoftRst- PME-Enable- DSel=0 DScale=0 PME-
        Capabilities: [68] Express (v1) Endpoint, MSI 00
                DevCap: MaxPayload 4096 bytes, PhantFunc 0, Latency L0s <64ns, L1 <1us
                        ExtTag+ AttnBtn- AttnInd- PwrInd- RBE- FLReset- SlotPowerLimit 0W
                DevCtl: CorrErr+ NonFatalErr+ FatalErr+ UnsupReq+
                        RlxdOrd+ ExtTag+ PhantFunc- AuxPwr- NoSnoop+
                        MaxPayload 512 bytes, MaxReadReq 512 bytes
                DevSta: CorrErr- NonFatalErr- FatalErr- UnsupReq- AuxPwr- TransPend-
                LnkCap: Port #0, Speed 2.5GT/s, Width x8, ASPM L0s L1, Exit Latency L0s <64ns, L1 <1us
                        ClockPM- Surprise- LLActRep- BwNot- ASPMOptComp-
                LnkCtl: ASPM Disabled; RCB 64 bytes, Disabled- CommClk-
                        ExtSynch- ClockPM- AutWidDis- BWInt- AutBWInt-
                LnkSta: Speed 2.5GT/s, Width x1 (downgraded)
                        TrErr- Train- SlotClk- DLActive- BWMgmt- ABWMgmt-
        Capabilities: [98] MSI: Enable- Count=1/1 Maskable- 64bit+
                Address: 0000000000000000  Data: 0000
        Capabilities: [b0] MSI-X: Enable- Count=1 Masked-
                Vector table: BAR=1 offset=00002000
                PBA: BAR=1 offset=00003000
        Capabilities: [100 v1] Advanced Error Reporting
                UESta:  DLP- SDES- TLP- FCP- CmpltTO- CmpltAbrt- UnxCmplt- RxOF- MalfTLP- ECRC- UnsupReq- ACSViol-
                UEMsk:  DLP- SDES- TLP- FCP- CmpltTO- CmpltAbrt- UnxCmplt- RxOF- MalfTLP- ECRC- UnsupReq- ACSViol-
                UESvrt: DLP+ SDES- TLP- FCP+ CmpltTO- CmpltAbrt- UnxCmplt- RxOF+ MalfTLP+ ECRC- UnsupReq- ACSViol-
                CESta:  RxErr- BadTLP- BadDLLP- Rollover- Timeout- AdvNonFatalErr-
                CEMsk:  RxErr- BadTLP- BadDLLP- Rollover- Timeout- AdvNonFatalErr-
                AERCap: First Error Pointer: 00, ECRCGenCap+ ECRCGenEn- ECRCChkCap+ ECRCChkEn-
                        MultHdrRecCap- MultHdrRecEn- TLPPfxPres- HdrLogCap-
                HeaderLog: 00000000 00000000 00000000 00000000

$ cat /sys/devices/platform/axi/*.pcie/pci*:*/*/0000:01:00.0/{vendor,device,subsystem_vendor,subsystem_device}
0x1000
0x0058
0x1000
0x3140

0x1000 is Broadcom, and 0058 is the SAS1068e, so this is the right sysfs directory ... unfortunately:

$ ls -l /sys/devices/platform/axi/*.pcie/pci*:*/*/0000:01:00.0/driver
ls: cannot access '/sys/devices/platform/axi/*.pcie/pci*:*/*/0000:01:00.0/driver': No such file or directory

so there's no driver bound, and I therefore assume there's nothing compiled (...because modprobe didn't find it), that matches this vid/pid pair.

I see several references to an mptsas driver (and in fact, I found a driver download for this device, from supermicro of all places, which generates an mptsas module with a vendor/device alias that matches here). I also think I've seen mptsas related drivers in the kernel configuration tree many years ago. Maybe I just have to turn one on. I think I've successfully cloned the raspbian kernel repository; looking at the best way to rebuild this. I've built a ton of kernels for x86, but none on a Debian system, and none on an aarch64 system. This is going to be fun! :)

[BryanKadzban]

OK, yeah. I found the linux-config-6.6 package, with a contained -2712 config file; decompressing it and doing a "make oldconfig", then trawling through menuconfig, finds that the Device Drivers -> Fusion MPT section is turned off. Turning it on provides a driver for SPI devices and one for SAS; the latter matches this card (according to the driver's pci_device table).

I also see a separate ioctl driver, which the help says can be used to flash firmware, so I'm enabling it as well. (I doubt this device is in IT mode. We'll see.)

I'll find out how long a pi5 takes to build a kernel now, with -j4. :)

[BryanKadzban]

Takes at least an hour. :)

At boot the driver loaded automatically, and mptsas got bound to the device in lspci. The connected drives did not show up ... but that turned out to be because I didn't have the SAS cable (the SFF-8087 end, at the LSI card) plugged in all the way. It hotplugged, spun up, and got recognized as /dev/sdb as soon as I clicked it all the way in.

With the mptctl module loaded, mpt-status -p doesn't find anything, which is odd, but maybe that's not a huge deal considering the drive does seem to work. Just have to figure out how to monitor for failed drives now; almost all the way there.

I did not flash into IT mode, but I don't know how to tell for sure if the card is already in that mode either (I think this card was new, but not 100% sure).