MacBookPro 15,1/2?

[aunali1]

Any support/documentation for the refreshed 2018 MacBook Pros?

I have one and I'll be willing to help with tinkering stuff especially since they are using the new T2 chips.

[roadrunner2]

Is the keyboard and touchbar (using the drivers at my clone of macbook12-spi-driver) working for you? I've had reports so far that at least the keyboard doesn't. Assuming that's true, help would be appreciated in trying to debug the issue (and in which case please open an issue for this specifically).

[aunali1]

So I went ahead and custom built a Arch Linux live image with @roadrunner2's driver. (Archiso releng profile + macbook12-spi-driver-dkms from AUR)

While the driver successfully loads, the keyboard fails to function and the touchbar remains blank.

EDIT: A link to the image for reference

[Dunedan]

Any support/documentation for the refreshed 2018 MacBook Pros?

You're the first person here who got one, so you might be the one being able to provide information. :smile:

As there are some significant differences compared to the previous models, mainly due to the introduction of Apple's T2-chip, I suspect fewer components will work for now.

As a start a PR with the output of the get-info.sh script would be really good.

And you can of course follow the instructions for the 2016/2017 models and have a look which components work and which don't. For example I expect that Linux won't have access to the sensors previously exposed by the system management controller (SMC), as that's now handled by the T2-chip. Same for access to the disk via NVMe. While I expect that this works, it might require a kernel patch as did the controller for the 2016 and 2017 models.

But as said: a PR with the information from the get-info.sh would be a great start and much appreciated. :+1:

[aunali1]

Well the SSD doesn't work at all, same with WIFI, looks like a new chip BCM4464 (nothing on google.)

I also attempted manually specifying the SSD PCI ID (106B:2005 - ANS2 NVME Controller) with the following,

$ modprobe nvme
$ echo 106b 2005 > /sys/bus/pci/drivers/nvme/new_id

after which lsblk showed a basic nvme dev, then after about 10 seconds the screen instantly went black alongside which the fans spun up to full speed, lasting for about a second, then the MacBook rebooted into macOS.

Not sure what went wrong, prob the T2 crashed or smth? I did receive a crash report after booting macOS with BridgeOS reporting a recoverable ANS2 failure.

FYI, sending in a PR with the info from get-info.sh soon.

[Dunedan]

Lots of interesting bits in the dumps. The obvious ones:

• no iBridge device shown in lsusb (which might explain why the Touch Bar doesn't work. Because of that I also expect the iSight webcam to not work as well)
• two unknown PCI-devices Apple Inc. Device [106b:1801] and Apple Inc. Device [106b:1802] (maybe that's what the T2-chip exposes for Touch Bar and iSight webcam?)
• applesmc failing to read stuff (kind of expected as the T2-chip includes the SMC functionality now)

[aunali1]

@Dunedan Would a dump from system_profiler on macOS be helpful?

[Dunedan]

Sure. More information is always good. :smile:

[aunali1]

Pastebin was complaining about file size so I just uploaded to here. (2.7 MB)

Of particular interest are the following,

• IOBC@0,1 matching [106b:1801] above (Bridge Controller?)
• SEPM@0,2 matching [106b:1802] above (Secure Enclave Processor?)

It seems that Apple is emulating a HCI controller through the IOBC to interface with the iBridge devices as USB, as evidenced by the recurring AppleUSBVHCIPorts and AppleUSBVHCIBCE virtual controller.

[wsy2220]

The biggest problem on my 15,1 is applesmc module, it refused to load with these messages:

[ 1124.528835] applesmc: send_byte(0x10, 0x0304) fail: 0xff
[ 1124.528837] applesmc: #KEY: read arg fail
[ 1124.719729] applesmc: send_byte(0x10, 0x0304) fail: 0xff
[ 1124.719732] applesmc: #KEY: read arg fail
[ 1124.907593] applesmc: send_byte(0x10, 0x0304) fail: 0xff
[ 1124.907595] applesmc: #KEY: read arg fail
[ 1125.095762] applesmc: send_byte(0x10, 0x0304) fail: 0xff
[ 1125.095764] applesmc: #KEY: read arg fail
[ 1125.283551] applesmc: send_byte(0x10, 0x0304) fail: 0xff
[ 1125.283553] applesmc: #KEY: read arg fail
[ 1125.471537] applesmc: send_byte(0x10, 0x0304) fail: 0xff
[ 1125.471539] applesmc: #KEY: read arg fail
[ 1125.659650] applesmc: send_byte(0x10, 0x0304) fail: 0xff
[ 1125.659652] applesmc: #KEY: read arg fail
[ 1125.847437] applesmc: send_byte(0x10, 0x0304) fail: 0xff
[ 1125.847439] applesmc: #KEY: read arg fail
[ 1126.035669] applesmc: send_byte(0x10, 0x0304) fail: 0xff
[ 1126.035671] applesmc: #KEY: read arg fail
[ 1126.223667] applesmc: send_byte(0x10, 0x0304) fail: 0xff
[ 1126.223668] applesmc: #KEY: read arg fail
[ 1126.411547] applesmc: send_byte(0x10, 0x0304) fail: 0xff
[ 1126.411549] applesmc: #KEY: read arg fail
[ 1126.599665] applesmc: send_byte(0x10, 0x0304) fail: 0xff
[ 1126.599667] applesmc: #KEY: read arg fail

without applesmc, it's always very hot.

My kernel is

Linux version 4.17.0-3-amd64 ([email protected]) (gcc version 7.3.0 (Debian 7.3.0-28)) #1 SMP Debian 4.17.17-1 (2018-08-18)

I haven't tried to get keyboard/touchpad to work. I'd like to provide any information needed.

edit: #70 solved the melting temperature, by adding 'acpi_mask_gpe=0x6e' kernel parameter

[mikeeq]

Have you tried to turn off secure boot? https://support.apple.com/en-us/HT208330 Maybe that's the reason why it doesn't see any NVMe devices.

[aunali1]

@mikeeq Yes, thats the only way I was able to boot the device.

[aunali1]

@wsy2220 Do you have FileVault enabled? If not it me be worthwhile to test the following commands I used to recognize the NVMe drive:

$ modprobe nvme
$ echo 106b 2005 > /sys/bus/pci/drivers/nvme/new_id

I have a hunch that with FileVault disabled it won't spontaneously crash like it did in my situation.

[wsy2220]

@aunali1 I have no FileVault. I tried those commands, and nvme device didn't show up, and the system got reset after about 10s.

[roadrunner2]

Regarding the keyboard, looking at the provided pci listing and dsdt, I see two things:

1. There does not appear to be any driver in the kernel for the SPI controller on these machines (pci id 8086:a324)
2. I cannot find any info about the keyboard in the DSDT, so I don't think it's attached to the SPI bus/controller (in fact, according to the DSDT there's nothing attached the SPI bus/controller)

Looking at the ioreg info is more informative: it appears it's back on the USB bus, and hanging off the Bridge-Controller/T2-chip. But since it doesn't show up in lspci, I guess something needs to initialize the BridgeController first?

[AndreasAZiegler]

Did someone have any success with the nvme of the MacBook Pro 2018? I have the same problems as other people described earlier (keyboard and trackpad not working and nvme not visible). The not working keyboard and trackpad doesn't bother me that much as I'm anyway using an external keyboard and mouse most of the time. The not recognized nvme however bother me quite a lot.

[Dunedan]

@wsy2220 I don't expect applesmc to work anymore for the MacBook Pro 2018, as Apple moved the SMC-functionality into the T2-chip. We saw the first step already with devices with the T1-chip, where the ambient light sensor isn't handled by the SMC anymore, but probably by the T1-chip, using a different interface to access it (and so far nobody has figured out how that looks like).

@AndreasAZiegler The NVMe controller isn't recognized by default, because Apple screwed up and put a typo in the class code for the NVMe controller. So to get it automatically detected a patch for the Linux kernel is necessary (e.g. check out the patch I submitted for the MacBook Pro 2016: http://lists.infradead.org/pipermail/linux-nvme/2017-February/008323.html). Without the patch, you can manually discover the NVMe controller with the following commands (as already mentioned in the thread by @aunali1):

$ modprobe nvme
$ echo 106b 2005 > /sys/bus/pci/drivers/nvme/new_id

The problem then seems to be that this is causing system resets soon after as mentioned above as well.

I did receive a crash report after booting macOS with BridgeOS reporting a recoverable ANS2 failure.

@aunali1 Any chance you (or somebody else) can provide such a crash report? Maybe it contains some useful information.

Updating to the latest macOS version could also fix the system reset, if it was caused by a bug in BridgeOS (not likely, but worth a shot).

[KTRosenberg]

Hello all, I recently received this model of macbook pro, but it looks like unfortunately Linux is crippled on it. I would have liked to boot off an external drive, but if I understand correctly, the keyboard and trackpad don't work, and even worse, audio and wifi don't work at all. I suppose that brightness controls also don't work then? Has any headway been made towards making Linux usable on these machines? Thanks.

[jboyens]

@KTRosenberg This entire Github project is designed to help you make it work on these laptops. I use it daily on a 14,3. There are compromises, but it's very useable.

[roadrunner2]

Could somebody provide the output from sudo acpidump? On the MBP13,3 at least the entries for the iBridge are in one of supplementary tables, not the main DSDT. And I'm wondering if the new chip(s) need some sort of ACPI call to activate them, since they don't appear on the USB bus. Of course since they show up as pci devices it's also possible they need to be activated via PCI.

@Dunedan Might I suggest changing the get-info.sh script to get the full acpi info rather than just the DSDT?

[KTRosenberg]

@jboyens I see. But just to confirm--is it true that the wifi and audio are still unusuable beyond connecting external adapters and audio interfaces?

[jboyens]

@KTRosenberg it depends on the model. But yes, audio and WiFi are not working consistently without some compromises. I use a small WiFi dongle and USB headphones.

[KTRosenberg]

I was referring to the 2018 model, which seems to be less functional than the previous models due to the T2 chip-related changes.

[Dunedan]

@roadrunner2 Sure. Free free to open a PR and add what you think would be beneficial as well.

[AndreasAZiegler]

While trying around with the latest Arch Linux installer (2018.10) and the current Ubuntu Live System, I made the following observations. When I run modprobe nvme echo 106b 2005 > /sys/bus/pci/drivers/nvme/new_id

in the Ubuntu Live system sometimes the nvme drive appears and sometimes it doesn't. In both cases the system resets after around 10s as desribed by @wsy2220. In the Arch Linux installer neither does the drive appear nor does the system resets. Does anyone has some more success yet?

[aunali1]

@roadrunner2 I'll spin up a new Arch image and test out the command. @Dunedan If the T2 crashes again, as it probably will, I'll post the dump of the report from macOS. @AndreasAZiegler Odd. It always showed up for me before crashing on Arch.

[AndreasAZiegler]

@Dunedan I posted the dump of the report from macOS here https://pastebin.com/UtvGz403 (I hope that's right).

Just for clarification how this report was generated: I booted from the Ubuntu Live USB stick, modified the nvme driver, recompiled the driver, loaded the driver "modprobe nvme", then my laptop crashed after around 10s, in macOS the aforementioned report was presented to me.

[roadrunner2]

Regarding the NVME related resets, there's this hack in the linux driver (drivers/nvme/host/pci.c line 2097 or thereabouts) for an older apple controller:

        /*
         * Temporary fix for the Apple controller found in the MacBook8,1 and
         * some MacBook7,1 to avoid controller resets and data loss.
         */
        if (pdev->vendor == PCI_VENDOR_ID_APPLE && pdev->device == 0x2001) {
                dev->q_depth = 2;
                dev_warn(dev->ctrl.device, "detected Apple NVMe controller, "
                        "set queue depth=%u to work around controller resets\n",
                        dev->q_depth);

Might be worth seeing if this is also needed for the new controller.

[wsy2220]

@roadrunner2

Could somebody provide the output from sudo acpidump?

Here is my acpidump on 15,1: https://gist.github.com/wsy2220/dddb13f3a3649a5cb63bf88e98bde6f6

[AndreasAZiegler]

@roadrunner2 I tried this modification as well but I does not seem to work.

[AndreasAZiegler]

Any new insights?

[Dunedan]

Actually I have an insight, even though it's just a pretty small one:

The crash report includes secure boot?: YES. With enabled secure boot I wouldn't be surprised about those NVMe controller resets. So can you please confirm again that you did disable secure boot in macOS prior to trying to run Linux?

[roadrunner2]

Thanks for the full acpidump. Looks like all of the iBridge2 relevant parts are in the DSDT after all.

I don't have a lot to add here, other than to summarize the device structure around the iBridge2/T2 device a bit. Parts of it have already been mentioned above.

These are the PCI devices exposed by the chip (from lspci):

02:00.0 Mass storage controller [0180]: Apple Inc. ANS2 NVMe Controller [106b:2005] (rev 01) (prog-if 02)
02:00.1 Non-VGA unclassified device [0000]: Apple Inc. Device [106b:1801] (rev 01)
02:00.2 Non-VGA unclassified device [0000]: Apple Inc. Device [106b:1802] (rev 01)
02:00.3 Multimedia audio controller [0401]: Apple Inc. Device [106b:1803] (rev 01)

In the DSDT the corresponding devices are (RP17 is the iBridge2 device itself, as evidenced not only by its children, but also the existence of a _DSM method that returns the "apple-coprocessor-version", just like the ASOC for the previous MBP's):

\_SB.PCI0.RP17
\_SB.PCI0.RP17.ANS2
\_SB.PCI0.RP17.IOBC
\_SB.PCI0.RP17.SEPM
\_SB.PCI0.RP17.ADIO

From the ioreg output this is an extract of the relevants parts of the tree in MacOS:

+-o Root  <class IORegistryEntry, id 0x100000100>
  +-o MacBookPro15,1  <class IOPlatformExpertDevice, id 0x100000114>
    +-o AppleACPIPlatformExpert  <class AppleACPIPlatformExpert, id 0x100000115>
    | +-o PCI0@0  <class IOACPIPlatformDevice, id 0x100000148> 
    | | +-o AppleACPIPCI  <class AppleACPIPCI, id 0x10000038b> 
    | |   +-o RP17@1B  <class IOPCIDevice, id 0x100000376> 
    | |   | +-o IOPP  <class IOPCI2PCIBridge, id 0x1000003c1> 
    | |   |   +-o ANS2@0  <class IOPCIDevice, id 0x100000377> 
    | |   |   | +-o AppleANS2Controller  <class AppleANS2Controller, id 0x1000003c6> 
    | |   |   |   +-o IONVMeBlockStorageDevice@1  <class IONVMeBlockStorageDevice, id 0x100000414> 
    | |   |   +-o IOBC@0,1  <class IOPCIDevice, id 0x100000378> 
    | |   |   | +-o IOBufferCopyController  <class IOBufferCopyController, id 0x1000003d5> 
    | |   |   |   +-o AppleUSBVHCIBCE@80000000  <class AppleUSBVHCIBCE, id 0x1000003ef> 
    | |   |   |   | +-o AppleUSBVHCIPort@80100000  <class AppleUSBVHCIPort, id 0x100000466> 
    | |   |   |   | | +-o iBridge@80100000  <class IOUSBHostDevice, id 0x1000010d0> 
    | |   |   |   | +-o AppleUSBVHCIPort@80200000  <class AppleUSBVHCIPort, id 0x100000467> 
    | |   |   |   | | +-o iBridge FaceTime HD Camera (Built-in)@80200000  <class IOUSBHostDevice, id 0x1000010d3> 
    | |   |   |   | +-o AppleUSBVHCIPort@80300000  <class AppleUSBVHCIPort, id 0x100000468> 
    | |   |   |   | | +-o iBridge ALS@80300000  <class IOUSBHostDevice, id 0x1000010d5> 
    | |   |   |   | +-o AppleUSBVHCIPort@80400000  <class AppleUSBVHCIPort, id 0x100000469> 
    | |   |   |   | | +-o Headset@80400000  <class IOUSBHostDevice, id 0x1000010d2> 
    | |   |   |   | +-o AppleUSBVHCIPort@80500000  <class AppleUSBVHCIPort, id 0x10000046a> 
    | |   |   |   | | +-o Apple Internal Keyboard / Trackpad@80500000  <class IOUSBHostDevice, id 0x100002af2> 
    | |   |   |   | +-o AppleUSBVHCIPort@80600000  <class AppleUSBVHCIPort, id 0x10000046b> 
    | |   |   |   | | +-o iBridge Display@80600000  <class IOUSBHostDevice, id 0x1000010d4> 
    | |   |   |   | +-o AppleUSBVHCIPort@80700000  <class AppleUSBVHCIPort, id 0x10000046c> 
    | |   |   |   | | +-o iBridge DFR brightness@80700000  <class IOUSBHostDevice, id 0x1000010d1> 
    | |   |   +-o SEPM@0,2  <class IOPCIDevice, id 0x100000379> 
    | |   |   | +-o AppleSEPIntelIOP  <class AppleSEPIntelIOP, id 0x1000003d6> 
    | |   |   +-o ADIO@0,3  <class IOPCIDevice, id 0x10000037a> 
    | |   |     +-o BridgeAudioControllerPCI  <class BridgeAudioControllerPCI, id 0x100000567> 

The IOBC device closely matches the old iBridge device (T1 chip), except that it now controls the keyboard and trackpad devices too, and instead of being a USB device hooked to a USB hub it's a PCI device that appears function as a USB hub. So it looks like that to get access to those devices (including the keyboard and trackpad in particular) somebody will need to reverse engineer and write a PCI driver for the IOBC that basically implements a USB VHCI.

[AndreasAZiegler]

@Dunedan I double checked the secure boot settings (turning on the MacBook and pressing Command + R) and secure boot is off. I created a new gist, the content is probably still the same https://gist.github.com/AndreasAZiegler/e85a4ee473e4ddb9b24000d10e5c223f

[dvulricht]

@Dunedan I double checked the secure boot settings (turning on the MacBook and pressing Command + R) and secure boot is off. I created a new gist, the content is probably still the same https://gist.github.com/AndreasAZiegler/e85a4ee473e4ddb9b24000d10e5c223f

You applied the patch to drivers/nvme/host/pci.c and booting the correct kernel ? You can try to boot the kernel with the following additional arguments added manually to linux-image line in grub: acpi_osi=! acpi_osi="Windows 2012"

[Dunedan]

@AndreasAZiegler If I'd look into the NVMe problems (unfortunately I don't have one of the latest MacBook Pros to do so), my next step would be to try to figure out if a certain NVMe command issued by Linux causes the system reset. We already know that the software on the T2 chip triggers an assertion shortly after enabling the NVMe controller, but we don't know why. If it is caused by a specific NVMe command, that'd narrow down the issue quite a bit. I'm afraid doing so is a bit of pita, because you'd need to compile your own kernel with additional logging and would need to capture the logs over network.

[MaxLeiter]

@dunedan: how would I go about capturing the logs over the network? I’m comfortable enough with C that I think I can figure out the logging if the kernel already has the systems for it in place.

[Dunedan]

@MaxLeiter NetConsole (https://www.kernel.org/doc/Documentation/networking/netconsole.txt) should work for that. :slightly_smiling_face:

[maenpaa24]

I would like to share some experiences:

With the latest iso of arch linux as well as the latest stable release of kali linux adding the bootflag "nomodeset" so that it can boot properly, otherwise the screen goes black.

When I boot these isos the NVMe module is already loaded and the drive is shown in lspci. When I execute the command:

echo 106b 2005 > /sys/bus/pci/drivers/nvme/new_id

Nothing happens and the drive is not shown for example executing the command "lsblk". So I can not reproduce the issue with these two distros.

To reproduce the issue I have to make use of the iso linked by @aunali1. Then I can effectively issue the command "echo 106b..." And the drive shows up in "lsblk" and after a few seconds the fan goes crazy and the MacBook turns off.

By the way with the iso provided by aunali1 adding the bootflag "nomodeset" is not required.

[maenpaa24]

I have been trying to debug the issue through netconsole but the drivers of every wireless to usb or rj45 to usb adapter that I have do not support polling. Do you guys know about any that supoorts it?

[zhuowei]

@maenpaa24 Could pstore help get logs from the device?

pstore is a kernel module that stores kernel panics in EFI variables, so they would survive a reboot. mjg59 has a tutorial for enabling it. By default it logs all oops and panics.

You may need to modprobe efi-pstore before using it.

If EFI variables don't work, there's also ramoops, which stores oops/panics/optionally kernel logs in RAM.

There's a guide here.

Basically, boot with mem=2G on the kernel command line, then

modprobe ramoops mem_address=0x80000000 mem_size=0x100000 ecc=1 console_size=0x10000 record_size=0x10000 ftrace_size=0x10000 pmsg_size=0x10000

to enable recording panics to ram.

If you're compiling a kernel, you can also enable CONFIG_PSTORE_CONSOLE to store everything from the kernel console, not just on panic/oops.

I haven't tested these since I don't have a Macbook with a T2 chip, but I did try pstore with ramoops in a virtual machine, and the above modprobe seems to work. It's the same panic log mechanism that Android phones and Chromebooks use, so it should probably work on Macbooks.

[imbushuo]

BTW: StorNVMe.sys chokes at Apple T2 too (but Windows bugchecks instead of T2 crash). Boot Camp supplies a customized Apple SSD driver for Windows.

I believe RAM-based crashdump is not sufficient for this issue, kernel traces via external bus (ThunderBolt-based Ethernet?) or RAM will probably help understanding what happened with the NVMe controller and the driver.

[jantman]

I don't really like to be the one to bump a thread that I haven't been involved in, but I was wondering if there's been any progress on this in the last two months?

I just got a work-issued A1990 (brand new, according to the bottom of the box a 15.4) today and was rather shocked when I did some cursory Googling and found this issue... as I've been running Arch on my MacBooks for ~7 years and was running (quite painfully) Linux on them as far back as PowerPC.

I have the new laptop in my hands and can assist with investigation/dumps/etc... for some amount of time until my corporate IT department demands that I return one of my now-two laptops (I convinced them to let me keep old and new "for a day or two" to transfer things over without losing work time).

[aunali1]

@jantman I would suggest submitting a PR with the output of get-info.sh. The 15,4 variant might have some hardware differences that would warrant a dedicated directory.

EDIT: Can you also upload as a comment here the zip file generated from $ sudo sysdiagnose -c. I've noticed that there are some new T2 codenames, j780ap (iBridge2,4) and j174ap (iBridge2,5)

[aunali1]

@Dunedan @roadrunner2 I've been perusing dumps from sysdiagnose -c (undocumented) and there are some really interesting bits in the ioreg dumps. These are performed on the T2 itself and are used for Apple 's diagnostics. Here it is: bridge_sysdiagnose_2019.03.13_04-26-03+0000_Bridge_OS_Bridge_16P2542.tar.gz

And the ioreg dumps for your convenience: IODeviceTree.txt IOPower.txt IOService.txt

[Serizao]

Hello all i have 2018 MBP with T2 chip if you want result of test i wil be happy to help you

[FrozenDroid]

Same here, also have the 2018 MacBook Pro 15". How could I contribute?

[aunali1]

Finally got a setup going to sniff the traces between the T2's Bridge Controller and a QEMU Windows 10 instance. Not an expert in PCI drivers but I have uploaded the traces for those interested, or willing to help. trace.txt

[zhuowei]

@aunali1 It looks like the trace only logs reads/writes to the config area: maybe NVMe transactions are DMAed and thus not present in the trace?

I wonder if reverse engineering the Windows Boot Camp NVMe driver or the T2 firmware would be a better way to figure out what's going on.

[brianbeardall]

I also need to follow this. The keyboard, touchpad, touchbar, ssd, and anything touching the T2 chips is not working even with security disabled. At least the USB, AMD GPU, charging, cpu controls work on Linux 5.0. The laptop doesn't run any warmer for me than in OSX. So in order to get everything working will likely require some level of T2 setup to unlock the other devices.

[KenaiTheWolf]

Is this still being looked at? How can I help. If I can locate a binary of the T2 driver would that help at all?

[coquizen]

I am also available to help. What can I do?

[gbrow004]

Nothing to add, other than I am interested to know if anyone has made in progress. I have a mid-2018 15" macbook pro, but I'm not really a programmer. Happy to try any suggestions to get more information.

[TRPB]

Same here. I'm happy to help provide info or test stuff out, been given one of these by work and it's collecting dust until I can install a proper OS on it.

[vamsi360]

Hi all, Is there any mailing list where any of the devs are working on getting the NVME drive to work?

[zhuker]

@vamsi360 seems like nobody is working on it :-/

[mikroskeem]

If someone wants Windows drivers for RE-ing, contact with me (peek into my homepage and find convenient way to contact me). I have Windows 10 installed via boot camp :)

[MCMrARM]

I attempted to get the SSD working today and failed. Though I might have collected some more interesting information.

First of all, the laptop does not crash and reboot on the latest Linux version. I downloaded a few Arch versions, and the last version where the crash happens is 2018/08. 2019/09 doesn't let us see the drive and doesn't crash the PC. Hmmm. dmesg shows IO queues not created as an error, that doesn't happen in 2018/08. I tracked down this to be related to this code change: Basically, adapter_alloc_cq returns 2, which means NVME_SC_INVALID_FIELD I think. Old code didn't catch this as an error and continued, new one catches it and fails.

And also, why does the bridgeOS crash in the first place on the 2018/08 image? I looked at the crash report, I don't think anyone pasted the error message, but it was this for me: panic(cpu 1 caller 0xfffffff01c74a81c): ANS2 Recoverable Panic - assert failed: [7437], src/drivers/apple/ans2/nvme_drv.c:395:SQ3 (Host 1) DB error. status_reg: 0x8, tail: 0 - Cmd(12)\n assert failed: [7437], src/drivers/apple/ans2/nvme_drv.c:395:SQ3 (Host 1) DB error. status_reg: 0x8, tail: 0

Ok, so then I started to try to figure out why does adapter_alloc_cq fail so I can use a newer kernel. I have got the drivers from both MacOS and from Windows and loaded them up in IDA. The MacOS driver is called IONVMeFamily.kext and the Windows one is called AppleSSD.sys. The OS X one has symbols so it's more helpful for reverse engineering. I looked up how the CQ creation code looked like and it was basically identical to the Linux one. However the OS X one hardcoded the interrupt vector to 0, unlike the Linux one which chooses it dynamically (as device interrupt count = 2). After changing this, the call no longer failed. However, after 10 seconds the laptop rebooted, this time with this panic text (which I read from macOS): panic(cpu 0 caller 0xfffffff01b74a81c): ANS2 DATA ABORT pc=0x00000000000d8654 Exception class=0x25 (Data Abort taken without a change in Exception level), IL=1, iss=0x4 far=0x00000010003ccef4 far_physical=000000000000000000 - Cmd(12)\nRTKit: RTKit_iOS-848.70.3.release - Client: t8012.release-AppleStorageProcessorANS2-177.77.71~1~177.77.71~1\nTime: 0x000000007e80cb68 which is a bit ????.

As for the kernel change, I used kernel v4.18.5 as base (commit 96158f3a9e7027056a2e35cff5212434294b3b34) and patched it like this (there's only a single queue anyways, so it is fine):

Collapsed diff

diff --git a/drivers/nvme/host/pci.c b/drivers/nvme/host/pci.c
index ddd441b1516a..77a65a16ad34 100644
--- a/drivers/nvme/host/pci.c
+++ b/drivers/nvme/host/pci.c
@@ -1479,7 +1479,7 @@ static int nvme_create_queue(struct nvme_queue *nvmeq, int qid)
         * A queue's vector matches the queue identifier unless the controller
         * has only one vector available.
         */
-       vector = dev->num_vecs == 1 ? 0 : qid;
+       vector = 0; // dev->num_vecs == 1 ? 0 : qid;
        result = adapter_alloc_cq(dev, qid, nvmeq, vector);
        if (result)
                return result;

[rnsc]

@MCMrARM nice! You mentioned kernel 4.18.5. Did you try with a 5.x kernel?

[MCMrARM]

Haven't tested it yet while writing that message, but I did just now and sadly it also crashed. I tried 5.1.5 to be specific, similar ANS2 DATA ABORT panic from the bridgeOS.

[zhuowei]

@MCMrARM Your second crash looks like the second crash report AndreasAZiegler pasted above: https://github.com/Dunedan/mbp-2016-linux/issues/71#issuecomment-430719127 but not their first crash (https://github.com/Dunedan/mbp-2016-linux/issues/71#issuecomment-427662920)

[MCMrARM]

Also does anyone know if it's possible to dump the bridgeOS firmware and kernel? That could turn out to be pretty useful...

[KenaiTheWolf]

Don’t think it’s possible and even if it is, BridgeOS is probably highly encrypted due to the amount of sensitive data it handles. On Jun 30, 2019, 3:38 PM -0600, MCMrARM [email protected], wrote:

Also does anyone know if it's possible to dump the bridgeOS firmware and kernel? That could turn out to be pretty useful... — You are receiving this because you commented. Reply to this email directly, view it on GitHub, or mute the thread.

[zhuowei]

@MCMrARM The bridgeOS kernel is not encrypted (you can grab it from https://www.theiphonewiki.com/wiki/Firmware/iBridge/3.x), but the firmware (RTKit) that actually drives the NVMe does seem to be encrypted. At least, I couldn't find any of the RTKit-related strings from the crash log inside the kernel image.

[MCMrARM]

I think I made a rather important finding. I found out that it's the read operation which crashes the controller (admin operations work just fine). I started reverse engineering the read method and found it's wrapped by AppleNVMeRequest::BuildCommandReadStruct(NVMeRWCommand *) in order to attach a decryption key there. I believe it's for offloading encryption to the bridgeOS chip, but that doesn't really matter for us much right now. What caught my attention was that they seemed to attach extra data to the end of the NVMe command, and that'd like, require making the command data larger or something. I saw a debug print in the code for creating queues, so I booted up OS X and checked the log:

$ log show --predicate 'processID == 0' --last 1h | grep NVMe
2019-07-01 16:05:18.726267+0200 0x197      Default     0x0                  0      0    kernel: (IONVMeFamily) AppleNVMe Assert failed: ( 0 != data )
2019-07-01 16:05:18.726270+0200 0x197      Default     0x0                  0      0    kernel: (IONVMeFamily) ReleaseIDNode
2019-07-01 16:05:18.726272+0200 0x197      Default     0x0                  0      0    kernel: (IONVMeFamily) file: /BuildRoot/Library/Caches/com.apple.xbs/Sources/IONVMeFamily/IONVMeFamily-356.71.1/IONVMeController.cpp
2019-07-01 16:05:18.726276+0200 0x197      Default     0x0                  0      0    kernel: (IONVMeFamily) line: 5429
2019-07-01 16:05:18.726278+0200 0x197      Default     0x0                  0      0    kernel: (IONVMeFamily) 
2019-07-01 16:05:18.727490+0200 0x197      Default     0x0                  0      0    kernel: (IONVMeFamily) IOReturn IONVMeController::CreateSubmissionQueue(uint16_t, uint8_t)::2745:SQ index=0 entrysize=64
2019-07-01 16:05:18.727495+0200 0x197      Default     0x0                  0      0    kernel: (IONVMeFamily) IOReturn IONVMeController::CreateSubmissionQueue(uint16_t, uint8_t)::2745:SQ index=0 entrysize=64
2019-07-01 16:05:18.727502+0200 0x197      Default     0x0                  0      0    kernel: (IONVMeFamily) IOReturn IONVMeController::CreateSubmissionQueue(uint16_t, uint8_t)::2745:SQ index=1 entrysize=128
2019-07-01 16:05:18.727507+0200 0x197      Default     0x0                  0      0    kernel: (IONVMeFamily) IOReturn IONVMeController::CreateSubmissionQueue(uint16_t, uint8_t)::2745:SQ index=1 entrysize=128

Uhm, queue nr 1 is twice as big, ugh. I don't think the Linux kernel supports that :/

[MCMrARM]

So yeah, only admin commands work (as they are done over queue number 0). Queue 1 is absolutely messed up, I'm trying to create a patch for this, but this is going to be really difficult to sanely upstream.

[MCMrARM]

Hey, that's a partition table! (Not a screenshot from my MBP, because I did it over SSH. But still, that's some progress! Read works at all and the laptop hasn't rebooted yet!)

[TRPB]

Wow, great work. Please provide a patch if you get it working! Assuming you get the NVME drive to work, what's the status of the keyboard and touchpad?

[MCMrARM]

The EFI partition mounted just fine as well. So the real issue right now is how to get this working in a sane way, that could be upstreamed. What I got working right now is a hack, I'll clean it up if anyone wants to try it out.

The main issue right now is that the Linux kernel in no way supports non-0x40 sized SQ command queues. This would have to be added. I have no idea if such a patch would even be accepted upstream (I never submitted a kernel patch before either).

[MCMrARM]

@TRPB The keyboard and mouse do not work. I don't know how hard is it to get them working, I did not look into that yet.

[TRPB]

The main issue right now is that the Linux kernel in no way supports non-0x40 sized SQ command queues. This would have to be added. I have no idea if such a patch would even be accepted upstream (I never submitted a kernel patch before either).

I know nothing about the linux kernel or driver development so this may be a stupid suggestion, but would it be possible to put a translation function in the driver that supplies the 0x40 size to the kernel but translates the command for talking to the hardware?

[zhuker]

@MCMrARM great work! Kudos

[MCMrARM]

Kernel patch for Linux v5.1.5

diff --git a/drivers/nvme/host/pci.c b/drivers/nvme/host/pci.c
index a90cf5d63aac..6328f4a62918 100644
--- a/drivers/nvme/host/pci.c
+++ b/drivers/nvme/host/pci.c
@@ -541,7 +541,12 @@ static void nvme_submit_cmd(struct nvme_queue *nvmeq, struct nvme_command *cmd,
 			    bool write_sq)
 {
 	spin_lock(&nvmeq->sq_lock);
-	memcpy(&nvmeq->sq_cmds[nvmeq->sq_tail], cmd, sizeof(*cmd));
+	if (nvmeq->qid == 1) {
+		memcpy(&nvmeq->sq_cmds[nvmeq->sq_tail * 2], cmd, sizeof(*cmd));
+		memset(&nvmeq->sq_cmds[nvmeq->sq_tail * 2 + 1], 0, sizeof(*cmd));
+	} else {
+		memcpy(&nvmeq->sq_cmds[nvmeq->sq_tail], cmd, sizeof(*cmd));
+	}
 	if (++nvmeq->sq_tail == nvmeq->q_depth)
 		nvmeq->sq_tail = 0;
 	nvme_write_sq_db(nvmeq, write_sq);
@@ -1376,11 +1381,15 @@ static void nvme_free_queue(struct nvme_queue *nvmeq)
 	if (!nvmeq->sq_cmds)
 		return;
 
+	int size = SQ_SIZE(nvmeq->q_depth);
+	if (nvmeq->qid == 1)
+		size *= 2;
+
 	if (test_and_clear_bit(NVMEQ_SQ_CMB, &nvmeq->flags)) {
 		pci_free_p2pmem(to_pci_dev(nvmeq->q_dmadev),
-				nvmeq->sq_cmds, SQ_SIZE(nvmeq->q_depth));
+				nvmeq->sq_cmds, size);
 	} else {
-		dma_free_coherent(nvmeq->q_dmadev, SQ_SIZE(nvmeq->q_depth),
+		dma_free_coherent(nvmeq->q_dmadev, size,
 				nvmeq->sq_cmds, nvmeq->sq_dma_addr);
 	}
 }
@@ -1466,8 +1475,12 @@ static int nvme_alloc_sq_cmds(struct nvme_dev *dev, struct nvme_queue *nvmeq,
 {
 	struct pci_dev *pdev = to_pci_dev(dev->dev);
 
+	int size = SQ_SIZE(depth);
+	if (nvmeq->qid == 1)
+		size *= 2;
+
 	if (qid && dev->cmb_use_sqes && (dev->cmbsz & NVME_CMBSZ_SQS)) {
-		nvmeq->sq_cmds = pci_alloc_p2pmem(pdev, SQ_SIZE(depth));
+		nvmeq->sq_cmds = pci_alloc_p2pmem(pdev, size);
 		nvmeq->sq_dma_addr = pci_p2pmem_virt_to_bus(pdev,
 						nvmeq->sq_cmds);
 		if (nvmeq->sq_dma_addr) {
@@ -1476,7 +1489,7 @@ static int nvme_alloc_sq_cmds(struct nvme_dev *dev, struct nvme_queue *nvmeq,
 		}
 	}
 
-	nvmeq->sq_cmds = dma_alloc_coherent(dev->dev, SQ_SIZE(depth),
+	nvmeq->sq_cmds = dma_alloc_coherent(dev->dev, size,
 				&nvmeq->sq_dma_addr, GFP_KERNEL);
 	if (!nvmeq->sq_cmds)
 		return -ENOMEM;
@@ -1561,7 +1574,7 @@ static int nvme_create_queue(struct nvme_queue *nvmeq, int qid, bool polled)
 	 * has only one vector available.
 	 */
 	if (!polled)
-		vector = dev->num_vecs == 1 ? 0 : qid;
+		vector = 0;
 	else
 		vector = -1;
 

This patch is provided with no guarantee, responsibility or liability. Use at your own risk. I have only tested file system reading, not writing, write should work just fine but you are taking the risk yourself, as it might brick the laptop just as well.

[MCMrARM]

Write seems to work fine....I'm trying to get Linux to show as a boot entry now.

[gbrow004]

@MCMrARM Terrific effort! Your'e my new hero!

[mikroskeem]

Current notes: installing GRUB/systemd-boot fails as touching EFI variables sets kernel on fire (read: panics).

To install Arch, I (and MCMrARM) used following solution:

1. pacstrap to /mnt like usual
2. instead of using arch-chroot, use systemd-nspawn -D /mnt -b
3. use bootctl (well MCMrARM went with systemd-boot as well for now)
4. exit systemd-nspawn environment by doing poweroff
5. reboot

Actually getting the boot entry show up hasn't yet achieved, I bet using bless solves this.

[FrozenDroid]

Amazing work @mikroskeem @MCMrARM Where do I donate?

[mikroskeem]

We set up a Discord server where we expect more people to join the discussion: https://discord.gg/Jayz5f5

[KenaiTheWolf]

Why are you looking to exclude people who don’t want to join your discord? You don’t even own this repo.

[KenaiTheWolf]

It needs to stay on GitHub because when people search the Internet trying to actually do this this is the result that shows up.

[mikroskeem]

It's not indended for moving all the convo there, I'm thinking about moving smaller conversation away from Github issue tracker as it'll constantly spam e-mails to subscribers. If you find that this is really bad idea then sure ¯\_(ツ)_/¯ react 👎 to this reply and I'll delete it.

[FrozenDroid]

It's not indended for moving all the convo there, I'm thinking about moving smaller conversation away from Github issue tracker as it'll constantly spam e-mails to subscribers. If you find that this is really bad idea then sure ¯_(ツ)_/¯ react 👎 to this reply and I'll delete it.

Nah, I think that's a fine idea :) There'll always be people who are against certain services.

[KenaiTheWolf]

I’m subscribed via email so I can get the conversation as it updates and go through on my personal time.

[Dunedan]

There is also a Gitter-chatroom for this repo (as linked in the README), which is accessible without the need to login and requires only an Github account for writing: https://gitter.im/mbp-2016-linux/Lobby

[MCMrARM]

I have no plans to leave this channel for discussions, so dw.

[KenaiTheWolf]

The scope of this isn’t just the MacBooks anymore though. It’s all future macs.

[rnsc]

@MCMrARM @mikroskeem thank you for your hard work and dedication to this! And thank you for sharing all that precious information with us!

I personally do agree that having a chat system aside is a good idea, it also sometimes help to have more quick feedback. As long as a summary of the findings is posted here, it's perfectly fine, imho.

[MCMrARM]

I have managed to install Arch on a real MBP on the internal SSD. Used Grub as the bootloader. Seems to work mostly fine, have yet to use it for extended periods of the time.

Next thing I have to investigate is how the bridge itself works so we could get keyboard to work. Though that sounds difficult and I might not be able to get it to work.

[NCLI]

Did you get the boot entry to show up as well?

[MCMrARM]

Yes, the boot entry shows fine.

I have done some reverse engineering on the Bridge PCI device, which is required to get the keyboard to work as far as I aware.

The keyboard is attached to a VHCI controller, the driver object for it is AppleUSBVHCIBCE. BCE stands for BufferCopyController. And BufferCopyController is, as far as I can see, an Apple proprietary PCIe device spec. So that has to be reverse engineered first.

Here's some initial documentation on it I have made: https://mrarm.io/u/Paste%202019-07-03%2019-56-58.txt ^ The information provided here are probably half the information you would need to create a BufferCopyController driver (however, on top of that an AppleUSBVHCIBCE driver will have to be implemented and that is also going to be very difficult, that's not part of the 50% done RE-side thing sadly). I sadly never wrote any Linux drivers, nor am I familiar with the PCIe spec heavily. If anyone knows someone who could help me and explain some things to me (has knowledge of PCIe in general and preferably Linux or OS X PCIe driver development), I'd appreciate help.

The AppleUSBVHCIBCE isn't the only thing attached to the BufferCopyController. Drivers from both kernel- and user- space can request a SQ+CQ pair and then somehow tell the BridgeOS what is their purpose. I haven't reverse engineered that part deeply enough yet, so I don't really know the details.

[Hi-Angel]

@MCMrARM I hope this will cheer you up: you're all over the news.

If anyone knows someone who could help me and explain some things to me (has knowledge of PCIe in general and preferably Linux or OS X PCIe driver development), I'd appreciate help.

Note, that you can ask other kernel devs on mailing lists, or chat with them on IRC server OFTC, channel #kernelnewbies.

[amluto]

The EFI partition mounted just fine as well. So the real issue right now is how to get this working in a sane way, that could be upstreamed. What I got working right now is a hack, I'll clean it up if anyone wants to try it out.

The main issue right now is that the Linux kernel in no way supports non-0x40 sized SQ command queues. This would have to be added. I have no idea if such a patch would even be accepted upstream (I never submitted a kernel patch before either).

I'm not entirely the right person, but I've contributed to the upstream NVMe driver, so here goes:

Unless I read it wrong, the NVMe spec is really crystal clear that submission queue entries are 64 bytes. Are you sure that the Mac queue 1 entries are 128 bytes, or is it possible that they are fused entries? Given how you're handling the indexing, it kind of looks like they're really 128 bytes.

The driver maintainers would be very likely to accept a patch that complied with the spec, but you're doing something that looks inconsistent with the spec. Do you think you could cleanly implement it as a quirk? It might also be important to avoid hurting performance on unaffected hardware.

[MCMrARM]

@amluto They are certainly 128 bytes, not fused entries. This is because they added that space to give space for an encryption key, so they can offload the encryption. It's a hack they did. I'm not sure about the right way to implement this. Apple's kernel driver simply has a member which decides how big the queue elements should be. That probably would be the cleanest way to implement this. I don't think the nvme_submit_cmd function is a hot path, so it shouldn't affect performance in any significant way.

[MCMrARM]

I have done some more reverse engineering on the IOBufferCopyController kernel driver: https://mrarm.io/u/Paste%202019-07-05%2013-24-16.txt This should be enough to implement a basic version of that driver, though this doesn't touch the USB protocol used for it, so that's what I am going to do next.

Also, if anyone is interested in getting the IDBs for the IOBufferCopyController and IOBufferCopyEngine kexts, with some structures RE'd and some decompiled code cleaned up, then send me a message.

[robin7g]

Awesome work BTW! Sorry to ask a newbie question. When you were diagnosing the SSD issue were you booting an actual MacBookPro15,1 from an external drive or were you building the Kernel on a different machine and then creating a separate install USB? I have an MBP15,1 happy to lend a hand.

[MCMrARM]

I built the kernel on my host PC and then booted the kernel off an USB drive to which I copied the kernel each time (it was writable). There might have been a faster way, but note that the stupid ANS2 controller crashed the whole laptop, so reboots weren't really avoidable, so editing the kernel on the MBP itself would be a huge pain.

[gbrow004]

Anyone want to write a guide for the adventurous but novice at this kind of stuff? I've got my stuff all backed up and not afraid of bricking my machine. Even got a partition on the SSD ready-ish (currently a free exfat partition).

[MCMrARM]

IOBufferCopyController RE document revision 4 (minor changes compared to r3): https://mrarm.io/u/Paste%202019-07-08%2022-41-49.txt VHCI RE document revision 1: https://mrarm.io/u/Paste%202019-07-08%2022-42-49.txt - Still missing a lot of stuff required to get it to work, decided to post it because it still has quite a bit of information that could be useful to others.

Again, if anyone wants an IDB of any of the KEXTs I'm reverse engineering with quite a bit of structures RE'd, message me somehow.

[axboe]

@MCMrARM Unless you can use the id->sqes field to properly set this up, and I don't think you can since it seems dependent on the qid, you'd have to base this on some sort of pci-id list.

nvme_submit_cmd() is very much a hot path, it's called for every command. Doesn't get much hotter than that...

[axboe]

If it truly is some weirdness with sqe+key and specific to qid 1, a better hack to support this might be to just not use qid 1.

[MCMrARM]

I mean the device only uses two queues. The command queue and a single submission queue - qid 1. No other queues. So it's not really specific to qid 1, just to the submission queues.

It's a "hot code path" but storing the size there, IMHO shouldn't incur a performance hit as the code is not really completely cheap anyways.