FlyGoat
Implement BAR relocation
If devices have above 4GiB BARs, they should be relocated (and possibly resized) to be below 4GiB.
This would fix, for example on my machine, the XHCI controller not being able to work. Possibly other things, but it should be done either way.
I actually tried to implement it and I found it's a mission impossible.
The size of low PCI hole is design to be a tight fit of total bar requirement on many platforms. i.e. UEFI firmware would scan all PCI device and calculate BAR requirement first, then set TOLUD based on total demand. This leaves no space for us to relocate anything to low 4G.
I think a more realistic approach would be implementing protected mode SeaBIOS, so we can use PAE to access higher MMIO space...
Ah I see. Well, that's a shame.
I guess CSMWrap could still try to do it, if there is enough low 4GiB space left over for PCI, maybe?
I want to check what other devices besides XHCI and video have above 4GiB bars. Maybe it's not too bad...
Isn't the XHCI issue a bug with SeaBIOS ? It should be able to handle 64-bit BAR ?
I did see around 2022 have done some work for 64-bit BARs but that's only allocating them ig.
Still this would be useful for 32-bit OS I guess.
@xCuri0 There is no 64bit flat mode, so supporting large physical address space would be challenging in SeaBIOS.
It's a shame that Wrap32Dis is an AMD only feature, otherwise we can use it to access extended address space.
Shouldn't this issue be reported to SeaBIOS, it's not really a CSMWrap issue ? Seems like they want to support 64-bit BARs since recently so I think it would be valid.
Probably will take a bit of effort to for a wrapper around the 64-bit MMIO writes to thunk into protected mode and setup all the stuff it needs.
I mean, in general this is only an issue if "Above 4G Decoding" is not disabled, which the newly contributed docs as well as the readme make it sound like it's mandatory.
Problem is, a lot of motherboards do not have a way to disable "Above 4G Decoding" at all, especially those not providing a CSM, which is why I think mandating having it disabled is a bit silly.
While having SeaBIOS support accessing 64-bit BARs for its own drivers is not a bad idea, nor something I oppose at all, the true solution would be to implement this relocation to low memory to effectively disable "Above 4G Decoding" after boot. Though that said, this may be exceedingly complicated if not borderline impossible, so the next best thing is the SeaBIOS support solution.
Makes sense, but I don't think SeaBIOS is in active development atm. So if we want it, we need to do it on our own and then consider upstreaming.
Makes sense, but I don't think SeaBIOS is in active development atm. So if we want it, we need to do it on our own and then consider upstreaming.
Thing is, before committing to doing that, I'd make absolutely sure that BAR relocation is impossible. Because even adding support for accessing above 4G BARs to SeaBIOS won't improve the framebuffer-above-4G situation and access to devices above 4G from 32-bit only OSes...
@mintsuki it's a huge pain moving BARs, setting up bridges and all that even ignoring the issues of no space.
Even most of the oldest systems with proper UEFI (LGA1155) don't expose TOLUD and have it as small as possible, I think the only free spaces in it are to keep BARs aligned.
I also found out that some AMD GPUs (my RX 580 atleast) do not like having their BAR downsized and booting into Windows (error 43), though Linux/macOS work fine. Maybe they need an additional step to work (I tried both OpenCore and my own EFI application).
Have you checked if there's even an aligned 256MB available for the GPU framebuffer in the 32-bit PCIe memory region on your system if everything except Intel ME was moved into 64-bit or disabled ?
I understand it being a huge pain, but realistically it's the only way to have CSMWrap be usable in the long run. Sadly I did not check because I don't actually have any PCI expertise nor am I even terribly familiar with PCI in general, I only get the broad strokes.
I understand it being a huge pain, but realistically it's the only way to have CSMWrap be usable in the long run. Sadly I did not check because I don't actually have any PCI expertise nor am I even terribly familiar with PCI in general, I only get the broad strokes.
You can check bus ranges from Linux dmesg and substrate BARs visible in lspci.
I understand it being a huge pain, but realistically it's the only way to have CSMWrap be usable in the long run. Sadly I did not check because I don't actually have any PCI expertise nor am I even terribly familiar with PCI in general, I only get the broad strokes.
Send output of /proc/iomem (as root otherwise doesn't show everything).
Windows Device Manager can show the allocations too but not everything so Linux would be better
I've been discussing this with people who know way more than I do about PCI and this stuff, and basically the consensus that was reached was that:
- There is indeed enough space to map most stuff.
- Resizable BAR is actually good because we can resize the GPU's BAR to fit in low memory.
- Doing this isn't particularily hard, it just requires scanning PCI low memory and properly fitting there any BAR that may be above 4G, resizing the GPU BAR if necessary.
- The idea that 4G minus TOLUD is as small as possible and nothing else fits there is largely unsubstantiated, and the iomem scan that I posted proves that that's not necessarily or always the case.
In any case, today we may try to implement this and open a PR if we get it working. This effectively undoes "Above 4G Decoding" for any machine that has no knob to disable it (which is most of the ones CSMWrap targets).
I've been discussing this with people who know way more than I do about PCI and this stuff, and basically the consensus that was reached was that:
- There is indeed enough space to map most stuff.
- Resizable BAR is actually good because we can resize the GPU's BAR to fit in low memory.
- Doing this isn't particularily hard, it just requires scanning PCI low memory and properly fitting there any BAR that may be above 4G, resizing the GPU BAR if necessary.
- The idea that 4G minus TOLUD is as small as possible and nothing else fits there is largely unsubstantiated, and the iomem scan that I posted proves that that's not necessarily or always the case.
In any case, today we may try to implement this and open a PR if we get it working. This effectively undoes "Above 4G Decoding" for any machine that has no knob to disable it (which is most of the ones CSMWrap targets).
Intel ARL platform has a problem is the csmwrap can not create correct ACPI table, will make the system A5 BSOD, even use win10/win11.
I've been discussing this with people who know way more than I do about PCI and this stuff, and basically the consensus that was reached was that:
1. There is indeed enough space to map most stuff.
This is good news! I'm glad that there is a solution for your board.
2. Resizable BAR is actually _good_ because we can resize the GPU's BAR to fit in low memory. 3. Doing this isn't particularily hard, it just requires scanning PCI low memory and properly fitting there any BAR that may be above 4G, resizing the GPU BAR if necessary.
I have a couple of tips for implementation, it's a pity that I deleted by my code after discover BAR relocation is not practical on most systems. I copied allocation logic from U-Boot and uses EFI_PCI_ROOT_BRIDGE_IO_PROTOCOL.Configuration for usable range.
- The final relocation step can only be done after ExitBS, my implementation takes a two-step approach, in first stage it only performs scans and pre-allocation but don't touch actual BAR. The second stage, after ExitBS will do the actual allocation.
- There are some BARs not really movable, such as iGPU GFX Stolen Memory BAR and ME BAR, we need to operate a blacklist.
4. The idea that 4G minus TOLUD is as small as possible and nothing else fits there is largely unsubstantiated, and the iomem scan that I posted proves that that's not necessarily or always the case.
Unfortunately, this is the case for many systems. On Intel platforms, this is controlled by platform Fsp configuration MmioSize and MmioSizeAdjustment. For default value both 0 it will try to shrink low PCI Hole as small as possible. In your case I guess your vendor changed the default value (or maybe Intel changed behaviour for Arrow Lake Fsp).
But as we do have #6, I guess there are many new platforms in wild do have extra Mmio space.
On AMD systems, this is handled by EFI_PCI_HOST_BRIDGE_RESOURCE_ALLOCATION_PROTOCOL. Where platforms will do a pre-allocation stage before actually commit to resources.
In any case, today we may try to implement this and open a PR if we get it working. This effectively undoes "Above 4G Decoding" for any machine that has no knob to disable it (which is most of the ones CSMWrap targets).
Also I would like to remind that when you are sizing the BAR you should raise TPL and operate hardware directly use ioports or MCFG instead of PciIo protocols. PciIo protocols have some side affect that may hang your system if you touch the BAR.
Also I would like to remind that when you are sizing the BAR you should raise TPL and operate hardware directly use ioports or MCFG instead of
PciIoprotocols.PciIoprotocols have some side affect that may hang your system if you touch the BAR.
I mean, yeah, we were gonna do this all after exiting boot services anyways, using either IO ports or ECAM MMIO space.
There are some BARs not really movable, such as iGPU GFX Stolen Memory BAR and ME BAR, we need to operate a blacklist.
Why is that? Why can they not be moved?
There are some BARs not really movable, such as iGPU GFX Stolen Memory BAR and ME BAR, we need to operate a blacklist.
Why is that? Why can they not be moved?
Wired platform constraints. They are not so standard on chip PCI device, moving those BARs will render your system unresponsive.
My recommendation would be only performing relocation for some known device classes.
PciRootBridgeIo protocol should be fine, it's lower level than PciIo.
Also FYI the gop driver freezes if you try to move the GPU BAR before disconnecting it.
SeaBIOS also has it's own PCI allocator, could we use that after reading the 32-bit PCI MMIO range from ACPI ?
It's not used in the CSM codepath but maybe a few changes could fix that it would make this a lot easier.
Update: we've done some work to get PCI relocation up and running, could anyone interested please try this:
NOTE: The .gz extension is fake, please just rename the file to omit it.
@Canonkong @FlyGoat @xCuri0
Update: we've done some work to get PCI relocation up and running, could anyone interested please try this:
NOTE: The .gz extension is fake, please just rename the file to omit it.
On Z890 mainboard, USB still can not work, even I use PCI-E to EHCI USB2.0 cards. But all sata ports can works now, my Z890 have 8 sata ports from the PCH. And ot still A5 BSOD or acpi BSOD with win7/8/8.1/10/11.
Not completely sure about this but IIRC the ACPI should tell which devices can safely have BARs moved and which can't, because moving stuff like Intel ME used by SMM will cause crash. Linux uses this when reallocating BARs such as when the amdgpu driver does it or pci=realloc.
Not completely sure about this but IIRC the ACPI should tell which devices can safely have BARs moved and which can't, because moving stuff like Intel ME used by SMM will cause crash. Linux uses this when reallocating BARs such as when the amdgpu driver does it or pci=realloc.
I was told by an ACPI expert that ACPI provides no such information. In any case you're welcome to try and help by opening PRs against my fork at https://github.com/mintsuki/csmwrap, in the pci-reloc branch.
Well I tried this new version (after seeing that patching the BIOS to disable Above 4GB MMIO seems impossible) and... It still freezes at CALL16 unfortunately, once again. You still do a good job guys, I hope it's gonna be fixed one day...
