Re-upstreaming and maintainership of the KGPE-D16 - Cost: 30 000E and counting

[tlaurion]

There was a proposition to do required work under refused grant application. If there is interest from third party to fund conjointly needed work, board could be maintained and reupstreamed, under u-bmc, coreboot and under Heads.

Edit:

• Proposition from 3mdeb here

• History here

• ~Possible funding path here~

  • EDIT: Funded by Immunefi https://github.com/osresearch/heads/issues/719#issuecomment-893024079
  • Work plan: https://github.com/osresearch/heads/issues/719#issuecomment-921877456

• Trail of 4.11 reported problems: https://github.com/osresearch/heads/issues/719#issuecomment-995879031

Last updates, newsletter, code source drop : https://github.com/osresearch/heads/issues/719#issuecomment-966729133

[tlaurion]

Traces: u-bmc kgpe-d16 actual board status

This board works as a QubesOS server while work is still needed. Freshly landed QubesOs-network-server code

A followup on https://github.com/osresearch/heads/issues/717#issuecomment-625825019

[Tonux599]

@tlaurion did 3mdeb decide not to take it on? It would be good to have as it's the only completely blob free board I can think of that's qubes compatible. Provided a 6200 series is used.

[tlaurion]

@Tonux599 3mbeb is absolutely willing to do the work.

[tlaurion]

@Tonux599 @mikebdp2

NlNet decided to not take it on (by funding the project's work). Q: Will we decide to take it on and make it happen? Let's see. :)

[tlaurion]

Applicable?

[tlaurion]

@pietrushnic

[pietrushnic]

@tlaurion sorry, for not replying, recently we have quite a lot on my plate. Definitely, we would like to publish here documents that we sent to NLNet while applying for a grant. There is also a slightly modified version that we provided to Insurgo on Slack. Both documents rely on coreboot state in December 2019. @miczyg I think you can push grant content to 3mdeb GitHub repo as PR and link here for review.

[miczyg1]

This Pull Request contains the application 3mdeb sent to NLNet to obtain the grant: https://github.com/3mdeb/kgpe-osf/pull/1 Feel free to comment and review.

[Tonux599]

@pietrushnic @miczyg1 When should we expect an outcome? Also could you briefly outline any ways that members of the community could support you if or if not you get funding?

[pietrushnic]

@Tonux599 there is no timeline since there is no official commitment from anyone to fully found the effort. We get some good words from various individuals and organizations, but that's all. @tlaurion mentioned that he can sponsor part of the effort, but we are not sure anyone would be happy with half-finished stuff. Vikings Gmbh sent us hardware so we have something to work on. NLNet did not approve of this grant, so I would not expect it will change in the future - I may be the wrong here. To sum the status of this project is "not founded" so we even do not put that on schedule and worry more about founded effort e.g. TrenchBoot and fwupd for Qubes OS.

How the community can help? We are very open here, whatever works for the community.

1. We can split this project and drive as an open and free contribution if there are developers and testers who willing to support the development and debugging. In such a situation we will support whoever will do that as much we can including, coordination review and help in merging code upstream.
2. We can open any channel that the community is comfortable with to try to gather remaining founds. @tlaurion suggested OpenCollective, we can align probably to any suggestion - we have no problem in handling the administrative part.
3. We have a PayPal donation button on our website, although this is probably one of the worst options.
4. We are fine with commercial assignments if anyone is willing to go that path. It can be even a very small part of the task, but at least it will move us forward.
5. If we can sell a platform in some form or bundle services, which will generate margin, then we can use it to bring back the platform to the mainline.

We would like to help, but without funding, we will do the best effort and it may take very long to bring back this platform. Please note our goal is not just to bring back the platform to mainline, but create a structure in which long term maintenance would not cost a lot. We did that for PC Engines routers for over 4 years.

[tlaurion]

@Tonux599 : For the moment, step would be to get people interested in the port. Let's remember that the KGPE-D16:

• was funded by Libreboot and put upstream by RaptorEngineering but previous work did not receive enough attention to be maintained and got dropped in coreboot 4.11 release for lack of maintainership.
• the community crowdfunded the port of the BMC chip to OpenBMC and that work wasn't maintained.
• NlNet refused to fund the work because they didn't understand the reasoning of funding old platform.

So the point here would be first to get enough attention so that the story doesn't repeat itself. Insurgo is interested to fund a big part of the work, but we concluded that we would not do this alone.

Some history:

• FOSDEM talk by 3mdeb on AMD coreboot status
• Reddit thread on announced drop of the KGPE-d16

Putting a bounty tag for this.

[tlaurion]

Something really interesting was proposed on this QubesOS ticket: https://github.com/QubesOS/qubes-issues/issues/2809#issuecomment-626111758

@pietrushnic, I would love to hear on your reception. This would basically mean that the independent tasks could be independently funded upon PR acceptation. Not sure how to orchestrate this.

We are currently experimenting with OpenCollective, while the fees are really high, which makes the move less interesting (a lot of money being lost in translation).

Other options might be more interesting, like Whonix did.

So the question here being mainly:

• How to raise attention toward the need of this board across technical communities who would need this RYF platform internally. Examples: self-hosters (Ex: younohost clients)
• How to manage donation and reduce its processing fees.
• How to organise those tickets (metatickets with daughters tickets for tasks? Broader?), and where so that money is raised for work to be done directly.

See this ticket to see how chicken/egg problems can last for a year even when money is available without a clear plan.

This is the inverse problem here, just like funded work normally work. Tasks here are defined, costs attached for work to be done is quantified with a fair price.

[pietrushnic]

@tlaurion XVilka suggestion looks interesting, but definitely key problem is orchestration. We would like to avoid the situation when 3 or more teams work on a given feature in parallel and then post results claiming the founds. There should be serious commitment based on proven reputation and trust, otherwise, we have public bidding problem where lower price wins with quality.

Whonix stuff is even better, but we would have to research how it looks from the taxation and administration perspective.

[tlaurion]

@pietrushnic I'm not sure there is conflict between the two ideas with XVilka avenue. If the ticket is clear saying that 3mdeb realizes the work in question (milestones) and funds are released upon proof of work (PR).

But you are better placed then I am to see if this would work or not for 3mdeb.

[mikebdp2]

@tlaurion and @pietrushnic : Have you contacted a Free Software Foundation? Since they have a lot of interest in RYF devices and are probably using these KGPE-D16 servers by themselves, if they'd learn about this critical situation - maybe they'll decide to co-fund?

P.S. also, some notes at #717 may be relevant.

[pietrushnic]

@mikebdp2 we (as 3mdeb) never contacted FSF. I'm not sure if the situation is critical but it will get worst over time. From experience longer platform lays unmaintained bigger would be effort to bring it back. Do you know anyone who we should reach?

@tlaurion I believe you are in a better position to talk with FSF, then 3mdeb.

[pkubaj]

22000E isn't that much different from what Raptor wanted for OpenBMC port (at https://www.raptorengineering.com/coreboot/kgpe-d16-bmc-port-offer.php). Maybe you should set up similar crowdfounding? Currently you only say "we are ready to work" etc. Raptor did the same and received no money, but when they officially started crowdfounding, it turned out there were enough people interested in it to gather the money.

[pietrushnic]

@pkubaj very good point. It looks like Martin was intermediary gating the process. In case of difference in pricing, please note this was a different time and they are in the US not in PL, their costs are way higher than ours. More to that we are Yocto Participants and have Embedded Linux people on board, OpenBMC uses Yocto/OE build system, so this also simplifies the situation for 3mdeb.

[tlaurion]

@mikebdp2 we (as 3mdeb) never contacted FSF. I'm not sure if the situation is critical but it will get worst over time. From experience longer platform lays unmaintained bigger would be effort to bring it back. Do you know anyone who we should reach?

@tlaurion I believe you are in a better position to talk with FSF, then 3mdeb.

@pietrushnic writing to them.

[alexmaloteaux]

@Tonux599 : For the moment, step would be to get people interested in the port. Let's remember that the KGPE-D16:

* was funded by [Libreboot and put upstream by RaptorEngineering](https://libreboot.org/docs/hardware/kgpe-d16.html) but previous work did not receive enough attention to be maintained and got dropped in coreboot 4.11 release for lack of maintainership.

* the community crowdfunded the port of the[ BMC chip to OpenBMC ](https://www.raptorengineering.com/coreboot/kgpe-d16-bmc-port-offer.php)and that work wasn't maintained.

* NlNet refused to fund the work because they didn't understand the reasoning of funding old platform.

So the point here would be first to get enough attention so that the story doesn't repeat itself. Insurgo is interested to fund a big part of the work, but we concluded that we would not do this alone.

Some history:

* [FOSDEM talk by 3mdeb on AMD coreboot status](https://fosdem.org/2020/schedule/event/coreboot_amd/)

* [Reddit thread on announced drop of the KGPE-d16](https://www.reddit.com/r/linux/comments/dz2hlt/boards_like_asus_kgped16_the_most_powerful/)

Putting a bounty tag for this.

Hi is there a list of the bugs/linux crash that Michal talk about in his fosdem talk ? Do they only happen when iommu is enabled ?

Btw i have a kgpe-d16 with jtag header soldered and an olimex. So if you are starting any group for dev , put me in, free time is limited but i can always help a bit and test.

[tlaurion]

@alexmaloteaux asked:

Hi is there a list of the bugs/linux crash that Michal talk about in his fosdem talk ? Do they only happen when iommu is enabled ? @miczyg1 ?

[miczyg1]

Simply have a look at the coreboot mailing list:

• https://mail.coreboot.org/hyperkitty/list/[email protected]/thread/ME4TULTDZGIYJEMNR2IB2W6P7ONG76EU/#GCJF7CIWGCKZCDWYPWAM2LKQW6APHZRJ (memory problems)
• https://mail.coreboot.org/hyperkitty/list/[email protected]/thread/ZR7LMBXOFKR5FWVQDI22UB6O3LO5WGQ3/#TH57VEGCY357ZH534PU5PKSDSMYLTLFH (S3 suspend/resume problems)
• https://mail.coreboot.org/hyperkitty/list/[email protected]/thread/VWFKH7FDWLO5KGZMIJ7VM67SIUWJRVZL/#OCHSI6UL2XZDSA3HCHOONNN5EX4HLUGE (kernel panic from new microcode)
• https://mail.coreboot.org/hyperkitty/list/[email protected]/thread/H5KHOUVDPQA2NSUE43IBRKSYBZQK4DWN/#H5KHOUVDPQA2NSUE43IBRKSYBZQK4DWN (SLUB memory allocator in Linux panics)
• https://mail.coreboot.org/hyperkitty/list/[email protected]/thread/RDHEDOTBYVD74NWQWYYNLAJWPD7CMDRZ/#RDHEDOTBYVD74NWQWYYNLAJWPD7CMDRZ (KVM/SMV not working)
• https://mail.coreboot.org/hyperkitty/list/[email protected]/thread/AW4DF557YDTTJGKDFRTHMZJBTF65ZJZT/#VOFCIRV24HT3OIG5IDYH4GEXY5VSS3W5 (kernel panics on kernel newer than 4.9)

[pietrushnic]

In case of what have to be done I believe Arthur email is quite good. @tlaurion you replied to it here

[alexmaloteaux]

I have read the various mail list entry and i cant reproduce any of them. Actually i have a pretty stable system so far so i will list here my config and test , maybe others can use it as baseline to try to triggers issues and/or star the long 4.12 migration process. please note i have a pure server usage for this board and not a desktop pc.

• Coreboot version : 4.10, commit ea58adddf4 with seabios and sgabios, all input/output redirected to uart 0x2f8 io port for OpenBmc (coreboot, sgabios, os)
• Heads Version : master but cant use atm as im facing TPM issue
• CPU : 2X Opteron 6378 2.4GHz 16M
• Ram : 2x Micron 16GB Ram DDR3 MT36KSF2G72PZ-1G6E1FE PC3L-12800R on slot A2/E2
• TPM : Asus TPM-L r2.0 TPM 20-1 pin (recognized by seabios but tpm-reset script or tpm-tool have issue ; unknown tag ...)
• Screen : Not using VGA at all, all input/output redirect to OpenBmc uart on io port 0x2f8
• Linux Boot : OK (ubuntu 18.04 / kernel 4.15)
• Freebsd Boot : OK (12.1 stable)
• Linux Boot with IOMMU enabled : OK (iommu=pt amd_iommu=on)
• Freebsd Boot with IOMMU : OK
• Linux install and boot VM in KVM : OK
• Freebsd install and boot VM in bhyve : not yet tested
• Linux PCI-passthrough to VM in KVM : OK (tested with a IBM M1015 flashed in it mode)
• Freebsd PCI-passthrough to VM in Bhyve : not yet tested

Ill try to post a more detail description of the tpm issue i m facing later on

BR

[Tonux599]

@alexmaloteaux regarding TPM, TPM2 is currently unsupported with heads, see #287 however the Asus 90-C1B0AU-00XBN0VZ is a TPM1.2 module compatible with heads and this board.

[pkubaj]

I have two of those boards, one is a desktop, the other one is a server. Both run coreboot 4.11 tag.

Both run FreeBSD, the desktop is sometimes booted to Linux.

Both are stable. I didn't test averything that @alexmaloteaux tested. I'm pretty sure PCI passthrough won't work with bhyve due to CPU's missing some required features.

[alexmaloteaux]

I have two of those boards, one is a desktop, the other one is a server. Both run coreboot 4.11 tag.

Both run FreeBSD, the desktop is sometimes booted to Linux.

Both are stable. I didn't test averything that @alexmaloteaux tested. I'm pretty sure PCI passthrough won't work with bhyve due to CPU's missing some required features.

63xx Series supports AMD-Vi and can support PCI passthrough , except os/coreboot bug

[Tonux599]

@mikebdp2 we (as 3mdeb) never contacted FSF. I'm not sure if the situation is critical but it will get worst over time. From experience longer platform lays unmaintained bigger would be effort to bring it back. Do you know anyone who we should reach? @tlaurion I believe you are in a better position to talk with FSF, then 3mdeb.

@pietrushnic writing to them.

@tlaurion any luck?

[tlaurion]

Waiting for an e-mail reply from FSF

[tlaurion]

I would suggest anyone interested in the d16 to be reupstreamed writing to FSF too?