heads
heads copied to clipboard
linuxboot
WiP change Intel PHYsical mac (gbe) from Heads
- Addition of nvmutil (nvm) from osboot project to play with gbe (Thanks @githubisnonfree!)
- Addition of ifdtool from coreboot project to extract gbe
- As of now, its implemented in a hacky way:
- ifdtool dir is copied over ifdtool_cross at coreboot's module configure step
- then initrd packing step checks for CONFIG_NVMUTIL and builds and pack ifdtool_cross/ifdtool
- As a result, what is build under build/coreboot/$BOARD is coreboot's real, where build/coreboot/ content follows Makefile rules
- CONFIG_NVMUTIL in board config adds both ifdtool_cross/ifdtool and nvmutil into initrd
- Added CONFIG_NVMUTIL to all hotp-maximized boards (to test for
size changes)
Manually tested (working!):
- backup rom from: `flashrom -p internal -r /tmp/backup.rom`
- go to that dir: `cd /tmp`
- extract gbe from ifdtool on backup.rom: `ifdtool -x backup.rom`
- show current PHY mac address: `nvm flashregion_3_gbe.bin dump`
- change mac from nvm on extracted gbe (random): `nvm flashregion_3_gbe.bin setmac`
- show mac to be changed: `nvm flashregion_3_gbe.bin dump`
- insert modified gbe into backup.rom.new with ifdtool: `ifdtool -i gbe:flashregion_3_gbe.bin backup.rom`
- flash back modified gbe only through flashrom: `flashrom -p internal --ifd -i gbe -w backup.rom.new`
Question now is what is the proper workflow from here. New menu options from configuration menu?
- Generate new MAC address -> show mac address : flash ROM with new mac address (Yes: flash, No: Generate new MAC)
- Specify new MAC Addresss -> go to console prompt for user to set new mac, validate with nvm output on extracted gbe.bin insertion -> flash if valid?
Fixes #916
the flash rom gui menu could use a tweak already IMO. i would like to see all options merged into submenus. It would be nice have it go like this:
Flash New Rom > GPG Settings > MAC Settings > Mount USB and Select rom/GPG > Confirmation
GPG Settings:
- Keep Current GPG
- Add New GPG from USB
- Clean Flash (generate GPG later)
MAC Settings:
- Keep Current MAC
- Generate New MAC
- Specify New MAC
**Mount USB and Select rom/GPG**
Confirmation Dialog:
- Old Heads vs New Heads version
- Old GPG vs New GPG
- Old MAC vs New MAC
**Flash/Cancel**
first option on both prompts retains settings, making routine updates as simple as possible.
maybe worth noting that the "add to standalone bios and re-flash" options in the existent gpg options menu could be removed as redundant if the Flash menu gets retooled this way.
@tlaurion I like this approach. I would be concerns that if a user flashed a maximized version over the standard HEADS rom then modified the MAC address before rebooting then the depending on if flashrom read the IFD from the buffer or from the chip may write to the wrong location. This would prevent that.
An input mask allowing only hex-ish characters (0-9A-F and a-f, :, ) where any missing characters are padded with zeros then sent through sed/tr to be only `0-9A-F``, would prevent the need for a complicated MAC check.
i try to be as correct as possible with what status is returned on exit from nvmutil. you can pretty much just run it and do || in your shell script to make it crap out if there's a problem (e.g. invalid checksum, invalid mac address) etc
so like, your build system would let the user specify a mac address and your build system could just...
./nvm gbe.bin setmac macaddresshere || exit 1
or whatever you want after that ||
that's the way i do it in osboot, the simple way. fail early, fail hard is the way i generally do things
by the way, this should work on basically all intel ifd based machines, all those gbe regions have the same layout in them, and nvmutil intentionally dosen't set anything unless the checksum is correct, to guard against a future possibility that intel changes it; presumably they'd use a different checksum validation then, or maybe put it in a different place
but it should work just fine on all the thinkpads you support. works on x200/t400 etc aswell. i also use it on t440p machines
i actually checked linux src and pretty much all these nics use that same algorithm for checksum: the words must add up to 0xBABA, and mac address is first 3 words
(linux verifies the checksum before doing anything with the nic, and will actually refuse to set up your nic if both checksums are bad)
@tlaurion I like this approach. I would be concerns that if a user flashed a maximized version over the standard HEADS rom then modified the MAC address before rebooting then the depending on if flashrom read the IFD from the buffer or from the chip may write to the wrong location. This would prevent that.
@Thrilleratplay : Yeah thought about that. We could wrap around checking the output of flashrom -p internal output to see if no warning are given there.
An input mask allowing only hex-ish characters (
0-9A-Fanda-f,:,) where any missing characters are padded with zeros then sent through sed/tr to be only `0-9A-F``, would prevent the need for a complicated MAC check.
@Thrilleratplay : We could, or we could base ourselves on the validation made by nvmutil (nv). I prefer the latest.
Long term (PR welcome) we could also borrow other MAC randomization tweaks. Some router will not permit some non existing OUI. So it would be logical to randomize only in the "Intel Corp" OUI space but there I do not want to go beyond user's needs?
i try to be as correct as possible with what status is returned on exit from nvmutil. you can pretty much just run it and do || in your shell script to make it crap out if there's a problem (e.g. invalid checksum, invalid mac address) etc
so like, your build system would let the user specify a mac address and your build system could just...
./nvm gbe.bin setmac macaddresshere || exit 1
or whatever you want after that ||
that's the way i do it in osboot, the simple way. fail early, fail hard is the way i generally do things
@githubisnonfree : The goal here is to build against musl-cross-make in the firmware and provide tools for users to change PHY whenever they want and flash back internally from flashrom. We could permit the user to build a ROM with a custom PHY, but that would go against reproducible goal and mode support. Unfortunately in Heads current use case, since we build ifdtool and nvutil from musl-cross-make to be packed in rom (against musl libc) we could document the process of building from the builder machine, but I kinda not see the point if the firmware is doing the proper thing, that being to pack a gbe.bin that is setuped as "DE:AD:C0:FF:EE" in the goal of having everyone having maximized builds to have reproducible roms in the long term, even if those ROMs contain a GBE, ME, IFD and BIOS regions. And have the user decide if he wants to change it. Let's not forget that user's are better having a randomized MAC setuped through NetworkManager nowadays, this feature is mainly aimed as a safeguard, and for Heads users having multiple machines to be able to connect multiple machines on LAN if they do not apply in software mac randomization.
The problem I foresee in the future is that Coreboot is starting to measure other components. 4.17 is now measuring CBFS content, and ME region is desired to be measured since forever. I do not see why GBE (and everything Intel IFD) not being measured at some point. That will modify PCR2 as of now, and in the future we will need to do some tpmfuturecalc to take that into consideration for future upgrades.... Should probably be considered in other future issues, just a note.
@weyounsix
the flash rom gui menu could use a tweak already IMO. i would like to see all options merged into submenus. It would be nice have it go like this:
Flash New Rom > GPG Settings > MAC Settings > Mount USB and Select rom/GPG > Confirmation
GPG Settings: - Keep Current GPG - Add New GPG from USB - Clean Flash (generate GPG later) MAC Settings: - Keep Current MAC - Generate New MAC - Specify New MAC **Mount USB and Select rom/GPG** Confirmation Dialog: - Old Heads vs New Heads version - Old GPG vs New GPG - Old MAC vs New MAC **Flash/Cancel**first option on both prompts retains settings, making routine updates as simple as possible.
maybe worth noting that the "add to standalone bios and re-flash" options in the existent gpg options menu could be removed as redundant if the Flash menu gets retooled this way.
Unfortunately, I am not sure this would please all stake holders here, me not being convinced with the approach either, since those actions are normally desired separately, and one rarely needs to change public key (and if he desires so, can do it under GPG options).
One that wants to change all of this would most definitely simply flash a clean rom without keeping settings and hit the OEM Factory reset/Re-Ownership wizard to accomplish all of this, minus the PHY MAC changing that is subjected here.
@Thrilleratplay @githubisnonfree @weyounsix What about simply
- adding a new submenu only if CONFIG_NVMUTIL exists? (BOARDS should not add NVMUTIL if not supported to Change PHY MAC Address (CONFIG_NVMUTIL would need to be exported then to be under /etc/config)
- Add logic under flash.sh to also take into account CONFIG_NVMUTIL presence, check if mac is different then DE:AD:C0:FF:EE through ifdtool -x backup.com, nvm showmac flashregion_3_gbe.bin and if different then rom, reinject user defined PHY MAC from actual rom? (Backup?)
That would not require any other logic to be modified, would have the MAC persist between firmware upgrades and let users decide if they want to change their mac address? Changing menu and orders of menu options is a complicated matter (check past PRs), where Purism and other boards constructors might not even have a Compatible Intel Ethernet card (those connectors tend to disappear, and if Ethernet port is there, they are not necessarily Intel made).
I prefer small approach and move from there.
As far as this PR goes, it is actually feature complete (but xx20 and xx30 board configurations should add the feature disabled). A problem here is that I am pretty sure xx20 board owners will as of today not be able to even add a public key (with a picture, or circle of trust signatures included in their public key) might not even fit since this PR is including ifdtool, as all other boards (since ifdtool addition is added for ALL, including platforms that do not have Intel Firmware Descriptor (kgpe-d16 boards, Talos II etc are now including a tool that they will never use.
So basically, next steps prior of being even able to merge this is:
- Makefile find a way to externalize ifdtool inclusion under nvmtool module? Some ideas based on the current hack on global Makefile, anyone?
Some notes on actual problems that won't permit t420/x220/w520 to be usable.
Master build log on x220-maximized:
Name Offset Type Size Comp
cbfs master header 0x0 cbfs header 32 none
fallback/romstage 0x80 stage 84972 none
cpu_microcode_blob.bin 0x14d00 microcode 26624 none
fallback/ramstage 0x1b580 stage 97628 none
config 0x33340 raw 776 none
revision 0x33680 raw 691 none
fallback/dsdt.aml 0x33980 raw 14615 none
vbt.bin 0x37300 raw 1400 LZMA (3985 decompressed)
cmos_layout.bin 0x378c0 cmos_layout 1848 none
fallback/postcar 0x38040 stage 25816 none
fallback/payload 0x3e580 simple elf 7209415 none
(empty) 0x71e780 null 71192 none
bootblock 0x72fdc0 bootblock 65536 none
This PR x220-maximized build log:
FMAP REGION: COREBOOT
Name Offset Type Size Comp
cbfs master header 0x0 cbfs header 32 none
fallback/romstage 0x80 stage 84972 none
cpu_microcode_blob.bin 0x14d00 microcode 26624 none
fallback/ramstage 0x1b580 stage 97639 none
config 0x33340 raw 776 none
revision 0x33680 raw 691 none
fallback/dsdt.aml 0x33980 raw 14615 none
vbt.bin 0x37300 raw 1400 LZMA (3985 decompressed)
cmos_layout.bin 0x378c0 cmos_layout 1848 none
fallback/postcar 0x38040 stage 25816 none
fallback/payload 0x3e580 simple elf 7225799 none
(empty) 0x722780 null 54808 none
bootblock 0x72fdc0 bootblock 65536 none
So the addition of ifdtool alone consumes from the free space available: 71192 - 54808 = 16384 bytes 54808 bytes is still ok when we add stuff into kernel or initrd (xz compressed) but never forget that when we add stuff with cbfs, those are uncompressed. Let's also remember this is not x220-hotp-maximized which requires additional hotp binaries and libs:
x220-hotp-maximized build log from this PR
Name Offset Type Size Comp
cbfs master header 0x0 cbfs header 32 none
fallback/romstage 0x80 stage 84972 none
cpu_microcode_blob.bin 0x14d00 microcode 26624 none
fallback/ramstage 0x1b580 stage 97639 none
config 0x33340 raw 786 none
revision 0x336c0 raw 691 none
fallback/dsdt.aml 0x339c0 raw 14615 none
vbt.bin 0x37340 raw 1400 LZMA (3985 decompressed)
cmos_layout.bin 0x37900 cmos_layout 1848 none
fallback/postcar 0x38080 stage 25816 none
fallback/payload 0x3e5c0 simple elf 7233479 none
(empty) 0x7245c0 null 47064 none
bootblock 0x72fdc0 bootblock 65536 none
Only has 47064 bytes free after ifdtool inclusion from global Makefile change.
So merging this PR would be the first case where xx30 and xx20 would differ in features that can be offered. #590 and some of its subtasks will probably need to be fixed prior of a merge of the current PR.
More direct todos:
-
[x] Seperate ifdtool building under Makefile or make it conditional on CONFIG_NVMUTIL
-
[ ] Work, test https://github.com/osresearch/heads/pull/1121 and merge
-
[ ] Work on https://github.com/osresearch/heads/pull/1184, test and merge
why not just have a script
download it into tmpfs and run it
(after checking signatures)
also, i've recently started porting openbsd userland to linux+musl, i'm months away from completing it probably but i want to replace busybox. when you take all the extra non-posix stuff away until you're left with only the same features as busybox, openbsd code is actually smaller in almost every case i've checked - even ksh is the same size as busybox sh, and that's with all the bells and whistles
i hope to have something to show later this year, that people can really use on their machines. i'm doing it for my own linux i'm working on, but you can be sure i'll be showing it to heads first: i'm doing it because i have the same problem as you: limited flash space. i want a flash distro for osboot, like heads though i'm basing my work off of alpine locally. i'll release later in the year when i have something complete
have a look at chimera linux, they have a few nice things in it
in particular, they have almost (if not entirely, haven't looked in great detail yet) a complete freebsd-based userland on top of linux and musl
they recently ported freebsd sh to linux
freebsd sh is about 12k source lines of code, busybox sh is like 19k last time i checked. you might find more gold like that in chimera linux to try out in your project
here's posix cat for free:
https://vimuser.org/cat.c.txt
(this is my one, after stripping down the openbsd one)
edit: and to be clear i'm recommending this because i'm aware your distro has the problem: 8MB flash setups have limited feature sets due to low flash space. i'm told alpine linux is also working on their own userland but i have no idea about the status of that
also, i've been experimenting with various compilers and i've found that in a lot of cases, tcc produces smaller binaries than gcc or clang
in cases where gcc wins (if one of them beats tcc, it's always gcc) like with -Os and LTO, that's uncompressed binaries, those tcc-built binaries when compressed (e.g. in CBFS with LZMA!) really get quite small indeed
freebsd sh (alone) can be made to go down to about 150KB potentially and that's when i built for x86_64, for i686 you can probably get much smaller, whereas busybox ash probably can't get much smaller than say 230KB. these are uncompressed sizes btw
how small? well for a program that is about 300KB compiled with GCC or clang, it's not uncommon for me to see that tcc is about 150KB, half the size, and that's with stuff like -Os and -flto used in GCC
you should look into using tcc if you haven't already. it could really save a lot of space in rom when putting all these utils together. tcc is great (also called tinycc)
@githubisnonfree all those comments unfortunately will be lost under a PR once merged. I think you of all other users (libreboot and osboot main collaborator) understand this.
All those comments are related to #590 and should be added there.
Thanks for your understanding and motivation. Getting away of dropbear would be interesting, while we accomplished a state where dropbear being stripped, as all other binaries and libraries also compressed with xz -9 and -Os missing (700kb freed) I'm not sure moving from musl to tcc is recommended here, where musl have a really good reputation on both security stability and size.
Yet again, #599 would be the place to debate those ideas not here, which will be lost in space and time soon enough tracking those ideas.
oh, well musl is a libc and tcc is a c compiler
i don't use github so i don't understand its nuances. i'll post on the issues instead
@githubisnonfree sorry for the confusion, heads uses local GCC to build musl-cross-make and then builds everything with it, including coreboot's buildstack which then builds itself with it.
Not against those ideas at all, just telling you a PR (merge request equivalent) is not the place to track sidequest ideas.
Issues are. As on notabug ;)
i mean notabug sucks. normally i just work with people directly on irc or they email me
Unfortunately, I am not sure this would please all stake holders here, me not being convinced with the approach either, since those actions are normally desired separately, and one rarely needs to change public key (and if he desires so, can do it under GPG options).
for context: my thinking here was merely that it would be nice to only have to flash the bios once for a totally new configuration. i find it strange that gpg operations are separate, considering any such operation involves reflashing. i wouldn't advocate removing the gpg operations menu, but just to merge the relevant options for "standalone bios images" into the flash menu, so that if a new rom is being flashed (not the "running bios" gpg operations) then you are able to fully configure it in one go.
understood and no sweat if this is not desirable. just a thought.
@githubisnonfree sorry for the confusion, heads uses local GCC to build musl-cross-make and then builds everything with it, including coreboot's buildstack which then builds itself with it.
Not against those ideas at all, just telling you a PR (merge request equivalent) is not the place to track sidequest ideas.
Issues are. As on notabug ;)
by the way, i've made a note about musl-cross-make. it sounds like exactly the sort of thing i need in my project
@tlaurion While having a difference between the xx30 and xx20 isn't ideal, I think using nvmtool is the best approach given the restrains of size. Also keep in mind that is it isn't a feature that is completely denied to the xx20 users as they could still use do this using a prebuilt version in external storage or through an OS. They are only losing the convenience of it being built in. Hopefully, this will be short term.
regarding randomization
i think the best way to have random mac addresses would be to implement it in nvmutil. command syntax example:
./nvm gbe.bin setmac 00:11:22:33:44:55
This example above is setting a static address. What if I made it possible to do:
./nvm gbe.bin setmac ??:??:??:??:??:??
or maybe
./nvm gbe.bin setmac 00:?1:??:2?:33:4?:5?
See that?
Basically, any character that is ? would have nvmutil to randomly generate a number.
This is the best way, because you might want a certain pattern. For example, some Intel NICs (on x200/t400 thinkpads) start with 00:1f:16:xx:xx:xx where x characters can be anything.
Some router will not permit some non existing OUI. So it would be logical to randomize only in the "Intel Corp" OUI space but there I do not want to go beyond user's needs?
The list of Intel Corp possible OUI is large.
This is the best way, because you might want a certain pattern. For example, some Intel NICs (on x200/t400 thinkpads) start with 00:1f:16:xx:xx:xx where x characters can be anything.
@githubisnonfree absolutely, this would fit the need, where collisions on a shared prefix should not happen on LAN.
Under Heads, we could lspci and apply Intel OUI accordingly, and/or define randomization patterns under Heads board config if users want to limit range as well.
From my limited tests, my own locally used switches never stopped me from using totally randomized Mac addresses in the past, as opposed to WiFi controllers. From my limited experience, and outside of NAC enforced network limitations (which reuses wireshark OUI list as pointed in previous post) I do not think wired MAC are filtered, but I might be wrong for certain corner cases (my Cisco switch for example permitted total local randomization as used under /etc/functions as of now under network-recovery-init under Heads, so took it as a positive PoC and good enough)
Would be a nice to have, definitely, though!
well C has rand() which is pretty decent. the only question is, what to use as a seed value.
a most conservative choice would be to simply fetch the current time (unix timestamp).
of course, another way is to directly read from /dev/urandom
i will think about this
there are libs that do randomization but for our purposes, what i want is something simple and reasonably random. this isn't AES
i think a good sane approach is:
read from /dev/urandom, get numbers from that. urandom is sufficiently random for our purposes
or just return -2 if can't read from urandom
currently, mac address validation returns -1 if invalid mac passed. i'm designing my patch to set -2 if randomization is impossible for whatever reason, then if -2, validation says: randomization failed. then it exists with non-zero status
basically an FU to non-unix systems
@tlaurion i did it
patch: https://notabug.org/osboot/nvmutil/commit/40f25c70d5fb7aeb6e56abf91e562a901c743e42
This adds randomization support for setmac, with the scheme previously described.
@tlaurion check again nvmutil git repo. i optimized it a ton. 11224kb compiled with tcc on my x86_64 machine
i'm planning to revive nvmutils too, with these optimized techniques from nvmutil. the benefit there is i could make for you much smaller binaries that only do single tasks
anyway, compressed with lzma, nvmutil when compiled with gcc will currently get you about 5kb, or with tcc, about 3kb. tested: with xz -9e
i'm going to experiment further, later; i will see if using mmap for files (instead of read/write) might yield a smaller binary. that's about all i can do before i start breaking features. this is the most riced gbe mac address changer ever
@githubisnonfree as stated under https://github.com/osresearch/heads/issues/590#issuecomment-1047293706, heads challenge is to optimize for space bigger tools and libraries from their modules definition. That means removing unused features of such tools, optimizing compilation for space, removing unused functions post linking and/or compressing more or other tricks.
I see your attempt to reduce from 5k to 3k as input to be able to reduce the biggest tools and libraries we currently need, eg lvm cryptsetup and other gpg toolstack required tools.
But the interest here is not to pass from 5k to 3k. As issues/PR pointing to/from https://github.com/osresearch/heads/issues/590, by reducing kernel size (compressed by xz) and removing in kernel compiled stuff, big gains can be accomplished there with a bigger ratio gain in total space wins.
Moving to tcc is interesting, but would require migrating all modules where libc is one of the biggest thing we compile for other modules to be linked against it.
ok, but nvmutil has been optimized for size now, and i just woke up as i write this. gonna finish nvmutil using mmap to see if that reduces size further
nvmutil could be modified to only have the ability to set mac addresses, like nvmmac from nvmutils. to reduce size of the binary further (even when using gcc)
@tlaurion i made a new release of nvmutil. version 20220808
https://notabug.org/osboot/nvmutil/src/20220808
Changes:
https://notabug.org/osboot/nvmutil/src/20220808/ChangeLog.md
Summary:
- Vastly reduced code/binary size
- OpenBSD pledge used (ifdef rule used, so it still compiles on linux/freebsd)
- Better/safer file handling
- Makefile:
-Osflag used instead of-O2 - Stricter exit status, upon program termination
- EDIT: and of course, random MAC addresses are now possible
I highly recommend upgrading to this version. Also, I'm contemplating making a standalone binary that just changes the mac address, for even smaller binaries.
Regarding ifdtool: want me to fork it and make it really small? Tell me whether you have any IFDv2 systems in heads. If it's just v1, that simplifies things a lot.