motorola-potter mainline kernel, 2nd attempt

[telent]

Convert motorola-potter to use the same msm8953-mainline kernel as is used in PostmarketOS. Requires lk2nd to be installed (or at least, running) on the device beforehand, maybe I should add a derivation for that in an overlay?

Worth noting that this may also give us a way forward for the very tight 16MB boot partition on potter, because lk2nd can find its boot.img by mounting each of "system", "system_a", "system_b", "cache", "userdata" as ext2 and looking for the file /boot.img there. I haven't succeeded in doing this yet, but haven't tried that hard either because I'm still using fastboot boot

$ ls -lh result/
total 2.0G
-r--r--r-- 1 root root  18M Jan  1  1970 boot.img
-r-xr-xr-x 1 root root  368 Jan  1  1970 flash-critical.sh
-r--r--r-- 1 root root  18M Jan  1  1970 recovery.img
-r--r--r-- 1 root root 2.1G Jan  1  1970 system.img

[telent]

works

• boots
• display
• adb (a bit kludgey in stage-2)
• volume up/down/power keys for selecting menu options
• touchscreen
• wlan

untested

• usb networking
• sensors
• telephony
• video accel etc

[telent]

Update

I've cleaned up the git history significantly, which may or may not help with reviewing but might in any case be useful for future developers looking to find when bugs were introduced. In terms of what woks and what doesn't it's basically on a par with the vendor kernel, with the addition that it has DRM support, or at least enough of it to run wlroots-based Wayland compositors

Questions and outstanding stuff

• the kernel and initrd are built into a boot image by concatenating them, but there's no builtin kernel Makefile support for doing this directly as us used on some other vendor-tree devices - CONFIG_BUILD_ARM64_APPENDED_DTB_IMAGE, for example, doesn't exist here. When I added support (in b95a8e80869961b1b737273764db3f956603f163) to do this ourselves, I put it in modules/system-types/android/bootimg.nix ratter than the kernel builder, because it seemed to me that the kernel and the dtb are philosophically separate things that come together in the image - even if in reality they're both built by the kernel builder and that's unlikely to change. Or maybe as the project grows there's a scenario where you'd want to build the same kernel for 4 or 5 msm8953 devices, and each one has a different DTB. Or maybe neither of these are good arguments and I should just do it in the kernel builder :shrug:

• I seem to have two different ways to say the kernel is modular (isModular and kernel.modular) and probably I should find out which one is the right one.

• it requires lk2nd - should I add a derivation for that? I don't imagine it's well suited for nixpkgs inclusion

• docs need updating

• boot.img is still bigger than 16MB. My preference here is to take advantage of the lk2nd support for putting boot.img on a filesystem, rather than spend hours or days eking out kB here and there. I can think of two ways

  • advise users to install their system.img onto userdata and mkfs.ext2 their system partition, then copy boot.img onto it using adb in recovery, or something like that
  • have the image builders include /boot.img as a filesystem entry inside the filesystem for system.img

I don't know how either of these will play out with generation selection

[samueldr]

• it requires lk2nd - should I add a derivation for that? I don't imagine it's well suited for nixpkgs inclusion

We should package it, just as U-Boot is packaged for Nixpkgs. Though at first this can be handled within Mobile NixOS in a device-specific overlay package.

For Nixpkgs inclusion, it can probably end-up packaged there too, just like U-Boot will always be available as mostly stock mainline U-Boot. That can happen once we're confident with the lk2nd packaging and it's easy to support multiple devices.

[samueldr]

• boot.img is still bigger than 16MB. My preference here is to take advantage of the lk2nd support for putting boot.img on a filesystem, rather than spend hours or days eking out kB here and there. I can think of two ways

  • advise users to install their system.img onto userdata and mkfs.ext2 their system partition, then copy boot.img onto it using adb in recovery, or something like that
  • have the image builders include /boot.img as a filesystem entry inside the filesystem for system.img

I don't know how either of these will play out with generation selection

I don't think generation selection will be affected in any way by either option, at least not the "kexec-less" generation selection.

It might be interesting to see if we can add more to lk2nd to support the extlinux-compatible file format and add generation selection at that level. It would solve a lot of headaches for those older devices, including with vendor kernels.

It might also be helpful to see if we can use system as /boot and have it load the boot image from there, should be a simple patch I think? "just" add an additional boot source in that partition?

[telent]

My Moto G5 Plus now boots Mobile Nixos mainline when I turn it on, without any manual steps needed :-)

Made a stab at some installation docs, entirely happy if you want to edit them for "house style". I haven't figured out how to build the website docs locally to be able to preview them.

There are a lot of manual steps in the installation - it would be good to smooth that over somehow (maybe a subsequent PR). Thoughts:

• lk2nd installation is a one-off, so probably shouldn't be combined with anything else
• flash-critical.sh is currently broken: given that it can't flash system and probably shouldn't flash lk2nd, it doesn't seem to offer much value over telling users to fastboot flash recovery recovery.img
• we could create a boot.e2fs.img which wraps an ext2 filesystsem around the boot.img so it can be flashed to system without needing to boot into recovery and mess with adb etc. It does feel a bit matryoshka doll

Another possible future enhancement would be to teach nixos-rebuild how to upgrade the kernel - it could generate a boot.img, mount system temporarily and copy it there. I don't think it would be impossible either to teach lk2nd to look at separate files for kernel/dtb/initramfs, but that would depend a lot on whether upstream would adopt the patch

[samueldr]

Made a stab at some installation docs, entirely happy if you want to edit them for "house style". I haven't figured out how to build the website docs locally to be able to preview them.

nix-build doc/

And open the HTML files, all refs should be relative or else it's a bug. If they're not you could use any ol' "http serve directory" thing e.g. ruby -run -ehttpd result/ -p8000 to serve it from the root.

[samueldr]

• flash-critical.sh is currently broken: given that it can't flash system and probably shouldn't flash lk2nd, it doesn't seem to offer much value over telling users to fastboot flash recovery recovery.img

Yeah, the initial intent was to follow the conventions set by Google for their pixel device flash scripts (and followed at some point by other ROMs). It is flawed in many ways, among those: buggy device-side fastboot implementations. So it's not only a problem for lk2nd.

we could create a boot.e2fs.img which wraps an ext2 filesystsem around the boot.img so it can be flashed to system without needing to boot into recovery and mess with adb etc. It does feel a bit matryoshka doll

I'm thinking it might be worthwhile to take inspiration from the target-disk-mode system and make something work with wither adb or exporting only the safe partitions with UMS to manage the system. Though that requires USB gadget to work on mainline, or to keep vendor going in parallel for that thing.

• https://github.com/NixOS/mobile-nixos/tree/master/examples/target-disk-mode

[samueldr]

Another possible future enhancement would be to teach nixos-rebuild how to upgrade the kernel - it could generate a boot.img, mount system temporarily and copy it there. I don't think it would be impossible either to teach lk2nd to look at separate files for kernel/dtb/initramfs, but that would depend a lot on whether upstream would adopt the patch

Yes, this is desirable. The main issue is that it's an unanswered question, and requires proper care to consider doing correctly.

An even more future enhancement would be to look into implementing extlinux.conf-compatible parsing to lk2nd to completely bypass that issue and "just" load the kernels like U-Boot does (when not UEFI). (I'm not asking you to do it, just proposing solutions)

[telent]

I've not looked at this PR recently, but it would be good to wrap up the loose ends to make it mergeable before I completely forget how it works. What are the loose ends for making it mergeable?

[samueldr]

One of the things I wanted to do was run the thing, but the global shipping program from eBay lost the potter I bought and there were none affordable when I last looked :/.

I wanted to try it in depth because I wanted to try and unify/harmonize ASAP the mainline approaches with the SDM845 PR.

So I'll take a look again today and see what is needed here. Probably not much? I'll see if anything is going in a different direction than with the SDM845 mainline PR, and if it does, try and gauge which way is better.

---

With a cursory look, one such difference (haven't evaluated either options) is the way I handled appended dtbs

• https://github.com/NixOS/mobile-nixos/pull/511/commits/c328a3105aeabbbce315295ef2290ac9a734b7c6
• https://github.com/samueldr-wip/mobile-nixos-wip/blob/30c583033569c2845c7a709f0e1ffc9a96637ff2/devices/families/sdm845-mainline/default.nix#L14-L23

~~The main difference with my approach is how it creates a discrete "FDT-free" kernel build, and it's being appended after the fact. This means there is a single costly rebuild for multiple devices. An implementation detail that could be solved with yours too is how it allows multiple dtb files to be appended (which has issues on Oneplus devices sigh).~~ I should have reviewed more closely, it does the same thing but applies the complexity to the Android builder.

[samueldr]

So, I've taken a look.

I have slightly re-organized the commits, and split some. (As I've told in the matrix room.)

• https://github.com/samueldr-wip/mobile-nixos-wip/commits/PR506

So everything under motorola-potter I have to leave to the maintainer to decide if it's correct. At face value, it looks fine.

Within the non-motorola-potter changes, four broad strokes:

• kernel-builder fix
• lk2nd
• appended DTB
• USB gadget

The kernel builder fix is simply correct. I'm almost tempted to shove it in its own PR and fast-track it.

The lk2nd changes seem correct enough.

The appended DTB change I still need to compare against the other implementation.

The USB gadget changes I've changed "a lot". I think the USB gadget changes should go into its own PR, since it's totally independent from these changes, and still requires more testing and maybe doing things correctly. Anyway what I've changed is moving the stuff out of the motorola-potter device and into a new module, and I've been testing it locally.

---

TLDR: seems almost all good, let's rewrite the history a bit (I've handled it, but not pushed on top of your changes) and drop the USB gadget changes.

Then we're left with determining the appended DTB approach to prefer.

---

EDIT: updated my branch and kept only the relevant changes, see the fresh PRs

[samueldr]

Noting here that ~~everything~~ most changes that's not related to motorola-potter (lk2nd, and motorola-potter stuff) has now been merged already. (Preserving authorship.)

We're left with the appended DTB strategy to handle, and then the device-specific changes.

I still want to take some time with you to verify you're okay with the rewritten history, and if you can, verify it still works just as well.

[samueldr]

Just correctly evaluated the appended DTB strategy used here. I preferred the way it was intended to work (append at the latest moment convenient), but disliked the API... But really, what I dislike is how tied to the kernel builder the Android builder is, so it's not your fault, you followed along with how things were implemented.

This has lead to me smushing together the implementations here:

• https://github.com/NixOS/mobile-nixos/commit/64518cd43e86ee3abf3eeda25a8f0fa4b6a597fb
• https://github.com/NixOS/mobile-nixos/blob/4ba441506cf7cdab688d6dde31f7fb6e2f621440/devices/families/sdm845-mainline/default.nix#L40-L42

As can be seen in the example usage here, the interface is not as verbose as my previous attempt, but still makes the device itself expose the DTB it needs. This, in turn, allows the kernel derivation to stay truly generic.

This particular usage (ab)uses the fact the DTB filenames are coherent with device codenames for SDM845, so it provides a correct default. With your mainline kernel efforts here, it might not apply as well because the SoC name may not be exposed "properly" (dtbs/qcom/sdm625-motorola-potter.dtb). But this is only an implementation detail for when it's shared in a family.

[telent]

At long last ... have built and installed https://github.com/samueldr-wip/mobile-nixos-wip/commits/PR506 on my handset and can confirm that it successfully boots into the "hello" system. Happy with this to be merged (if you are, obviously)

(Side note: have not had any feedback on the lk2nd PRs - I don't know if you're any better placed than I am to excite some interest in them from upstream)