My-Book-Live
My-Book-Live copied to clipboard
ewaldc
Merge your work with the OpenWRT project
Hi Ewald, thanks for your interesting work. I got it running recently and was really excited by the performance.
However, the outdated Debian Jessie made plenty of problems. I had expired repo keys, expired Root CAs and even missing mirrors. Took me a while to get it at least working, but as time goes, it won't get better.
So I tried the OpenWRT image, thats busybox with a 5.10 kernel. Really easy setup, nice web UI, useful package repo, but the performance is .... meh. I am getting ~30MB/sec over SMB, and not more than ~550Mbit on iperf.
So my dumb question: can we have your lightning fast kernel and the OpenWRT ecosystem? I tried to replace the uImage with your 4.19.99 image (and modules), but it did not boot. And since I never managed to get the netcat thingy to work I am blind, and have no idea, what's happening, sadly.
Hi Marc, I have played with the idea, even had it up and running about 2 years ago. While it was very stable, it was nowhere near the performance on Debian, at least with my tests. Possible reasons could have been the NFS/Samba implementation, or compilation options for reduced size versus performance. After all, it's an OS for routers/embedded devices. Some packages were missing at the time such as DLNA etc. I would have had to create a whole series of "special" packages for NFS/Samba/minDLNA. Then here was the issue that all your data would have to be wiped, compared to the "dd" alternative of just overwriting the OS partition with a pre-build Debian 8.11 and re-initilizing the SWAP. There was also the issue of OpenWRT supported kernel versions where the faster kernels in my tests (4.9 and 4.19) were not supported by the corresponding APM82181 OpenWRT releases. Kernels 4.4 and to a lesser extend 4.14 had performance issues for PowerPC that would have required for me to provide a large amount of patches. In fact since 3.18 Linux kernel performance has gone down, which is why many embedded OS's like Android have only recently moved up from 3.18 to 4.9 and 4.14.
Two years later, looking at OpenWRT 21 and with embedded devices becoming more performant, it's maybe time to re-evaluate. The good news is that I have already ported my optimized kernel to 5.4, so there is a match between OS and suppored OpenWRT 21 kernel release. The not so good news is that 5.4 is 10 to 15% slower than 4.9. This is not just for PowerPC, but the same overall 10% loss has been noticed, in very extensive testing, for Intel/AMD CPU's (link). By re-writing the TCP/IP stack I was able to reduce the loss to around 5%, but it would have been a maintenance nightmare.
I would stay away of 5.10 for now, as indications are another 5 to 10% performance loss.
Do you know if packages like MiniDLNA/Samba/rsync/NFS server are available for the APM82181?
Hello Ewald, thanks for your response. While I consider myself an "above-the-average" linux guy, most of your work is a magic black box for me (I am doing LoB Software on c# for a living, reasonably far away from embedded stuff). However, here's what I found on my MBL running OpenWRT currently (APM82181 is powerpc_464fp on their wiki):
[x] Samba: https://openwrt.org/packages/pkgdata/samba4-server (that's what I'm running currently) [x] MiniDLNA: https://openwrt.org/packages/pkgdata/minidlna [x] rsync: https://openwrt.org/packages/pkgdata/rsync [x] nfs: https://openwrt.org/packages/pkgdata/nfs-kernel-server
Or another idea: have you considered prebuilding (can we cross-compile gentoo?) a Gentoo image? I had it running, but just emerging samba took days, so I canceled it. Or even https://archlinuxpower.org/ while I have no idea how active resp. stable it is
Edit: have you seen this? https://github.com/chunkeey/mbl-debian
This project's build.sh generates an adapted Debian Sid/Unstable (As it still has the powerpc target today) image for the Western Digital MyBook Live Series devices.
Hello Marc, this is very useful information.
- Excellent news for OpenWRT.
- I've had Gentoo running successfully, but it had a memory leak back then which caused a crash every 2 to 3 weeks. It was fixed about 6 months later if I recall well, but by then I had already abandoned Gentoo and removed the whole build environment.
- Must admit that I had not seen Christian's work on Debian PowerPC. Christian is absolute world-class, no doubt that this would be a fantastic route. Running modern Debian would give endless possibilities.
So we have two great routes. Let me start with the easiest one: OpenWRT. I will need some time to setup the OpenWRT compile environment and MUSL libraries, that is how OpenWRT build its OS, kernel, packages and overall image(s). It's been a while since I have done that...
The one thing I will not port is the Asynchronous DMA driver. It was never published anyway. With it, I was able to outperform the original WBL significantly with sustained 120MB/s sequential writes over (modified) Samba 4. The DMA driver copies memory blocks from user space to kernel and back, but it touches many parts of the kernel, including several system calls. The biggest issue though is that the smallest change or error destroys the HW (at least, I was never able to bring two boards that crashed in the DMA driver I was porting to 5.4), and that is why I have not published the DMA source code for kernels 4.9.x and 4.19.x.
Sucessfully compiled Debian image with kernel 5.17.12. Had to fork the repo because of a few small errors and to integrate my kernel into the build process over time. Now need to find an MBL to test...
UPDATE: I have mbl-debian working... Somehow it does not boot with the 5.17.13, but it does boot with 5.17-rc8. Strange, need to figure out why that is....
Hi @ewaldc and everyone else still making the MBL possible to run with up to date kernels. Thank you so much for your work.
I'm sorry to jump in, but considering you've mentioned Gentoo I'd like to join some of the discussions. I have successfully booted Gentoo using your precompiled kernel 4.9.99-mbl+.
I have been trying to cross compile a kernel on Gentoo but so far without success. My last attempt has been 4.9.169, with your netconsole 4.9.149 config. Netcat doesn't show anything until it falls back to the safe kernel. I've applied the patches and still had to edit another file due to yylloc being defined multiple times. I've used vanilla sources and had to finish the compilation of uImage on mbl, since I'm having issues installing u-boot-tools under qemu (fails with an illegal instruction).
Unfortunately I do not have access to the serial console yet and maybe I'm not even using netcat correctly, since on a normal boot I only see:
random: fast init done
cgroup: cgroup2: unknown option "nsdelegate"
I was expecting to see the whole contents of dmesg and even echoing into the kernel log to be shown on netcat.
Any recommendations? Would I have better chances of compiling the kernel natively on MBL (either on Gentoo or debian), even if it were to take a long time? Would cross-compiling on Debian (or other distro) get better chances of working?
Are you compiling using your cross-compilation instructions?
Any guidance or tips from past experiments is appreciated and I hope I'll reach a point where I can be more helpful with kernel compilation and OS support.
@ewaldc Congrats on getting your MBL working again!
Hello @Kvitz,
Kernel compilation on the MBL gets more dificult with every new gnu compiler/linker. Also more recent kernels (4.x, 5.x) have trouble with OOM (out-of-memory) conditions and will kill off the largest memory consumer such as the linker.
That is why I almost exclusively use cross-compilation on Ubuntu, Debian, Mint etc. With an 8 core CPU and NVME drive, it takes just 5 minutes! In the past I used the OpenWRT build system (on Linux Mint), but recently I am just using the native gnu gcc cross compilers. Take a look at this repository. There is a "build-kernel.sh" script that builds a Debian kernel package for the MBL by checking out the kernel from Github, applying changes and patches. Right now it produces a Debian kernel installation package, but if you change the last line in the script to "make ARCH="$ARCH" CROSS_COMPILE=powerpc-linux-gnu- -j
BTW, it is key to get netconsole working. This is an example of "double netconsole" enabled by "boot.scr":
setenv nc 'setenv bootdelay 10; setenv stderr nc; setenv stdout nc; setenv stdin nc; printenv'
setenv ncip '192.168.1.25'
setenv ncmac 'c8:d9:d2:13:6b:47'
setenv ipaddr '192.168.1.6'
saveenv
run nc
setenv ncIPLan '192.168.1.25'
setenv ncMacLan 'c8:d9:d2:13:6b:47'
setenv bootargs 'root=/dev/sda1 earlycon earlyprintk rw rootfstype=ext4 rootflags=data=ordered ipv6.disable=1'
setenv bootargs_lan 'setenv bootargs netconsole=6663@${ipaddr}/,6664@${ncIPLan}/${ncMacLan} ${bootargs}'
setenv load_sata 'sata init; ext2load sata 1:1 ${kernel_addr_r} /boot/uImage; ext2load sata 1:1 ${fdt_addr_r} /boot/apollo3g.dtb'
setenv load_sata_rcvr 'sata init; ext2load sata 1:1 ${kernel_addr_r} /boot/uImage.safe; ext2load sata 1:1 ${fdt_addr_r} /boot/apollo3g.safe.dtb'
setenv load_tftp 'tftp ${kernel_addr_r} ${bootfile}; tftp ${fdt_addr_r} ${fdt_file}'
setenv boot_kernel 'run bootargs_lan addtty; bootm ${kernel_addr_r} - ${fdt_addr_r}'
run load_sata_rcvr
run boot_kernel
On your Linux host, open two windows, one with "nc -u -l -p 6664" (kernel) and one with "nc -u -l -p 6666" (mbl-uboot). You will even be possible to type u-boot commands to the MBL.
Right now it have Debian SID (bullseye+) running without problems with unoptimized kernel 5.17.
Let me know how it goes.
Thank you for pointing me to https://github.com/ewaldc/mbl-debian. The kernels I compiled didn't boot with Gentoo and netconsole only shows something when falling back to the safe kernel (one of your precompiled ones). I changed the config to support 64K pages and netconsole, copied the dtb and uImage but still didn't boot Gentoo.
However, I decided to give the whole build process a go and I now have kernel 5.17.0+ (from the 5.17 branch, since I couldn't find 5.17.12 as per your script) and Debian bookworm :)
Now that I have a build environment that works it gives me a bit more confidence to play around. I'll study your scripts and maybe I'll figure out why Gentoo doesn't work. I've been trying Gentoo because they still support powerpc, unlike Debian, but I will be testing Debian for now.
I like how the LED works, staying off while the disk is inactive and blinking blue while accessing the disk.
SSH failed to start, but cockpit worked fine and allowed me to start SSH.
Sorry for hijacking your thread @marcwittke. I'll report any progress/problems on a new issue.
The SSH issue is due to Dropbear and Openssh both being installed. The goal was to have Dropbear, but I think Openssh is a dependency of one of the packages. Will need to fix that. In may case, SSH did function though, but I had to wait a long time before it opened up... Something to fix. I have not managed so far to get my 4.x kernels running, so eventually I might have to port all my driver changes to 5.x e.g. 5.17. Not sureif it is even possible to a 4.x kernel on Debian Sid...
Sorry about the confusion with kernel 5.17.12. It is really the 5.17 branch that is checked out and it resulted in kernel 5.17.12 being build in my case (and today it will be 5.17.14). It is not a good thing that you can not specify the exact version since my patches can be subversion dependent, I will change that so one can checkout the exact kernel subversion. Will let you know when I published the changes...
UPDATE:
- git source updated to allow version specific checkouts
- support of version specific patches and config files
- kernel checkout performance improved by ~10x
- README updated
- double netconsole working OK on 5.17.x
- fw_printenv/fw_setenv fixed (team effort with Christian Lamparter)
- larger boot partition (implemented by Christian)
- initial work completed to integrate ("eliminate") initramfs. NOTE: MBL Duo with raid will require full initramfs with the more recent kernels.
I have managed to compile kernel 5.15.41-gentoo (meaning from Gentoo sources, not vanilla), using a Gentoo crossdev environment. With it I can boot Debian but also Gentoo, which I've prepared on the same Gentoo crossdev environment. I used your patch as well as your config, but with netconsole enabled. I also used the .dtb generated by the Debian environment but I guess I should be able replicated those steps on Gentoo. I still need to understand how to deal with these .dts/.dtb files and that expansion step on your scripts and anything else that might be kernel version dependent I guess.
I picked this kernel version because it's the latest marked as stable in Gentoo. Double window netconsole is now working fine and I even managed to interact with it when I made a mistake on the boot.scr.
However, I'm still using the initramfs generated your Debian environment (the v5.17.0+), so now I just need to understand whether I need an initramfs in the first place and if so, how can I generate one using my own "independent" Gentoo environment, rather than Debian tools.
Once I manage this I will play with both Debian, Gentoo, maybe test different kernel configurations inspired by the kernel 4.x configs and test some software packages.
Once again, a big thank you!
Very nice! With respect to removing initramfs, I am not there either. Previously I managed to do this for kernel 5.4, but with the newer kernel versions and Debian SID it looks a bit more complicated, or maybe a simply forgot how to do it...
On 5.17, there is no way to remove initramfs from the kernel. But that is no issue, you can have an "emtpy" ramfs. But there at least 4 other changes required which I am working on:
- All drivers that are required at boot must be compiled in. That is not the case today, just do a "lsmod" right after boot. This part I managed to do. I removed "mdadm" as quick fix to remove a whole seriers of RAID related kernel drivers. I will post the modified kernel config file.
- Remove any UUID from critical files (/etc/fstab, kernel boot command line). That part is complete.
- Modifiy "boot.scr" to remove the ramdisk and eliminate UUID's. I have not managed to do that. I don't know about 5.15, but in 5.17 there are some changes that impact legacy u-boot mode.
- Create an "empty" initramfs/boot with initrd. Haven't started that part.
The biggest advantage to eliminating initramfs is the faster boot time, simpler network booting and the possibility to just "drop in" a new kernel, which during development is a big bonus. That said I managed to improve the build-kernel script so that consecutive debian kernel package builds of the same kernel version succeed and can installed over the previous version.
I looked at my lsmod for my Gentoo system and there were only a few modules. I've adjusted the kernel config to include them, changed the UUID for /dev/sdX and managed to boot without initramfs. Gentoo lsmod now shows no modules.
Using this kernel I also managed to boot the Debian partition that I had created using your build scripts. The lsmod on Debian now shows:
root@mbl-debian:~# lsmod
Module Size Used by
sg 23481 0
fuse 91583 1
nfsd 268447 13
auth_rpcgss 44710 1 nfsd
ip_tables 13584 0
x_tables 19960 1 ip_tables
I haven't checked how many were there before.
Anyway, this seems like good news, because I've been able to create an initialramfs using dracut on Gentoo on MBL, but when I tried using dracut on my Gentoo cross-compilation environment it didn't work, even though I pointed it at the cross-compiled modules with which I've been booting Gentoo on the MBL.
Maybe if I need an initialramfs on a more recent kernel having the ability to boot this kernel without initramfs for the purpose of using dracut to generate one for a more recent kernel might be useful.
I'm only just starting exploring ways of generating the initialramfs, so let me know if any of this makes sense. This is still on 5.15-41-gentoo, I haven't tried any recent ones.
Congratulations on your progress with Gentoo. The reason why you have al these modules on mbl-debian is because the kernel compiles these drivers as loadable modules. I managed to compiled them all in, but the resulting kernel does not boot: it complains about a corrupt partition and panics. Sadly, I have not found the root cause yet and worse, I also happens on Christian Lamparter's unmodified latest version for kernel 5.19. Started to doubt my HW, but it boot an old disk fine. It may be some change in the latest kernel versions or some partition alignment problem on my WD Green drive. So for the moment, dont use the current mbl-debian repo...
The mystery has been solved! It is one of the challenges of using an unstable version ! As I was testing, and restesting, at every test a new, "unstable" Debian image was being created from the daily (unstable) builds, It so happened, as I finally discovered, that during this time the images contained a critical error that caused (certain) systems not to boot. Of course, as I was making lots of changes to the kernel and the Deban image, I was thinking that maybe I did something wrong and started to reverse changes, ultimately going back to the original unmodified code, only to find that nothing helped. Because I managed to damage the HW/FW before, I expected the worse, until I saw other users reported similar Debian problems and a pacth was released. Now I have set up my own Debian image server so I can always roll back to the "last known good" version. It is now clear to me that we can't rely on daily unstable images, It's doable though to maybe pick one or two releases per year, test those thoroughly and then just install critical security patches. Anyhow thanks for your generous offering!