OpenWireless
OpenWireless copied to clipboard
Run firmware under qemu
We should be able to take a pre-release firmware image and start it up under qemu for testing without requiring a physical router.
Qemu supports the MIPS CPU in the WNDR3800. The OpenWRT build targets a few different MIPS platforms - i.e., whole systems that include the MIPS CPU and other components. The WNDR3800 uses the ar71xx platform. However, my understanding from various web pages is that OpenWRT is only expected to run on qemu if it is compiled for the Malta platform, which is well-supported under Qemu.
I've been able to build OpenWRT (/ CeroWRT) targetting the Malta platform, and start the resulting openwrt-malta-be-vmlinux-initramfs.elf image. The config for this is checked in under OWrt/config-OWrt-qemu. Note that the MIPS CPU can be either little endian or big endian (-le- vs -be- in the output file names). Since the ar71xx platform is big endian, we configure our Malta build to be big endian too.
I think the next step is to to use the bare kernel from the Malta build (openwrt-malta-be-vmlinux.elf) and provide it with the filesystem extracted from a production ar71xx build. I think this is likely to look something like:
qemu-system-mips -kernel ./openwrt-malta-be-vmlinux-initramfs.elf -nographic -m 64
-append "mtdparts=spi0.0:320k(u-boot)ro,128k(u-boot-env)ro,15872k(firmware),64k(art)ro rootfstype=squashfs,jffs2"
-mtdblock mtd.bin
However, the Malta platform has a hardcoded partitioning scheme for its (emulated) mtdblock device, so the mtdparts command line option is ignored. I'm not sure how best to resolve this.
Additionally, if we build for the Malta platform, the network devices available don't match the network devices our firmware for the WNDR3800 expects.
I did try building a firmware image, including initramfs, for the ar71xx platform and running that under qemu, but that build froze during startup.
One thing I forgot: The latest OpenWRT compiles using gcc 4.8, but the resulting image won't run under qemu. You need to build using gcc 4.6, which is a setting under make menuconfig. The config-OWrt-qemu I mentioned above includes this setting.
References: http://wiki.openwrt.org/doc/howto/qemu https://dev.openwrt.org/ticket/16881
This would be very cool to complete. Hope someone takes this up and makes it to work soon.
I'd like to take a look at this. First question, has anyone taken a look at running this image on real hardware?
Awesome, thanks @monocasa! I have not tried running the Malta build on real hardware. I'm not actually sure which real platforms run it, but it sounds like finding one and trying it out would be a good start.
@jsha one thing about gcc 4.8 vs 4.6 issue. The malta platform is broken in trunk since gcc was switched to 4.8. Compile with gcc 4.6 to get it working or disable MIPS16. To disable MIPS16 just need to comment CONFIG_USE_MIPS16 = y in config.
With gcc 4.6 gets "Kernel panic - not syncing: Attempted to kill init! exitcode=0x00000004"(https://dev.openwrt.org/ticket/16881). Going to check gcc 4.8 with disable MIPS16.
QEMU's MALTA emulation only provides minimal MTD support, which is intended just to be used for starting up the board, and can't be used as a block device for accessing the firmware. Alternatively, it provides an IDE port, which can be used with block2mtd for using an JFFS2 image as root.
That said, I think the goal of this issue should be clarified. If the final purpose is running an unmodified WNDR3800 firmware image, a non-trivial amount of work should be invested on modifying QEMU to simulate a variety of devices (specially on the network front).
But, on the other hand, I suppose OpenWireless will start targeting other devices in a foreseeable future, which would mean supporting more CeroWRT targets. If that's the case, I think adding support for an specialized QEMU/x86, using it as an alternative reference platform, would be a better option. Obviously, this wouldn't be of use for verifying hardware support, but it could be used for testing general functionality, perhaps even featuring an automatized network test bench.
Good point, @sip. The goal is indeed to run an unmodified WNDR3800 firmware image, or at least a the filesystem from a WNDR3800 firmware + a modified kernel. It sounds like you're saying even the latter will probably not work because the network devices (among others) will be too different.
When we started the project we were doing testing on an OpenWRT x86 VM (in VirtualBox), but it drifted too quickly out of sync with the real platform we were targeting. One option would be to modify our firmware build process to always generate an x86 image in addition to the MIPS image. @Rangak can comment more on how practical that is. I think we'd want two separate build directories so we wouldn't have to clear out built artifacts when switching between architectures.
One other problem with using x86 emulation rather than MIPS emulation: Some of the packages we depend on are readily available on x86, but challenging to get for MIPS. That why, for instance, we opted for a pure-Python PBKDF2 library rather than a compiled binary one.
You're right about the problem with binary packages, a MIPS-based platform would be better as the userland should be (almost) the same. But, implementing all physical devices from ar71xx on QEMU is a huge task by itself, and I'm not sure if it's worth the effort.
Perhaps, a pragmatic approach would be using the MIPS Malta platform as an alternative supported target. I think a sensible roadmap could be something like this:
- Changes on CeroWRT (pushed to upstream, if possible)
- Improve kernel configuration for building Malta with IDE, BLOCK2MTD, PCNET and USB support.
- Modify pre-init scripts for mounting a overlay FS with two block2mtd devices, one squashfs and the other JFFS2. This mimics the conventional layout of CeroWRT.
- Extend Malta target support to provide the same features as other targets.
- Changes on QEMU:
- Verify that is possible running a machine with various (at least two, LAN and WAN) emulated PCNET devices.
- Add support for USB passthrough. This should make possible to use some USB WiFi dongles as HostAP devices (well, at least in theory).
- (OPTIONAL) Implement a switch device supported by swconfig.
Do you think I'm missing something?
Hey, I'm looking to see if this is a project that a final year college student could do. (I have some students who might be interested)
It seems like there's a few ideas being thrown around. Is the Malta, ar71xx or x86 target the one we want to focus on? We'd be more interested in the Malta or ar71xx targets because they're larger and fit more with some of the things we're trying to do at prpl.
Right now Malta is not a target supported by open wireless. The only reason Malta is in this discussion is that it is the closest (i.e MIPS) platform supported in QEMU. We don't think x86 is useful to address as the differences etc make it problematic. Ideally if you are able to find the resources we would get ar71xx running under QEMU so we can run our actual openwireless image. If that is too much work let us talk about supporting Malta.
@Rangak I'd love if we could get ar71xx support working. Do we have a sense of what is missing in QEMU currently? Is this a configuration issue or are there features missing in QEMU?
Just thinking out loud, I'd love we could add more support to QEMU so people could run basic smoke tests on builds for their routers before they upload them and then have to go through the process to unbrick them. It'd make it easier for everyone involved in the OpenWrt ecosystem to have more confidence in the support for routers.
@jsha in your original post you used -mtd mtd.bin. How and where did you create "mtd.bin?"
@ericschultz I believe I did something like dd if=/dev/mtdblock
on the device. However, I am far from sure that this was the correct approach. I'm very inexperienced with embedded device programming and Flash memory block devices in general. I did later find a tool binwalk
that can be used to extract data from a firmware image, including the partitions that get written to Flash. That is probably a better approach in the ideal solution.
I agree with @Rangak that it would be awesome if we could get ar71xx platform support working in QEMU, since our current target is the WNDR3800, which is ar71xx-based. Do keep in mind that we are looking for a new target platform (along with CeroWRT) because the WNDR3800 is discontinued and becoming difficult to buy. However, even if OpenWireless eventually moves on past ar71xx, it is probably a good student project, and would help our project in the medium-term. It should be a good chunk of work, and there are evidently a bunch of FOSS projects targetting ar71xx so it should be reasonably well documented.
I also like @slp's pragmatic approach. One good bite-sized chunk to start with would be his first item under QEMU support:
Verify that is possible running a machine with various (at least two, LAN and WAN) emulated PCNET devices.
This is important no matter what path we take. We'd like to make sure QEMU can support two ethernet devices and two wifi devices.
@ericschultz To answer your question a little more clearly: I am not sure what needs to be added to QEMU for it to support ar71xx as a platform. Per @slp's comments, it seems like some of the devices, especially network devices, may need to be implemented or ported.
And yes, I expect this will be great for smoke tests. Two other reasons I'd really like to have QEMU support: automated testing and quick start. I think we could have a broader development community if people did not need to possess a physical router to hack on the software. Already we've split out the web frontend a little bit and had good contributions there from people who don't own a router. But a lot of the backend settings change (like bandwidth limits) are impossible to properly test without a full environment.
@ericschultz @jsha Basically, you have support for the CPU, but lack everything specific to this board. You'll need to take a look at Linux sources to see which devices it expects to find, how the early boot process is implemented, if it needs some kind of non-standard memory layout, etc... Then you'll need to extend QEMU to add support for all of the stuff you found.
This is easily 2-3 month full time worth of work, and that's assuming you're already familiar with Linux and QEMU code base.
Hey, sorry for the delay. I've busy at work so I haven't checked in, but I've been adding wndr3800/ar71xx support directly to qemu. It's turns out that it's easily one of the simplest SoCs I've seen. : ) Do any of guys actually have root access to a wndr3800 so I can give you guys some scripts to run to answer some outstanding questions? Or, better yet, mind giving me ssh and root access to one of your machines?
@monocasa That's terrific, thanks! I've got a WNDR3800 and I'm happy to run some scripts. If that becomes too tedious I can probably arrange SSH access too, but I'd want to figure out how to isolate the router a bit better from the rest of my home net. :-)
@monocasa Actually, I can just mail you my WNDR3800 for a few weeks to hack on if you'd like. Send me email at [email protected] if you're interested.
@monocasa I can also run the scripts on my wndr3800 if that helps?
Note for anyone finding this via google (in 2021 or later!)
Booting off a malta-compiled kernel in qemu and then swapping to a 'native compiled' root filesystem roughly works.
Fixing the odd malta partition names: They're not defined in qemu, they're defined in the openwrt malta build configurations. Look for "malta_platform.c" in openwrt, there is:
static struct mtd_partition malta_mtd_partitions[] = {
{
.name = "YAMON",
.offset = 0x0,
.size = 0x100000,
.mask_flags = MTD_WRITEABLE
}, {
.name = "User FS",
.offset = 0x100000,
.size = 0x2e0000
}, {
.name = "Board Config",
.offset = 0x3e0000,
.size = 0x020000,
.mask_flags = MTD_WRITEABLE
}
};```
which defines a 4MB device. Adjust size and partition names to taste.
Networking: include "pcnet32" kernel module, and on quemu use "-device pcnet.."
@raplin thanks, after your comment I resolve my problem :) There is my example. Here I can run rootfs inside of QEMU and reproduce bootchain: u-boot -> kernel -> init system.