Linux Kernel Cannot Print where Bootrom and BBL can

[Kendidi]

Hi,

I tried to load and run Bootrom, BBL and Linux Kernel on FPGA.

Bootrom and BBL can print on UART console without any issue, but the Linux Kernel cannot, even it went pass through a few places in setup.c and init.c where it should have printed something on the console. I think the ariane.dts in bootrom should be OK, otherwise BBL may not be able to print.

I wonder: 1) What may be the issue? Where should I look at? 2) How does the UART/console get initialized normally so that functions like pr_info( ) and printk( ) can print?

Thanks a lot in advance!

[jctullos]

@Kendidi

Did you ever find a fix? I'm at the same issue right now. BBL and Bootloader print just fine, and when it hands over to Linux, nothing gets printed. I figured it has to do with the UART, but still trying to work out how to fix it.

[Kendidi]

@jctullos

Not yet. I am still trying to figure it out. I initially thought early_init_dt_scan( ) takes care of UART initialization. But it appears in a Qemu environment, Linux Kernel can already print to console before early_init_dt_scan( ) is launched. Therefore I think the UART config was either done before MMU is enabled, or before Linux Kernel is launched.

[jctullos]

So if you pass earlycon through bootargs in defconfig, it will setup an early console with the uart. Are you running the Open Piton build or just regular CVA6/Ariane? What FPGA?

I'm running the Open Piton build, that uses ns16550, on a VCU118. I think the uart device isn't getting picked up correctly in the device tree. Right now I'm rebuilding the FPGA bitstream. My initial hunch is that the uart in the device tree passes "ns16550" as the compatible name, and maybe "ns16550a" is what the Linux driver is looking for, even though it should work fine with "ns16550". I'll let you know how it goes.

@zarubaf or @jbalkind Would either of you have seen this issue before?

[Jbalkind]

@jctullos please be sure you're building from the openpiton branch. Also, have you set earlycon=sbi?

[jctullos]

I didn't on this build, but I'll add it. I didn't see it on the openpiton branch for the linux defconfig, that might need to be added if it's required. I'll try it right now.

[Jbalkind]

At what point in the boot process are you not seeing output? With the existing defconfig built from the openpiton branch it should just print. Does the prebuilt bbl.bin work? https://www.princeton.edu/~cloud/openpiton/os_images/openpiton_ariane_linux_r12.tar.gz

[jctullos]

The prebuilt doesn't work either, I just tried again with the link you posted. Here's the output:

[Jbalkind]

Could you share the start of the output too? How did you prepare the SD?

[jctullos]

SD was prepared the way it shows on the openpiton repo. Here's the start:

[jctullos]

I am also working on another bbl build, it will print the device tree just fine, but hangs when Linux normally takes over. This is in minit.c and after which nothing outputs.

At the jump to payload. I had a debug statement there previously which was the last thing it printed before jumping to the payload.

[Jbalkind]

The output looks reasonable. Partition table is correct, bbl is clearly located in the right place, and system frequency is reported correctly which means the ariane.dts should also be correct. Since the partition table is correct, the SD reading clearly isn't totally broken... Timing was probably met too I'm going to guess?

Looks like you're building from openpiton-dev... Have you done a git submodule update --init --recursive piton/design/chip/tile/ariane by any chance?

I don't really trust bbl to be honest. It has a kernel memory corruption bug which has caused us some strife. You could try opensbi instead? I shared some instructions here previously: https://groups.google.com/g/openpiton/c/8J9IX1VLRHA/m/poid3PVbCgAJ

[jctullos]

I haven't done an update in a few days, I'll check it out.

Also, earlycon=sbi didn't change anything. And I'd like to move to opensbi, but my research is using Keystone and they still use bbl unfortunately.

I've seen this issue in the past, I just can't remember how I fixed it. I had issues with 64 bit memory addressing in the dts file getting parsed correctly. That would have been my next attempt, to change address and size cells to 1 instead of 2. But the openpiton build has high addressing, so I'm assuming that's out unless it can be easily changed.

[Jbalkind]

There shouldn't be any new ariane updates in terms of our submodule pointer, it was more in case you were going from the openpiton branch to openpiton-dev and hadn't got the accompanying changes in doing so.

That's a shame. I'll warn you that linux boot can be flaky with bbl. If you see random stalls and stack traces, it's likely to be bbl's fault. When we switched to opensbi, all of that went away. This was a problem both for standalone ariane and op+ariane. Others had reported the same for rocket I believe, too.

You can move the ariane peripherals to a different location by changing the base addresses in piton/design/xilinx/vcu118/devices_ariane.xml and see if that helps? But we were using bbl for quite a while and I don't remember seeing this issue. That does make me suspect that perhaps not all of the SD woes have passed. May be worth trying our sdctrl_test design on vcu118 and seeing if it passes?

[Kendidi]

@jctullos

I use Xilinx FPGA, bootrom.elf built off from the CVA6 fpga tree and bbl built from ariane-sdk.

[Kendidi]

@Jbalkind and @jctullos

I initially thought bootrom.elf (from cva6/fpga/) + bbl (from ariane-sdk) should be enough for booting Linux on the Xilinx FPFA we built. Is bbl not good enough? What approach would you recommend? Thanks.

[Kendidi]

@jctullos

Please let me know if you have any success or have found anything with your bbl build. Thanks.

[Kendidi]

@jctullos

So if you pass earlycon through bootargs in defconfig, it will setup an early console with the uart.

Can you please elaborate?

What I have in "\ariane-sdk\buildroot\output\build\linux-ariane-v0.7\arch\riscv\configs\defconfig":

• CONFIG_FIX_EARLYCON_MEM=y
• CONFIG_SERIAL_EARLYCON=y
• CONFIG_SERIAL_EARLYCON_RISCV_SBI=y

So which routine is supposed to setup the console?

[Moschn]

Also, earlycon=sbi didn't change anything. And I'd like to move to opensbi, but my research is using Keystone and they still use bbl unfortunately.

@jctullos I have also noticed no uart output with keystone on stock Ariane (without openPiton). If I remember correctly I solved this with using a newer linux kernel version. I do not know where this comes from though. I would advise you to try a few different kernel versions (I believe I used 5.3).

[Kendidi]

@Moschn

Where in Ariane SDK we can select which Linux Kernel version to use?

[Kendidi]

According to https://github.com/pulp-platform/linux/, Linux 5.1-rc7 appears to be the latest.

[Moschn]

@Kendidi It is not supported to select this. You need to build it manually. I guess you could take a look at our build process and try to adapt it to an upstream kernel.

[jctullos]

thanks @Moschn although I'm using 5.3 right now for Keystone, so there must be something more.

[jctullos]

@Jbalkind

I think you might be right regarding the SD issues. After some random resets, Linux will boot. It's not consistent. So I assume data isn't getting copied over all the time. It hasn't completed a boot to the login prompt yet, right now it's failing on the RPC module, but at least there's output.

[Kendidi]

@jctullos

So what changes did you make so it can print now?

[Kendidi]

@Moschn

Ok. Thanks. After I downloaded Kernel 5.3, how should I modify before compiling it?

[jctullos]

@kendidi I thought I replied yesterday, sorry!

So I'm using a Linux 5.3 build. And passing console=hvc and earlycon=sbi to bootargs. FYI, Linux boots maybe 1/3rd of the time. And it always lags by a few seconds before it starts printing.

[Kendidi]

Thank you @jctullos!! I will try to find a suitable copy of Linux 5.3 and try it.

[Kendidi]

Besides "console=hvc" and "earlycon=sbi", what else do we need to put into bootargs?

[jctullos]

@Jbalkind

So I believe I've fixed the issue with Linux not showing the console. I remembered that I also had issues with the memory still holding data that wasn't getting cleared. I wrote a small while loop in the bootloader that just clears the memory where BBL is going, and then the sd_copy starts. So far, this has fixed it. There's no more issues between BBL and Linux handover.

One question though, I still have issues with the Linux boot stopping halfway. Looking at the vmlinux system map, it seems to be when clock source switches over. Earlier, I had changed the timebased-frequency value in the DTS to half of the clock frequency, this was only because the Ariane/CVA6 repo had a commit that fixed their DTS to half the system clock.

Does openpiton use another RTC build for the timebased-frequency? I know in the DTS generation script, it's the Sysclock/128. I'm changing it back to the default openpiton setting to see if this fixes Linux booting. But wanted to verify the frequency.

[Kendidi]

@jctullos

I am using Kernel v5.3. I have added the following to ariane.dts but it's still not printing once Kernel has taken over

bootargs = "console=hvc earlycon=sbi";

Is there anything else I have missed?