linux-xtensa
linux-xtensa copied to clipboard
jcmvbkbc
ESP32-S3?
I guess the same way you run it on ESP32. You'd need to figure the address where the kernel goes and FLASH/PSRAM caches setup. Take this branch for a template, add a core variant for ESP32-S3 at this step, add different DTS and change kernel config specifying correct core and addresses at this step and use a compiler for the ESP32-S3 to build the kernel.
Where would I find the headers to add the ESP32-S3's core variant. After that, what would I need to change in the DTS. Then I need to know what the correct core and address are so I can change them.
Where would I find the headers to add the ESP32-S3's core variant
I'd start here, but I see that these overlays are incomplete. Maybe ask for the linux parts? Or maybe the overlay was published elsewhere, I just couldn't find it by quick googling.
what would I need to change in the DTS
Take the DST for the ESP32 and look at the devices and memories that you care about. Fix the register addresses/sizes that are wrong.
what the correct core and address are
The correct core is esp32s3, or whatever you name it when you import it. The address -- you need to think about it: it must be in the FLASH, be mappable through the FLASH cache and you'd need to make a partition in the FLASH for it. It's not the single correct answer, but rather a range of more or less convenient options.
I'd start here, but I see that these overlays are incomplete. Maybe ask for the linux parts? Or maybe the overlay was published elsewhere, I just couldn't find it by quick googling.
Ask Espressif for the Linux parts? What do I exactly need? What's missing?
I don't think I need to change anything else besides the core variant headers. I'll try to build it and give it a test in QEMU. Please tell me if I need to change anything else.
I think I found the missing headers.
Yeah, that's correct.
tell me if I need to change anything else.
For the linux kernel itself I think that's all.
For the linux kernel itself I think that's all.
I don't need to change anything else at all? If so, then what about buildroot?
I don't need to change anything else at all?
Check FLASH cache setup in the bootloader and rebuild it with the esp32s3 compiler.
what about buildroot?
No changes, just rebuild with the esp32s3 compiler.
I don't need to change anything else at all?
Check FLASH cache setup in the bootloader and rebuild it with the esp32s3 compiler.
what about buildroot?
No changes, just rebuild with the esp32s3 compiler.
So all I need to do is switch to the ESP32-S3 compiler?
There were problems when compiling Linux, so I don't think it'll work for now.
I have compiled Linux for the ESP32 S3 successfully. I don't think I've done no more than importing the missing headers, and adding a DTS.
Keep in mind that I've not tried booting it yet, because QEMU would need a port for the ESP32 as well(at least the correct memory mapping). Also the bootloader currently uses ESP32-specific ROM functions, and so currently doesn't work on the ESP32 S3.
There were problems when compiling Linux, so I don't think it'll work for now.
@zap8600 there shouldn't be any if things are configured correctly. I can take a look if you share the build command, build log and the source tree.
I don't think I've done no more than importing the missing headers, and adding a DTS.
Can you provide the DTS?
I'm currently trying to do the same.
I've gotten Linux to compile, and I've added a new DTS for the ESP32 S3.
There are some more differences between the ESP32 and the S3.
You'll at least need to adjust the hardcoded values in arch/xtensa/platforms/esp32/setup.c to setup the watchdog timer(I've commented out that section for the moment).
Here's the DTS I've used:
// SPDX-License-Identifier: GPL-2.0
/dts-v1/;
/ {
compatible = "esp32";
#address-cells = <1>;
#size-cells = <1>;
interrupt-parent = <&pic>;
chosen {
bootargs = "earlycon=esp32uart,mmio32,0x60000000,115200n8 console=ttyS0,115200n8 debug no_hash_pointers ";
};
aliases {
serial0 = "/soc/serial@60000000";
};
memory@0 {
device_type = "memory";
reg = <0x3D000000 0x00200000>;
};
cpus {
#address-cells = <1>;
#size-cells = <0>;
cpu@0 {
compatible = "cdns,xtensa-cpu";
reg = <0>;
clocks = <&osc>;
};
};
clocks {
osc: clk160 {
#clock-cells = <0>;
compatible = "fixed-clock";
clock-frequency = <160000000>;
};
apb: clk80 {
#clock-cells = <0>;
compatible = "fixed-clock";
clock-frequency = <80000000>;
};
};
pic: pic {
compatible = "cdns,xtensa-pic";
/* one cell: internal irq number,
* two cells: second cell == 0: internal irq number
* second cell == 1: external irq number
*/
#interrupt-cells = <2>;
interrupt-controller;
};
soc {
#address-cells = <1>;
#size-cells = <1>;
compatible = "simple-bus";
interrupt-parent = <&intc>;
ranges;
intc: intc@600C2000 {
compatible = "esp,esp32-intc";
reg = <0x600C2000 0x114>;
/* first cell: interrupt matrix row
* second cell: internal xtensa core IRQ number to connect to
*/
#interrupt-cells = <2>;
interrupt-controller;
interrupt-parent = <&pic>;
};
serial0: serial@60000000 {
device_type = "serial";
compatible = "esp,esp32-uart";
reg = <0x60000000 0x80>;
interrupts = <34 1>; /* UART0, external IRQ1 */
clocks = <&apb>;
};
flash: flash@42800000 {
#address-cells = <1>;
#size-cells = <1>;
compatible = "mtd-rom";
reg = <0x42800000 0x00400000>;
bank-width = <2>;
device-width = <2>;
partition@40000 {
label = "linux";
reg = <0x00040000 0x001c0000>;
};
partition@200000 {
label = "data";
reg = <0x00200000 0x00200000>;
};
};
};
};
You also need a new bootloader(the current one uses ESP32 specific ROM functions). I'm still unsure about some things though. The DTS and bootloader assumes the following about the memory mapping:
Physical memory on the SPI flash:
0x00040000 - flash - Start of Linux partition on flash(1.75MB)
0x00200000 - flash - End of Linux partition
ESP-internal memory:
0x4037a000 - Internal SRAM1 - TODO: Used as vector base CONFIG_VECTORS_ADDR. Is this ok?
External memory, accessed through the MMU:
0x3C000000 - dcache - start of external memory region
0x3C800000 - dcache - flash_d_vaddr - start of Linux partition SPI Flash, mapped in linux_boot_main.c
0x3D000000 - dcache - start of PSRAM memory, mapped by esp-idf psram component
0x42000000 - icache - start of external memory region
0x42800000 - icache - flash_i_vaddr - start of Linux partition SPI Flash, mapped in linux_boot_main.c
Specifically, 0x4037a000 is just some "free" memory in SRAM1, and used as CONFIG_VECTORS_ADDR. Is this sensible? I'm not sure what that configuration option does, I assume it's some location for ISR-like things(vectors) that always need to be available? Or is this some hardware-hardcoded vector base address?
My bootloader currently looks like this: (I'm not sure if the bootloader works as expected, as I have not seen it boot anything useful yet, and it certainly has some bugs)
/* Linux boot Example
This example code is in the Public Domain (or CC0 licensed, at your option.)
Unless required by applicable law or agreed to in writing, this
software is distributed on an "AS IS" BASIS, WITHOUT WARRANTIES OR
CONDITIONS OF ANY KIND, either express or implied.
*/
#include <stdio.h>
#include "sdkconfig.h"
#include "freertos/FreeRTOS.h"
#include "freertos/task.h"
#include "rom/ets_sys.h"
#include "hal/mmu_hal.h"
#include "hal/cache_ll.h"
#include "hal/cache_hal.h"
#include "soc/reg_base.h"
#include "soc/dport_reg.h"
#include "soc/dport_access.h"
// debug printf functions that don't use any MMU mapped functions.
#define UART_BASE 0x60000000
#define UART_FIFO_REG 0x00
#define UART_STATUS_REG 0x1c
#define UART_TXFIFO_CNT_MASK 0x00ff0000
static void IRAM_ATTR writel(uint32_t value, volatile uint32_t* addr)
{
*(volatile uint32_t *)addr = value;
}
static uint32_t IRAM_ATTR readl(volatile uint32_t* addr)
{
return (*(volatile uint32_t *)addr);
}
static void IRAM_ATTR dbg_echo(const char *s)
{
volatile void *base = (volatile void *) UART_BASE;
while ((readl(base + UART_STATUS_REG) & UART_TXFIFO_CNT_MASK) != 0) {
;;
}
while (*s) {
if (*s == 0x0a) {
writel(0x0d, base + UART_FIFO_REG);
}
writel(*s, base + UART_FIFO_REG);
++s;
}
}
uint32_t flash_d_vaddr = 0x3C800000;
uint32_t flash_i_vaddr = 0x42800000;
uint32_t flash_paddr = 0x00040000;
uint32_t flash_size = 0x003c0000;
// start the APP cpu with specified entry address
static void IRAM_ATTR boot_app_cpu(uint32_t entry_addr) {
// check if the APP CPU is enabled.
if (REG_GET_BIT(SYSTEM_CORE_1_CONTROL_0_REG, SYSTEM_CONTROL_CORE_1_CLKGATE_EN)) {
dbg_echo("APP CPU already booted!\n");
REG_CLR_BIT(SYSTEM_CORE_1_CONTROL_0_REG, SYSTEM_CONTROL_CORE_1_CLKGATE_EN);
REG_SET_BIT(SYSTEM_CORE_1_CONTROL_0_REG, SYSTEM_CONTROL_CORE_1_RUNSTALL);
REG_SET_BIT(SYSTEM_CORE_1_CONTROL_0_REG, SYSTEM_CONTROL_CORE_1_RESETING);
}
// reset & enable APP CPU
dbg_echo("Booting APP cpu...\n");
REG_SET_BIT(SYSTEM_CORE_1_CONTROL_0_REG, SYSTEM_CONTROL_CORE_1_CLKGATE_EN);
REG_CLR_BIT(SYSTEM_CORE_1_CONTROL_0_REG, SYSTEM_CONTROL_CORE_1_RUNSTALL);
REG_SET_BIT(SYSTEM_CORE_1_CONTROL_0_REG, SYSTEM_CONTROL_CORE_1_RESETING);
REG_CLR_BIT(SYSTEM_CORE_1_CONTROL_0_REG, SYSTEM_CONTROL_CORE_1_RESETING);
ets_set_appcpu_boot_addr(entry_addr);
dbg_echo("Booted APP cpu!\n");
}
static void IRAM_ATTR jump(uint32_t entry_addr) {
dbg_echo("\n========== JUMPING ==========\n");
// jump to the entry address
asm volatile ("jx %0" :: "r"((uintptr_t)entry_addr));
}
static void IRAM_ATTR map_flash(uint32_t core_num)
{
uint32_t actual_d_size = 0;
uint32_t actual_i_size = 0;
dbg_echo("Mapping flash...\n");
// stop FreeRTOS
vTaskSuspendAll();
// disable the caches
cache_hal_disable(CACHE_TYPE_ALL);
// map the Linux flash region on the data bus
mmu_hal_map_region(core_num, MMU_TARGET_FLASH0, flash_d_vaddr, flash_paddr, flash_size, &actual_d_size);
//dbg_printf("mapped data vaddr=%.8lx paddr=%.8lx size=%lx actual_size=%.8lx\n", flash_d_vaddr, flash_paddr, flash_size, actual_d_size);
// map the Linux flash region on the instruction bus
mmu_hal_map_region(core_num, MMU_TARGET_FLASH0, flash_i_vaddr, flash_paddr, flash_size, &actual_i_size);
//dbg_printf("mapped inst vaddr=%.8lx paddr=%.8lx size=%lx actual_size=%.8lx\n", flash_i_vaddr, flash_paddr, flash_size, actual_i_size);
// enable the region on the data bus
cache_bus_mask_t bus_mask = cache_ll_l1_get_bus(core_num, flash_d_vaddr, actual_d_size);
cache_ll_l1_enable_bus(core_num, bus_mask);
// enable the region on the instruction bus
bus_mask = cache_ll_l1_get_bus(core_num, flash_i_vaddr, actual_i_size);
cache_ll_l1_enable_bus(core_num, bus_mask);
// re-enable the caches
cache_hal_enable(CACHE_TYPE_ALL);
dbg_echo("Mapped flash!\n");
}
static void IRAM_ATTR loop_forever() {
while(1) { ;; }
}
void app_main()
{
dbg_echo("\n========== ESP32 LINUX BOOTLOADER ==========\n");
if (0) {
// single-core mode
map_flash(0);
jump(flash_i_vaddr); // should never return
} else {
// dual-core mode
map_flash(1);
boot_app_cpu(flash_i_vaddr);
dbg_echo("\n========== PRO CPU STILL RUNNING ==========\n");
loop_forever();
}
}
Unfortunately, I've had some problems with using both the USB JTAG with OpenOCD, and running a ESP32 S3 in QEMU(which doesn't support it yet, even after trying to hack in a compatible memory mapping and core headers, but maybe I'm not capable enough for this). So debugging will be a little slower than possible, maybe.
EDIT: At least for the ESP32 S3 I have hardware with external PSRAM, if only 2MB. So I can and do test using that.
I'm finally going to come back to this. So from what I understand so far, the Linux kernel was able to be compiled. However, the bootloader needs to be rewritten because it uses functions that are only available on the ESP32, not the ESP32-S3. Your current bootloader hasn't booted Linux on a ESP32-S3 module yet. Is this correct? I may be able to put together some hardware to test this. If this does work, it would be faster than running it on the ESP32. With some more improvement, it could actually have use for low-power tasks that just can't be done outside of a Linux environment. Getting WiFi and a USB stack up would be great as well.
the bootloader needs to be rewritten because it uses functions that are only available on the ESP32, not the ESP32-S3
Yeah, it looks like it could use spi_flash_mmap or esp_partition_map. We're losing control over the chosen virtual address (anyway it will be the same for each boot), but getting portability. E.g. the following works for me on esp32:
const void *ptr;
spi_flash_mmap_handle_t handle;
spi_flash_mmap(0x00040000, 0x001c0000, SPI_FLASH_MMAP_INST, &ptr, &handle);
printf("ptr = %p\n", ptr);
spi_flash_mmap(0x00200000, 0x00200000, SPI_FLASH_MMAP_INST, &ptr, &handle);
printf("ptr = %p\n", ptr);
spi_flash_mmap_dump();
Or the same thing but with partitions:
static void IRAM_ATTR map_flash_and_go(void)
{
const void *ptr0, *ptr;
spi_flash_mmap_handle_t handle;
esp_partition_iterator_t it;
const esp_partition_t *part;
it = esp_partition_find(ESP_PARTITION_TYPE_ANY, ESP_PARTITION_SUBTYPE_ANY, "linux");
part = esp_partition_get(it);
esp_partition_mmap(part, 0, part->size, SPI_FLASH_MMAP_INST, &ptr0, &handle);
printf("ptr = %p\n", ptr0);
it = esp_partition_find(ESP_PARTITION_TYPE_ANY, ESP_PARTITION_SUBTYPE_ANY, "rootfs");
part = esp_partition_get(it);
esp_partition_mmap(part, 0, part->size, SPI_FLASH_MMAP_INST, &ptr, &handle);
printf("ptr = %p\n", ptr);
spi_flash_mmap_dump();
asm volatile ("jx %0" :: "r"(ptr0));
}
However I see a different issue here (as well as with my original code): although the mapping succeeds for the whole 4MB - 256KB FLASH range, not all of it is actually readable. The kernel is accessible, but part of the rootfs is not ):
the same thing but with partitions
Somehow it doesn't work on esp32s3, esp_partition_mmap returns ESP_ERR_NO_MEM. However it works when FLASH is mapped as SPI_FLASH_MMAP_DATA. Looks like a bug to me, I'm playing with the following workaround:
diff --git a/components/soc/esp32s3/include/soc/ext_mem_defs.h b/components/soc/esp32s3/include/soc/ext_mem_defs.h
index 9bb1d2d3125a..b2d04f49c87f 100644
--- a/components/soc/esp32s3/include/soc/ext_mem_defs.h
+++ b/components/soc/esp32s3/include/soc/ext_mem_defs.h
@@ -46,7 +46,7 @@ extern "C" {
//TODO, remove these cache function dependencies
#define CACHE_IROM_MMU_START 0
-#define CACHE_IROM_MMU_END Cache_Get_IROM_MMU_End()
+#define CACHE_IROM_MMU_END (0x1ff * 4)
#define CACHE_IROM_MMU_SIZE (CACHE_IROM_MMU_END - CACHE_IROM_MMU_START)
#define CACHE_DROM_MMU_START CACHE_IROM_MMU_END
@zap8600 You're right, AFAIK there is currently no working boot loader for the ESP32 S3. I've been busy with other things(and was somewhat held back by brocken openOCD, having the wrong board with too little RAM).
You also probably need to modify some memory-mapped IO addresses(e.g. UART, inerrupt controller, watchdog).
I having static address mappings would be great, since new addresses means recompiling the kernel every time the bootloader is built, and kernels aren't portable between bootloader versions.
@jcmvbkbc I don't think that's a bug; I think the implementation tries to map the required address space as separate "pages", and since there are only so many "page table entries", and FreeRTOS already allocated some, there are not enough left(AFAIK the ESP32 S3 only has enough "pages" on the IROM to map IROM once). I also don't know if the entire mapped region is "contigous". Also make sure to disable all the relevant " safety checks" in the esp-idf(memory protections, watchdog timers, etc.).
I think the solution is writing a bootloader that reads, then re-writes the complete page table to map the entire address range at once.
To successfully do that, a working version of OpenOCD would be helpful, as well as QEMU for the ESP32 S3.
@zap8600 You're right, AFAIK there is currently no working boot loader for the ESP32 S3. I've been busy with other things(and was somewhat held back by brocken openOCD, having the wrong board with too little RAM).
You also probably need to modify some memory-mapped IO addresses(e.g. UART, inerrupt controller, watchdog).
I having static address mappings would be great, since new addresses means recompiling the kernel every time the bootloader is built, and kernels aren't portable between bootloader versions.
I can probably make an ESP32-S3 board with the RAM required. However, that's not my main concern. Are you saying that there are addresses in the ESP32-S3's flash or memory that aren't static?
@jcmvbkbc I don't think that's a bug; I think the implementation tries to map the required address space as separate "pages", and since there are only so many "page table entries", and FreeRTOS already allocated some, there are not enough left(AFAIK the ESP32 S3 only has enough "pages" on the IROM to map IROM once). I also don't know if the entire mapped region is "contigous". Also make sure to disable all the relevant " safety checks" in the esp-idf(memory protections, watchdog timers, etc.).
Addressing this issue, something we can do (just for testing the port to the ESP32-S3), we can use a file system with only a basic /init file. It's only for testing though.
The workaround I mentioned here is working, the FLASH partitions get mapped. The code is in the https://github.com/jcmvbkbc/esp-idf/tree/linux-5.0.1
And with that bootloader I was able to boot this kernel on esp32s3 with 8MB PSRAM/8MB FLASH up to the switch to userspace.
The workaround I mentioned here is working, the FLASH partitions get mapped. The code is in the https://github.com/jcmvbkbc/esp-idf/tree/linux-5.0.1
And with that bootloader I was able to boot this kernel on esp32s3 with 8MB PSRAM/8MB FLASH up to the switch to userspace.
I'll look for your ESP32-S3 code in that repo and modify my scripts to build it.
I'll look for your ESP32-S3 code in that repo and modify my scripts to build it.
I've added a few notes on the whole process here
I've added a few notes on the whole process here
I have a question. Are directories like build-xtensa-fdpic-toolchain-esp32s3 and build-xtensa-fdpic-nothread-esp32s3 created by the user?
Are directories like build-xtensa-fdpic-toolchain-esp32s3 and build-xtensa-fdpic-nothread-esp32s3 created by the user?
Yes. I guess it's going to be easier if I just integrate toolchain building into the crosstool-NG, it makes things easier to reproduce, but slows down development a little. I'll get to it soon.
I've added a few notes on the whole process here
AWESOME! You're an absolute legend, kudos to you! Your previous post is what got me started on this ESP32 Linux journey, and the reason I wrote my first kernel code.
Is this using the more efficient XIP-aware FDPIC that can map from cramfs? My previous experiments all used bFLT, and the way worse memory usage that causes.
Just as an aside, since you're the expert, do you think it's plausible to run Linux and FreeRTOS "in parallel"(e.g. on separate cores?). As far as I'm aware, the APP/PRO CPUs should have a separate enough memory mapping. If somebody/me implements such a hack, this would make using the hardware from Linux way easier(e.g. just do a "RPC" call to FreeRTOS to hardware stuff). I'm specifically thinking about Wifi/Bluetooth which is using a binary blob driver for FreeRTOS making porting hard, and USB which seems underdocumented and hard to implement. I think the combination of a embedded RTOS with a full Linux could be very useful.
Thank you for your time and awesome contributions!
Is this using the more efficient XIP-aware FDPIC that can map from cramfs? My previous experiments all used bFLT, and the way worse memory usage that causes.
I do believe so.
As far as I'm aware, the APP/PRO CPUs should have a separate enough memory mapping. If somebody/me implements such a hack, this would make using the hardware from Linux way easier(e.g. just do a "RPC" call to FreeRTOS to hardware stuff). I'm specifically thinking about Wifi/Bluetooth which is using a binary blob driver for FreeRTOS making porting hard, and USB which seems underdocumented and hard to implement. I think the combination of a embedded RTOS with a full Linux could be very useful.
I'm somewhat familiar with using RPC systems, so I might be able to write a driver to help with this.
Is this using the more efficient XIP-aware FDPIC that can map from cramfs?
Yes, the code is now truly shared between the processes and it can run directly from FLASH. A few things are missing at the moment in the FDPIC support: there's no TLS and hence no NPTL, no -mlongcalls and hence limited distance between the functions that need to call each other, no stack unwinding and thus no C++, and its only hand-tested, i.e. I didn't run the compiler testsuite, so I don't really know what else might be broken.
do you think it's plausible to run Linux and FreeRTOS "in parallel"(e.g. on separate cores?)
It should be. But since the cores share some hardware (e.g. the interrupt matrix) the IDF code may have issues with that and need some modifications.
What if we use the binary blobs for WiFi to make WiFi drivers for Linux? There is a WiFi stack for the ESP32 and the ESP32-S3. There is also a binary blob for the BT stack for both the ESP32 and ESP32-S3 available, so we could make BT/BLE drivers for Linux. It's obviously available to the ESP32, but I'm not completely sure about for the ESP32-S3. I only think that because the repository says that it's a stack for BT/BLE, and the ESP32 only has BT, so the BLE stack would have to be for the ESP32-S2/3. For a reference, we can use this or this in order to make a basic structure. It won't work for actual communication though.