[Feature Request]: Migrating ESP32 to ESP-IDF 5.5

[phaseloop]

Platform

ESP32

Description

Tracking issue for framework update.

Reasoning:

Due to PlatformIO politics, they support only old Arduino 2.x framework based on ESP-IDF V4 which has many bugs - including bluetooth stability and power management issues.

Example of gains from switching to version 5.5:

https://github.com/meshtastic/firmware/issues/8876

What we can do is to switch to pioarduino which is basically newest arduino-esp32 developed by Espressif but repacked to match Platformio metadata.

Switching to newer version will allow us to enable also FreeRTOS power saving features in the future. Initial work on this was done by @m1nl at https://github.com/m1nl/esp32-arduino-lib-builder

I have done experimental migration in my private branch and tested on Heltec V3. According to @m1nl there can be issues with different boards which will probably be solved on a case-by-case basis.

https://github.com/meshtastic/firmware/pull/7121#discussion_r2598373695

We have this already as ESP32-C6 support variant imports pioarduino instead of default one.

Known incompatibilities and migration work needed:

• migrate battery ADC reading to new ESP-IDF API.

This would be best done by moving battery ADC management to separate code in /platform/esp32 and call it from Power.cpp to avoid ton of #idefs

• remove Syslog class definition from source code and rework syslog calls to new library

It seems arduino-esp32 already imports external Syslog library (which is just newer version of what we have) and there are linker conflicts. There are minor API issues to fix between old and new one. Worst case scenario just keep using the old one but rename it.

• fix Paxcounter library as it's incompatible with new ESP-IDF API. It can be disabled and fixed last as paxcounter is not a critical feature.

Work plan (best as separate issues):

• implement ESP32 platform ADC management and battery readings as separate API living in /platform. API should be universal enough to be implemented by other platforms as well in the future.

• implement ESP-IDF V5 implementation of abovementioned API

• enable pioarduino for Heltec V3 boards as base testing platform

• work for each board one by one to figure out and fix problems

[mverch67]

Switching to V5.x also requires migrating all I2C clients (and those of external libraries) to the new I2C API as a mix of both old and new API in a single application is not allowed. I did some of this work here: https://github.com/meshtastic/firmware/pull/7180

[phaseloop]

Huh, that looks really great - what help do you need with finishing this part? I can extend and test it for Heltec V3.

[mverch67]

Huh, that looks really great - what help do you need with finishing this part? I can extend and test it for Heltec V3.

The CI build adaptations for pioarduino are missing. It requires also changes in https://github.com/meshtastic/firmware/blob/develop/bin/platformio-custom.py to support all pioarduino features.

[phaseloop]

What adaptations and features are you talking specifically?

[mverch67]

What adaptations and features are you talking specifically?

For the CI it obviously has to use pioarduino instead of platformio. Things that have been implemented in pioarduino python scripts is e.g. the possibility to select a certain bootloader which is not handled correctly by platformio-custom.py. You'll also notice when you run the environment in vscode that the vscode upload function does not work anymore due to an error inside this same script.

[phaseloop]

I'll pull your branch and see - when I did my proof of concept hacky migration - VS Code upload worked perfectly fine

[mverch67]

I'll pull your branch and see - when I did my proof of concept hacky migration - VS Code upload worked perfectly fine

did you replace (remove/disable) the vscode platformio plugin with the pioarduinoIDE plugin? Also, delete the entire old $HOME/.platformio directory before installation of the new plugin?

Edit: Or at least used pioarduino-core-installer to create the new pio environment?

[phaseloop]

@mverch67 I will need to kindly ask you to explain it to me like I'm five because for the last two days I've been trying to understand PlatformIO/Espressif drama and technical implications of that.

Starting with question: why would I need to use completetely different IDE fork because just one out of 30 or 40 platform plugins has some pissing contest with the vendor?

My current understanding is that switching to "pioarduino" aka arduino-esp32 3.x involves just using different platform link (let's ignore code rework and API changes). There is also arduino-lib-builder compiled version used by @m1nl without changing IDE plugin.

It seems we already do it:

https://github.com/meshtastic/firmware/blob/5671e9d96f76dc2310911864618b74cdcac196f8/variants/esp32c6/esp32c6.ini#L5

I followed exactly same path for my Heltec V3. I needed to nuke .platformio folder because it has some dependency issues but after that it compiled fine and I used regular Build + Upload VS Code features.

Edit: It seems C6 implementation was done by @caveman99 so probably he can share some experience about 3.x :)

[miky2k]

I feel that #8827 and #7918 should be related to old sdk, because on last tasmota (obviously without lora and display working) heltec v3 and v4 work like charm. Edit: Tasmota use arduino 2.x tooo

[mverch67]

@phaseloop

https://github.com/Jason2866/platform-espressif32

Although by the same author this is not pioarduino. The pioarduino framework is much more advanced, e.g. it allows configuring esp-idf and building the arduino-libs and does not require fixed pre-build libs as platformio. As such it comes with its own ecosystem (ide plugin, build scripts, platformio-core, etc.).

[phaseloop]

Yeah, you're right - I went back to my branch and it used pioarduino esp32 platform and after nuking platformio cache it forced installation of Pioarduino IDE.

But!

It seems linked repo - "Tasmota Espressif32" supports custom sdkconfig and other features pioarduino platform has:

https://github.com/Jason2866/platform-espressif32/tree/7848eb59b3c7dfa4edd5a48df6f2af7ff3f6d01b/examples/espidf-blink

Tasmota second repo (https://github.com/tasmota/platform-espressif32) even on main page states that they support hybrid compile.

So maybe if there is feature parity, we can just use this repo instead of reworking our whole workflow and depenencies just for one lib.

@miky2k - it seems tasmota releases Arduino 3.x too:

[Tasmota Arduino core 3.1.7 based on ESP IDF 5.3.4.251205-gcc15](https://github.com/tasmota/arduino-esp32/releases/tag/v3.1.7)

[miky2k]

@miky2k - it seems tasmota releases Arduino 3.x too:

[Tasmota Arduino core 3.1.7 based on ESP IDF 5.3.4.251205-gcc15](https://github.com/tasmota/arduino-esp32/releases/tag/v3.1.7)

Ok i feel upgrading to esp idf 5.3 should be high priority for project.

[phaseloop]

I agree. Yesterday I forked @mverch67 branch and used tasmota repo instead of pioarduino. It seems it compiled fine with some minor tweaks to the config. I don't have exact board so I will try to port those settings to Heltec V3.

I suppose most of sdkconfig can be made shared per CPU type to avoid bloat.

Then we can port each variant individually.

[MartinEmrich]

Got my Heltec V4 mostly working with latest pioarduino (including "Hybrid Compilation", enabling PM,...). The promised power saving features seem to work:

https://github.com/MartinEmrich/meshtastic-firmware/tree/heltec-v4-power/variants/esp32s3/heltec_v4_lp (see README.md).