feat: Dynamic light sleep support for ESP32 platform

[m1nl]

fixes #6660

I've finally implemented the Feature Request I created a few months ago. Since then, I've been testing the dynamic light-sleep functionality and haven't encountered any issues.

In the initial version, legacy light-sleep support with the default Arduino framework could lead to problems after my changes. To address this, I had to rewrite and refactor parts of the code to ensure compatibility with builds compiled using the default PlatformIO Arduino framework.

I tested the updated implementation on a Heltec V3 board in the following scenarios:

• default Arduino, bluetooth, light-sleep disabled
• default Arduino, bluetooth, light-sleep enabled
• default Arduino, bluetooth, light-sleep enabled, force disable support for EXT wake-up
• customized Arduino, bluetooth, light-sleep disabled
• customized Arduino, bluetooth, light-sleep enabled
• customized Arduino, bluetooth, light-sleep enabled, force disable support for EXT wake-up
• customized Arduino, bluetooth disabled, light-sleep enabled
• customized Arduino, WiFi enabled, MQTT enabled, light-sleep enabled

There are a few topics that need discussion:

1. Power Management (PM) Support for ESP32 requires a custom variant of the Arduino framework. I’ve forked it from Espressif’s GitHub repository and am currently hosting it under my personal GitHub account. If these changes are accepted, I’d like to move the repository under the meshtastic organization.
2. In my opinion, re-enabling Bluetooth on the ESP32 without restarting is not possible, so NB PowerFSM state seems redundant. I'm open to restoring it if necessary, but I’d appreciate guidance on the intended logic behind this state.
3. I updated the platformio.ini files for the hardware variants I was able to test. I shared customized builds with others, who confirmed they were working as expected.
4. I updated the variant.h files for the hardware variants which do have 32kHz oscillator, according to Espressif docs, it helps with Bluetooth stability when dynamic light-sleep is enabled

Links to repositiories mentioned in 1) :

• https://github.com/m1nl/arduino-esp32
• https://github.com/m1nl/esp32-arduino-lib-builder

🤝 Attestations

• [x] I have tested that my proposed changes behave as described.
• [x] I have tested that my proposed changes do not cause any obvious regressions on the following devices:
  • [x] Heltec (Lora32) V3
  • [ ] LilyGo T-Deck
  • [x] LilyGo T-Beam
  • [ ] RAK WisBlock 4631
  • [x] Seeed Studio T-1000E tracker card
  • [ ] Other (please specify below)

[fifieldt]

This is a ton of work, thank you very much

[fifieldt]

If you have time .... I "maintain" the GPS code - just wondering what the impact might be there or whether we might need further extensions?

[m1nl]

AFAIK dynamic light-sleep in ESP32 platform makes the chip light-sleep when FreeRTOS waits for an event or there is an explicit vTaskDelay call (=~ sleep in Arduino). So most of the functionality shouldn't be impacted as Arduino runs all the code in one loop, which waits different intervals depending on the type of thread (using vTaskDelay under the hood). However, for critical modules or core functionality I added an explicit call to PowerFSM.trigger to exit LS state and allow for at least 300ms of uninterrupted processing time; another approach is to use "preflightSleepObserver" which prevents entering LS state if there is some work pending (like TX queue is not empty). I took a look at GPS code and I think we should exit LS state when we change state to GPS_ACTIVE and prevent entering LS as long as the state doesn't change. I'll take a look at the code and propose some changes soon.

[m1nl]

If there is a chance to have this feature merged, I'd appreciate getting some feedback on the additional repos to be hosted under meshtastic Github organization. As mentioned above, default Arduino framework provided by Espressif doesn't enable PM functionality at all (which is really bad IMO).

[m1nl]

@fifieldt changes committed; device should not go into light-sleep when GPS position is requested (=module is in GPS_ACTIVE state). I'm unable to test as I'm on vacation right now.

[m1nl]

hi,

1. Bumping my last question I'd appreciate getting some feedback on hosting additional repos under meshtastic Github organization. Default Arduino framework provided by Espressif doesn't enable PM functionality at all. I'm happy to move content of my repos to new ones. There are two repos to be created - arduino-esp32 and esp32-arduino-lib-builder

2. I took a look at the code and it reminded me I did change the WiFi code a bit - I'm enabling power saving by default (regardless of powersaving setting in module options); I think I should revoke that change to ensure compatibility with older versions; let me know what you think

3. I'd appreciate getting feedback on removing NB sleep state - as mentioned in the description for this PR 1) it doesn't work for ESP32 (no way to reenable BT after disabling it without reboot) 2) it doesn't make much sense for NRF platform. However I see it's used in Router so Bluetooth is disabled on NRF platform when device goes into this state - is this expected? IMO this creates some confusion as ESP32 FW behaves differently than NRF one.

[metala]

General critique about having a lot of assert(res == ESP_OK);:

While I've already make a boot loop with those asserts because of a wrong setting, the more concerning part is that assert() signifies an unrecoverable error and this should never happen, unless the system is in unstable/undefined state.

Surely some of the calls can fail in runtime or due to a wrong setting. Should we then crash the device? There is a known issue here that some ESP32 devices corrupt their device configuration when running on solar power. This not only increases the risk, but also cause the device to become irrecoverable, because of a wrong setting.

Example: esp_sleep_enable_timer_wakeup() can return ESP_ERR_INVALID_ARG if value is out of range. Therefore if the sleep time (sleepLeft = config.power.ls_secs * 1000LL - sleepTime;) gets somehow messed due to a wrong ls_secs setting, the device might be hard to restore through the radio.

I'd recommend an approach that would fail in a recoverable way.

[metala]

/work/.platformio/packages/toolchain-xtensa-esp32/bin/../lib/gcc/xtensa-esp32-elf/8.4.0/../../../../xtensa-esp32-elf/bin/ld: .pio/build/heltec-v2_1/firmware.elf section `.iram0.text' will not fit in region `iram0_0_seg'
/work/.platformio/packages/toolchain-xtensa-esp32/bin/../lib/gcc/xtensa-esp32-elf/8.4.0/../../../../xtensa-esp32-elf/bin/ld: IRAM0 segment data does not fit.
/work/.platformio/packages/toolchain-xtensa-esp32/bin/../lib/gcc/xtensa-esp32-elf/8.4.0/../../../../xtensa-esp32-elf/bin/ld: region `iram0_0_seg' overflowed by 776 bytes
collect2: error: ld returned 1 exit status

The linker fails on ESP32 (not S3).

[m1nl]

/work/.platformio/packages/toolchain-xtensa-esp32/bin/../lib/gcc/xtensa-esp32-elf/8.4.0/../../../../xtensa-esp32-elf/bin/ld: .pio/build/heltec-v2_1/firmware.elf section `.iram0.text' will not fit in region `iram0_0_seg'
/work/.platformio/packages/toolchain-xtensa-esp32/bin/../lib/gcc/xtensa-esp32-elf/8.4.0/../../../../xtensa-esp32-elf/bin/ld: IRAM0 segment data does not fit.
/work/.platformio/packages/toolchain-xtensa-esp32/bin/../lib/gcc/xtensa-esp32-elf/8.4.0/../../../../xtensa-esp32-elf/bin/ld: region `iram0_0_seg' overflowed by 776 bytes
collect2: error: ld returned 1 exit status

The linker fails on ESP32 (not S3).

This fails because the ESP32 platform has very limited IRAM available, and enabling PM capabilities pushes it beyond the limit. I’ll explore making the ESP-IDF components lighter to reduce memory usage without affecting Meshtastic functionality, but that may not be feasible. In the end, we may have to accept that dynamic light-sleep isn’t compatible with the ESP32.

In newer ESP-IDF versions (5.x compatible with Arduino 3.2.x), there’s a new sdkconfig option CONFIG_ESP_SYSTEM_ESP32_SRAM1_REGION_AS_IRAM, which makes more RAM available as IRAM, allowing the build to succeed. However, adopting this would be a breaking change for the project, since it requires moving from Arduino 2.x to 3.x. I’ve already created and tested such a build with the current Meshtastic version, but this shift is too significant to include in this PR.

[m1nl]

General critique about having a lot of assert(res == ESP_OK);:

While I've already make a boot loop with those asserts because of a wrong setting, the more concerning part is that assert() signifies an unrecoverable error and this should never happen, unless the system is in unstable/undefined state.

Surely some of the calls can fail in runtime or due to a wrong setting. Should we then crash the device? There is a known issue here that some ESP32 devices corrupt their device configuration when running on solar power. This not only increases the risk, but also cause the device to become irrecoverable, because of a wrong setting.

Example: esp_sleep_enable_timer_wakeup() can return ESP_ERR_INVALID_ARG if value is out of range. Therefore if the sleep time (sleepLeft = config.power.ls_secs * 1000LL - sleepTime;) gets somehow messed due to a wrong ls_secs setting, the device might be hard to restore through the radio.

I'd recommend an approach that would fail in a recoverable way.

Understood, I'll check if there is any assertion, which relies on user settings and make them fail safe. However I'd recommend to keep assertions for all calls, which use predefined values (board configuration, etc.). IMO this is the only way to validate the changes with all devices during alpha testing phase. Let me know what you think.

[akohlsmith]

I'm testing this on ESP32C3 and so far the testing is looking very good.

My only nitpick is that I think that there should be a comment in variants/esp32c3/heltec_esp32c3/variant.h (and others) which mentions the HAS_32768HZ macro and that it should be uncommented if your board has a 32kHz crystal. One of the big issues with supporting so many platforms is that these definitions tend to get buried in code.

e.g.

// uncomment this is your board has a 32kHz crystal
//#define HAS_32768HZ 1

[gxcreator]

I was digging in same direction. Any updates here?

[m1nl]

I was digging in same direction. Any updates here?

I had to switch context to another project. But I plan to revisit this - for now the main task here is to setup Arduino framework repos with Github actions. IMO code is stable and I'm using it for several months right now.

I'd appreciate testing, especially with less common devices - you can use this branch https://github.com/m1nl/meshtastic-firmware/tree/esp_light_sleep_release_stable ; it's rebased on latest stable release (2.7.11).

[gxcreator]

@m1nl I've tested that on Heltec wireless paper. For some reason it gives less runtime than original FW tweaked to 40MHz. 500mAh battery is ~5h on original firmware + 40MHz or ~4h on this one. Scenario is node jusr attached to Wi-Fi and sitting on the desk. Probably I am missing somethnig, will retest one more time.

[m1nl]

@m1nl I've tested that on Heltec wireless paper. For some reason it gives less runtime than original FW tweaked to 40MHz. 500mAh battery is ~5h on original firmware + 40MHz or ~4h on this one. Scenario is node jusr attached to Wi-Fi and sitting on the desk. Probably I am missing somethnig, will retest one more time.

Thanks! Can you share the changes you did to tune frequency down to 40MHz? Did you just replace 80MHz with 40MHz in the code? Did you remove the part which is forcing max frequency whenever WiFi is enabled?

Please note you need to enable Power Saving for a node to make it light-sleep; tunables (wake up time / etc.) is ignored.

[gxcreator]

@m1nl I am indeed just removed this workaround to max CPU freq when connected to WiFi and just set it to 40MHz. Will share exact change a bit later.

Hmm, I was not enabling Power Saving mode - maybe I got it wrong, but shouldn't tickless mode still function fine(i.e. putting cores to sleep on idle - when there is no freeRTOS process running) ?

[m1nl]

@m1nl I am indeed just removed this workaround to max CPU freq when connected to WiFi and just set it to 40MHz. Will share exact change a bit later.

Hmm, I was not enabling Power Saving mode - maybe I got it wrong, but shouldn't tickless mode still function fine(i.e. putting cores to sleep on idle - when there is no freeRTOS process running) ?

I wanted to keep this feature optional; PM config is not enabled unless Power Saving is requested by the user - moreover sleep code is managing PM locks to keep app running without any sleep intervals when busy.

Please enable Power Saving and keep screen on time low, retest and share the results :)

[iuvi7]

Volcano work! Any plans for implement for V4? THANKS

[m1nl]

thx, you guys motivated me to rebase and update this PR this week; for now you can try to use https://github.com/m1nl/meshtastic-firmware/tree/esp_light_sleep_release_stable - it should support V4; I'll check and let you know

[gxcreator]

Tried with power save mode, it was 10h+ instead of ~5h on original firmware + 40MHz !

[art-den]

Hi @m1nl. Thanks for your work!

I noticed some strange behavior on the tbeam. The initial data is as follows:

• BT is in use
• WiFi is disabled
• GPS is disabled
• Power saving mode is enabled
• Device supply voltage is 3.7 volts

Immediately after a reset, the device consumes 110 mA for several minutes. This high consumption either decreases on its own (to 56 mA) or if I connect to the device via Bluetooth (to 30 mA).

On original 2.6.11 it consumes 48 mA without power saving enabling

I also tested your firmware on the Heltec-V3 and it works perfectly there.

[m1nl]

Hi @m1nl. Thanks for your work!

I noticed some strange behavior on the tbeam. The initial data is as follows:

• BT is in use

• WiFi is disabled

• GPS is disabled

• Power saving mode is enabled

• Device supply voltage is 3.7 volts

Immediately after a reset, the device consumes 110 mA for several minutes. This high consumption either decreases on its own (to 56 mA) or if I connect to the device via Bluetooth (to 30 mA).

On original 2.6.11 it consumes 48 mA without power saving enabling

I also tested your firmware on the Heltec-V3 and it works perfectly there.

Thanks for testing, I can try to help if share the bootlog (serial output from the board).

[art-den]

I also tested your firmware on the Heltec-V3 and it works perfectly there.

I found strange thing for Heltec-V3 too. It consumes 10 mA when connected with my phone via BT and 19 mA when not connected. This is strange for me.

@m1nl how better to get bootlog for sharing?

[m1nl]

You can send me an email (check my GitHub profile).

If you didn't try it before, please disable Bluetooth completely (in node settings) and check the current draw; I suspect it may be BT modem looking for your phone causing more frequent wake-ups.

[iuvi7]

This one from Xiao Seed Studio quite popular as well, have few at shelf, will test it later by possibility from one 18650 :)

[m1nl]

This one from Xiao Seed Studio quite popular as well, have few at shelf, will test it later by possibility from one 18650 :)

Please keep in mind you may need to modify platformio.ini for a given variant to customize Arduino framework (https://github.com/m1nl/meshtastic-firmware/commit/5b28e11a75ef2b65f54f9698291f2f2e5f632123)

[art-den]

You can send me an email (check my GitHub profile).

Done. Main difference in logs with and without power safe enabling are too much [GPS] records on power safe mode. [GPS] records go about minute and this minute device consumes 110 mA. Then consumption decreases to 40 mA

PS: GPS is disabled in settings

[m1nl]

You can send me an email (check my GitHub profile).

Done. Main difference in logs with and without power safe enabling are too much [GPS] records on power safe mode. [GPS] records go about minute and this minute device consumes 110 mA. Then consumption decreases to 40 mA

PS: GPS is disabled in settings

From the logs, I can see that the system is actively searching for the GPS module but eventually stops because no GPS device is found. ("Disabled" in this context means that the system attempts to discover a GPS module but does not actually use it.)

Sleep functionality is quite limited while the device is searching for GPS, which could explain the increased power consumption. Was the GPS receiver disconnected from the board when power saving was enabled? If not, I’ll need to check whether the GPS module is being detected correctly when power saving mode is active.

[art-den]

Was the GPS receiver disconnected from the board when power saving was enabled?

On T-BEAM module GPS is always connected

[phaseloop]

@art-den power difference between BT connected/disconnected state is expected as there are different frame timings involved. This is something to be improved anyway with different PR.