Usermod libs matrix

[netmindz]

Run a build of each usermod if a PR contains a usermod change

Summary by CodeRabbit

• New Features

  • Introduced automated continuous integration for user modifications, enabling dynamic detection and building of usermod directories across multiple ESP32 hardware environments.
  • Added a new configuration supporting builds for various ESP32 variants, streamlining development and testing for usermods.

• Chores

  • Implemented caching strategies to optimize build times for dependencies and build artifacts.

[coderabbitai[bot]]

"""

Walkthrough

A new GitHub Actions workflow and a PlatformIO override configuration file have been introduced. The workflow automates continuous integration for user modifications under the usermods/ directory, dynamically discovering usermod directories and building them across multiple ESP32 hardware environments in parallel using the specified PlatformIO configuration. Additionally, the Si7021_MQTT_HA usermod's dependencies and README installation instructions were updated. The buzzer usermod's pin definition was made conditional.

Changes

Files/Groups	Change Summary
.github/workflows/usermods.yml	Added a GitHub Actions workflow for CI of usermods, including dynamic discovery and matrix builds for multiple ESP32 environments.
usermods/platformio_override.usermods.ini	Added a PlatformIO override configuration defining build environments for ESP32, ESP32-C3, ESP32-S2, and ESP32-S3 with usermod support.
usermods/Si7021_MQTT_HA/library.json	Added SPI and adafruit/Adafruit BusIO libraries as new dependencies alongside existing BME280 and Adafruit Si7021 dependencies.
usermods/Si7021_MQTT_HA/readme.md	Simplified installation instructions by removing explicit build flag and library dependency additions, referencing only custom_usermods.
usermods/buzzer/buzzer.cpp	Made USERMOD_BUZZER_PIN definition conditional on GPIO_NUM_32 macro, removing unconditional default and commented usage instructions.

Possibly related PRs

• wled/WLED#4620: Updates and fixes for the BME68X_v2 usermod, which is excluded from the new usermod CI workflow builds, indicating related usermod maintenance.

Suggested reviewers

• willmmiles """

---

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

• X
• Mastodon
• Reddit
• LinkedIn

🪧 Tips

Chat

There are 3 ways to chat with CodeRabbit:

• Review comments: Directly reply to a review comment made by CodeRabbit. Example:
  • I pushed a fix in commit <commit_id>, please review it.
  • Explain this complex logic.
  • Open a follow-up GitHub issue for this discussion.
• Files and specific lines of code (under the "Files changed" tab): Tag @coderabbitai in a new review comment at the desired location with your query. Examples:
  • @coderabbitai explain this code block.
  • @coderabbitai modularize this function.
• PR comments: Tag @coderabbitai in a new PR comment to ask questions about the PR branch. For the best results, please provide a very specific query, as very limited context is provided in this mode. Examples:
  • @coderabbitai gather interesting stats about this repository and render them as a table. Additionally, render a pie chart showing the language distribution in the codebase.
  • @coderabbitai read src/utils.ts and explain its main purpose.
  • @coderabbitai read the files in the src/scheduler package and generate a class diagram using mermaid and a README in the markdown format.
  • @coderabbitai help me debug CodeRabbit configuration file.

Support

Need help? Create a ticket on our support page for assistance with any issues or questions.

Note: Be mindful of the bot's finite context window. It's strongly recommended to break down tasks such as reading entire modules into smaller chunks. For a focused discussion, use review comments to chat about specific files and their changes, instead of using the PR comments.

CodeRabbit Commands (Invoked using PR comments)

• @coderabbitai pause to pause the reviews on a PR.
• @coderabbitai resume to resume the paused reviews.
• @coderabbitai review to trigger an incremental review. This is useful when automatic reviews are disabled for the repository.
• @coderabbitai full review to do a full review from scratch and review all the files again.
• @coderabbitai summary to regenerate the summary of the PR.
• @coderabbitai generate docstrings to generate docstrings for this PR.
• @coderabbitai generate sequence diagram to generate a sequence diagram of the changes in this PR.
• @coderabbitai resolve resolve all the CodeRabbit review comments.
• @coderabbitai configuration to show the current CodeRabbit configuration for the repository.
• @coderabbitai help to get help.

Other keywords and placeholders

• Add @coderabbitai ignore anywhere in the PR description to prevent this PR from being reviewed.
• Add @coderabbitai summary to generate the high-level summary at a specific location in the PR description.
• Add @coderabbitai anywhere in the PR title to generate the title automatically.

CodeRabbit Configuration File (.coderabbit.yaml)

• You can programmatically configure CodeRabbit by adding a .coderabbit.yaml file to the root of your repository.
• Please see the configuration documentation for more information.
• If your editor has YAML language server enabled, you can add the path at the top of this file to enable auto-completion and validation: # yaml-language-server: $schema=https://coderabbit.ai/integrations/schema.v2.json

Documentation and Community

• Visit our Documentation for detailed information on how to use CodeRabbit.
• Join our Discord Community to get help, request features, and share feedback.
• Follow us on X/Twitter for updates and announcements.

[netmindz]

Looks like the CI part is doing what I want now and trying to build each usermod for each env so we can easily see if anything breaks on certain envs

https://github.com/netmindz/WLED-MM/actions/runs/15655590001

@willmmiles

[willmmiles]

Looks like the CI part is doing what I want now and trying to build each usermod for each env so we can easily see if anything breaks on certain envs

https://github.com/netmindz/WLED-MM/actions/runs/15655590001

@willmmiles

OK, if I'm understanding correctly, the goal is to do full-usermod builds on all platforms, but only if the usermods themselves change; while the base ESP32 platform still does an all-usermod build for core firmware changes.

From a code standpoint, I think you can clean out a lot of the environment text by picking envs/boards we're already building for. Something like:

[env:usermods_esp32s3]
extends = esp32s3dev_16MB_opi
board_build.partitions = ${esp32.big_partitions}
custom_usermods = *

... is probably sufficient.

From a resource utilization standpoint, we can get a lot of savings if we can do the usermods builds in the same session as the reference build, thanks to the build cache. That's one of the big advantages of the new usermod system -- no extra -Ds in the build_flags means the build cache is fully compatible, since all the assembly happens at link time. So. if possible, building esp32s3dev_16MB_opi and usermods_esp32s3dev_16MB_opi in the same session should save half the build time.

[netmindz]

Looks like the CI part is doing what I want now and trying to build each usermod for each env so we can easily see if anything breaks on certain envs https://github.com/netmindz/WLED-MM/actions/runs/15655590001 @willmmiles

OK, if I'm understanding correctly, the goal is to do full-usermod builds on all platforms, but only if the usermods themselves change; while the base ESP32 platform still does an all-usermod build for core firmware changes.

Correct

From a code standpoint, I think you can clean out a lot of the environment text by picking envs/boards we're already building for. Something like:

[env:usermods_esp32s3]
extends = esp32s3dev_16MB_opi
board_build.partitions = ${esp32.big_partitions}
custom_usermods = *

... is probably sufficient.

Incorrect, you also need board, platform and board_build.flash_mode as a minimum. build_flags needed for the release name, custom_usermods needed to limit to only the single usermod so it's really only build_unflags and lib_deps that you can save

From a resource utilization standpoint, we can get a lot of savings if we can do the usermods builds in the same session as the reference build, thanks to the build cache. That's one of the big advantages of the new usermod system -- no extra -Ds in the build_flags means the build cache is fully compatible, since all the assembly happens at link time. So. if possible, building esp32s3dev_16MB_opi and usermods_esp32s3dev_16MB_opi in the same session should save half the build time.

If you look at the build output you will see that it's already making extensive use of the build cache so doesn't need to be in the same session for that. While we could do a single session that had all the envs rather than using a matrix CI build, saving things like checking out the code and the cache, we would lose the clear visibility of what usermods are failing

[willmmiles]

OK, if I'm understanding correctly, the goal is to do full-usermod builds on all platforms, but only if the usermods themselves change; while the base ESP32 platform still does an all-usermod build for core firmware changes.

Correct

How do you intend to handle inter-module dependencies? Not all modules can be built in isolation. PWM_fan is a good example - it depends on certain other usermods being included to build successfully. (Worse, it has an 'either a or b' type dependency, which can't be represented in pure PlatformIO lib_deps - it has to be analyzed via some custom script logic. :( )

From a code standpoint, I think you can clean out a lot of the environment text by picking envs/boards we're already building for. Something like:

[env:usermods_esp32s3]
extends = esp32s3dev_16MB_opi
board_build.partitions = ${esp32.big_partitions}
custom_usermods = *

... is probably sufficient.

Incorrect, you also need board, platform and board_build.flash_mode as a minimum. build_flags needed for the release name, custom_usermods needed to limit to only the single usermod so it's really only build_unflags and lib_deps that you can save

Those should be supplied by the environment being extended (in this example esp32s3dev_16MB_opi). I'm suggesting using extends over our fully specified target environments that include a board, eg. esp32s3dev_16MB_opi, instead of the "common code for all S3 boards" fragments. Since we're building for those boards anyways, should the env requirements for all boards change again (like WLED_RELEASE quoting), we won't have to maintain a second set of the extra stuff we need to go from common code to a full build.

If you look at the build output you will see that it's already making extensive use of the build cache so doesn't need to be in the same session for that. While we could do a single session that had all the envs rather than using a matrix CI build, saving things like checking out the code and the cache, we would lose the clear visibility of what usermods are failing

Ah, it's already sharing a build cache between jobs! That's good enough for me. (I should look at getting the HTML outputs in the cache, that'll save even more replication.)

[netmindz]

How do you intend to handle inter-module dependencies? Not all modules can be built in isolation. PWM_fan is a good example - it depends on certain other usermods being included to build successfully. (Worse, it has an 'either a or b' type dependency, which can't be represented in pure PlatformIO lib_deps - it has to be analyzed via some custom script logic. :( )

I don't at the moment is the simple answer. We would need to add something to these modules to define their dependencies and then process this when the CI generates the usermod environment.

In the short term I think we just blacklist any of these usermods from the CI to avoid false positives

From a code standpoint, I think you can clean out a lot of the environment text by picking envs/boards we're already building for. Something like:

[env:usermods_esp32s3]
extends = esp32s3dev_16MB_opi
board_build.partitions = ${esp32.big_partitions}
custom_usermods = *

... is probably sufficient.

Incorrect, you also need board, platform and board_build.flash_mode as a minimum. build_flags needed for the release name, custom_usermods needed to limit to only the single usermod so it's really only build_unflags and lib_deps that you can save

Those should be supplied by the environment being extended (in this example esp32s3dev_16MB_opi). I'm suggesting using extends over our fully specified target environments that include a board, eg. esp32s3dev_16MB_opi, instead of the "common code for all S3 boards" fragments. Since we're building for those boards anyways, should the env requirements for all boards change again (like WLED_RELEASE quoting), we won't have to maintain a second set of the extra stuff we need to go from common code to a full build.

What you might expect the behaviour of extends and what actually happens in platformio are very different things. This is why I said how you still need to define platform and board in every env, just try your proposed example and you will see what I mean

[willmmiles]

What you might expect the behaviour of extends and what actually happens in platformio are very different things. This is why I said how you still need to define platform and board in every env, just try your proposed example and you will see what I mean

Sorry, I should be more careful when sketching examples -- 'env:' is required in the extends to reference the correct section. The correct code should be:

[env:usermods_esp32s3]
extends = env:esp32s3dev_16MB_opi
custom_usermods = ${usermods.custom_usermods}

Suggested code is at https://github.com/willmmiles/WLED/commit/68540c60ebf131d8d64f24be0aece2732a52175e ; you can see that it works at https://github.com/willmmiles/WLED/actions/runs/15688280452

[willmmiles]

How do you intend to handle inter-module dependencies? Not all modules can be built in isolation. PWM_fan is a good example - it depends on certain other usermods being included to build successfully. (Worse, it has an 'either a or b' type dependency, which can't be represented in pure PlatformIO lib_deps - it has to be analyzed via some custom script logic. :( )

I don't at the moment is the simple answer. We would need to add something to these modules to define their dependencies and then process this when the CI generates the usermod environment.

In the short term I think we just blacklist any of these usermods from the CI to avoid false positives

This is ultimately why I went with the "build all the modules in one env" CI approach; I found that the number of modules that couldn't be combined in one build was smaller than the number of modules that had build time dependencies.

I agree that a blacklist is probably the best workaround. I don't think we want to invest in a more complex solution - longer term I am hoping to move towards out-of-tree usermods, so making sure the dep list is correct can be left as an exercise for the individual module's own CI. (That said, I'll need to make sure the example repo demonstrates a working CI -- something that runs nightly and checks for WLED updates, perhaps...)

On an quasi-related note, another objective with env:usermods was to keep any extra build requirements (necessary defines, etc.) in one place. I did try and migrate such requirements to defaults in the modules themselves whenever possible, but there's still one left (-DTOUCH_CS=9). We should see if we can make it go away, or consider setting up another PlatformIO variable to store it in so it can be used by both kinds of usermod test build.

[netmindz]

What you might expect the behaviour of extends and what actually happens in platformio are very different things. This is why I said how you still need to define platform and board in every env, just try your proposed example and you will see what I mean

Sorry, I should be more careful when sketching examples -- 'env:' is required in the extends to reference the correct section. The correct code should be:

[env:usermods_esp32s3]
extends = env:esp32s3dev_16MB_opi
custom_usermods = ${usermods.custom_usermods}

Suggested code is at willmmiles@68540c6 ; you can see that it works at https://github.com/willmmiles/WLED/actions/runs/15688280452

Thanks, that is applied now

[netmindz]

How do you intend to handle inter-module dependencies? Not all modules can be built in isolation. PWM_fan is a good example - it depends on certain other usermods being included to build successfully. (Worse, it has an 'either a or b' type dependency, which can't be represented in pure PlatformIO lib_deps - it has to be analyzed via some custom script logic. :( )

...

I agree that a blacklist is probably the best workaround. I don't think we want to invest in a more complex solution - longer term I am hoping to move towards out-of-tree usermods, so making sure the dep list is correct can be left as an exercise for the individual module's own CI. (That said, I'll need to make sure the example repo demonstrates a working CI -- something that runs nightly and checks for WLED updates, perhaps...)

Actually, there is perhaps a simpler option - given you have proved that you can build the usermods when both A and B dependencies are included, given the new usermod system is so much more user friendly, why not just include A and B as dependencies in library.json? Ok that technically that will make the bin slightly larger than a totally slimmed down version, but it solves out build problem and makes it easier for users too as they just depend on PWM fan for example and don't then need to work out which dependencies they need to also add to build

[willmmiles]

Actually, there is perhaps a simpler option - given you have proved that you can build the usermods when both A and B dependencies are included, given the new usermod system is so much more user friendly, why not just include A and B as dependencies in library.json?

For PWM_fan, If both deps are included, the internal ifdefs bind to Temperature and ignore sht. The module would have to be recoded to allow runtime selection. If the library.json lists both as deps, sht becomes unusable for production builds.