Arduino as ESP-IDF component

[anilanilb]

Hi, Copying issue from https://community.platformio.org/t/arduino-h-not-found-esp32-with-esp-idf-framework/1257 to platform-espressif32.

Currently I am building for ESP32 using PlatformIO + Atom with the ESP-IDF Framework. The Platform.ini file details are as below: [env:esp32dev] platform = espressif32 framework = espidf board = esp32dev

According to the docs https://github.com/espressif/arduino-esp3217 (section: Using as ESP-IDF component) the ESP-IDF Framework can have Arduino as component (and this works with ESP-IDF using make flash). Trying the same with PlatformIO, but compiler throws an error that Arduino.h cannot be found.

Is this an issue or do I need to make some changes which I am not aware of? hope somebody can help here.

------------- Log Start ----------------- Processing esp32dev (platform: espressif32; lib_ldf_mode: deep; board: esp32dev; framework: espidf)

Verbose mode can be enabled via -v, --verbose option Collected 0 compatible libraries Looking for dependencies... Project does not have dependencies Compiling .pioenvs\esp32dev\src\main.o

Generating partitions .pioenvs\esp32dev\partitions_table.bin Generating LD script .pioenvs\esp32dev\esp32_out.ld Compiling .pioenvs\esp32dev\bootloader\bootloader_start.o Compiling .pioenvs\esp32dev\bootloaderSupport\src\bootloader_flash.o Compiling .pioenvs\esp32dev\bootloaderSupport\src\bootloader_random.o Compiling .pioenvs\esp32dev\bootloaderSupport\src\efuse.o Compiling .pioenvs\esp32dev\bootloaderSupport\src\esp_image_format.o Compiling .pioenvs\esp32dev\bootloaderSupport\src\flash_encrypt.o Compiling .pioenvs\esp32dev\bootloaderSupport\src\flash_partitions.o src\main.c:8:52: fatal error: components/arduino/cores/esp32/Arduino.h: No such file or directory compilation terminated. *** [.pioenvs\esp32dev\src\main.o] Error 1 ------------- Log End ------------------

Regards, Andy

[williamesp2015]

CHECK THIS OUT FOR PLATFORMIO http://docs.platformio.org/en/latest/platforms/espressif32.html#using-arduino-framework-with-staging-version

[ivankravets]

https://github.com/espressif/arduino-esp3217 - broken link

Have you resolved this issue?

[bryanmtdt]

Having the same problem here, I want to add components to the ESP-IDF but it seems that the arduino core is not been loaded.

[radaiko]

same here

[ghost]

same here

[Redferne]

Anyone succeeded with building Arduino as IDF component using PlatformIO?

[ivankravets]

Could someone provide a source code to reproduce this issue?

[ivankravets]

@me-no-dev is it our issue? Does it work with Arduino IDE?

[BuzzRage]

same here, with WiFi.h, FS.h WString.h etc... It seems like something isn't well set. Maybe a path ? Maybe we have to precise lib_extra_dirs in the platformio.ini file ?

Everything works well in command line (setting paths manually, cloning arduino repo as component etc..), but with PlatformIO in Atom, it failed. I tried to move required libraries everywhere without success. even an "example" of WiFi.h didn't work.

Any news ?

[me21]

I think builder/frameworks/espidf.py script is not setting include and source paths for Arduino framework used as ESP-IDF component. In fact, it isn't searching for components subfolder in the project directory at all.

I believe that Arduino as ESP-IDF component would be so useful. As far as I understand, many of the components located in tools/sdk/lib subfolder of Arduino framework are located in components subfolder of ESP-IDF and are built from source, making modifying them a breeze.

[sticilface]

I am trying desperately to get this to work but the issues seem to be circular...

Are there any clear examples of how to use arduino as a component of the IDF.

I currently get fatal error: components/arduino/cores/esp32/Arduino.h: No such file or directory

If I start including the relevant directories from Arduino-esp32 It just keeps spiralling with more and more requirements..

What I am really after is a way of enabling the core-dump feature of the IDF so that I can debug a crash... I'm heavily dependent on the Arduino framework though so cannot port it all to the IDF.

Ideally I'd like the menuconfig options to be available in the arduino framework.. failing that as I understand it is WIP requiring the IDF to change to cmake... is there a way to get arduino to compile whilst using the idf?

I'm also a bit stuck as to how to configure menuconfig (where is the sdkconfig meant to me located for an idf project?)

[atanisoft]

@sticilface please see PR #193. You should only have the sources for YOUR project in your project tree. All IDF bits and arduino-esp32 bits should come from the PIO frameworks.

[MatheusAbel]

I'm having the same issue. I have an entire project which relies on 2/3 consolidated Arduino libraries and they doesn't have equivalent for espidf, so having Arduino as component working would save plenty of time of recoding those libs.

Also my project needs ULP, so on PlatformIO only espidf is available. I have tried to clone @atanisoft commit from 07-02-2019 but couldnt manage to make it work too. I'd really be thankful with a solution to this.

[atanisoft]

Also my project needs ULP, so on PlatformIO only espidf is available. I have tried to clone @atanisoft commit from 07-02-2019 but couldnt manage to make it work too. I'd really be thankful with a solution to this.

What issue did you run into? I don't have any ULP+Arduino+IDF examples to work with to test this combination.

[MatheusAbel]

Also my project needs ULP, so on PlatformIO only espidf is available. I have tried to clone @atanisoft commit from 07-02-2019 but couldnt manage to make it work too. I'd really be thankful with a solution to this.

What issue did you run into? I don't have any ULP+Arduino+IDF examples to work with to test this combination.

My scenario is: I have a project which was written based on arduino framework. Now i want to implement ULP code to read sensors which is already done and working in a 'test project' lets say.

As Platformio only supports espidf and I need to get my Arduino project working on idf the best way i've found is to use Arduino as a component o espidf as mentioned in this issue. I get the same error than @anilanilb, Arduino cant be found on platformIO build.

I've followed the steps from https://github.com/espressif/arduino-esp32/blob/master/docs/esp-idf_component.md to add Arduino component.

Also my sdkconfig.h contains

#define CONFIG_AUTOSTART_ARDUINO 1 #define CONFIG_ENABLE_ARDUINO_DEPENDS 1

Any clue how could i make it work? btw I'm using VisualCode.

[atanisoft]

As Platformio only supports espidf and I need to get my Arduino project working on idf the best way i've found is to use Arduino as a component o espidf as mentioned in this issue. I get the same error than @anilanilb, Arduino cant be found on platformIO build.

Start with an IDF only app in PIO and use the sdkconfig.h from arduino-esp32 only. Apply my PR #193 to your local copy and build as a normal IDF project in PIO.

If you see failures in resolving arduino specific libraries try lib_compat_mode=strict in your platformio.ini file.

[MatheusAbel]

@atanisoft

Start with an IDF only app in PIO and use the sdkconfig.h from arduino-esp32 only.

What do you mean by sdkconfig.h from arduino-esp32 only? Would it be starting a project in PIO with arduino-esp32 platform and use that sdkconfig.h? Sorry if i didn't get it.

[atanisoft]

Start with an IDF only project in PIO. Use the sdkconfig.h from arduino-esp32 since it includes the necessary definitions for arduino-esp32 (dependencies, etc). You don't need to add arduino-esp32 as a component when building with PIO and PR #193, the builder will take care of including the necessary components automatically with your project only containing YOUR files and not IDF/Arduino pieces.

[MatheusAbel]

I managed to get it working. Thanks to @atanisoft which guided me. One thing I was missing is that I dont have to add the Arduino component, its just change de skdconfig and PIO makes all the hard work.

Now starting to test if libs can work properly. I Will let you know.

[valeros]

Hi @anilanilb @williamesp2015 @bryanmtdt @radaiko @Redferne @BuzzRage @sticilface @MatheusAbel! Finally, we've found some spare time and added initial support for this feature. Many thanks to @atanisoft who did a great job and provided a classy PR that served as the basis for the current implementation. Two new examples blink and wifiscan from @atanisoft PR have been added to this repository. Now, the most of Arduino libraries should be compatible with IDF framework but keep in mind that this feature is still in an early experimental stage and IDF projects that need it must have a special version of sdkconfig.h file (see examples). To test it, just install the platform from the upstream branch.

[atanisoft]

@valeros I see a couple possible issues with the updated code:

1. arduino-esp32 has a few extra CCFLAGS which are not included in espidf.py when arduino is enabled, this may need to be reviewed further.
2. CPPDEFINES should also include F_CPU, ARDUINO_VARIANT and ARDUINO_BOARD
3. PIO_FRAMEWORK_ESP_IDF_ENABLE_EXCEPTIONS should likely be derived from CONFIG_CXX_EXCEPTIONS in sdk_params.

[MatheusAbel]

@valeros As I said here:

My scenario is: I have a project which was written based on arduino framework. Now i want to implement ULP code to read sensors which is already done and working in a 'test project' lets say.

It would be nice to have ULP support also on ArduinoFramework. I'm having a lot of trouble in order to use some Arduino libs on espidf since i2c driver from Arduino framework seems not to work well with espidf.

To clarify, i'm using an i2c sensor which library was written for Arduino and so on it uses Wire and TwoWire libs to make i2c work. Now i'm struggling to make the things happen. I'll tell here when i have an answer

[atanisoft]

I'm having a lot of trouble in order to use some Arduino libs on espidf since i2c driver from Arduino framework seems not to work well with espidf.

Can you come on https://gitter.im/espressif/arduino-esp32 and we can discuss the issues you are having with the arduino-esp32 I2C system? It should have no issues with the IDF i2c drivers (provided you are not trying to use both simultaneously)

[valeros]

@atanisoft Good points!

1. Just compared our build flags and flags from make log when Arduino is built as an IDF component and didn't notice any significant difference (for exceptions see 3). Am I missing something?
2. Same as (1), looks like the original build scripts don't add that macro, however F_CPU is already specified in our build.
3. PIO_FRAMEWORK_ESP_IDF_ENABLE_EXCEPTIONS config was added to allow users to enable exceptions even if they're using default sdkconfig.h. Do you think exceptions should be enabled by default for Arduino with ESP-IDF?

[atanisoft]

@valeros

1. These flags are in the PIO build script in arduino-esp32 and also in platform.txt used by Arduino ide.
2. Same as (1).
3. I don't think they should be enabled always but instead allow the user to define it via sdkconfig (or menuconfig if they opt for that). In my PR I added a check for the sdkconfig entry in addition to the PIO_FRAMEWORK_ESP_IDF_ENABLE_EXCEPTIONS flag.

[sticilface]

I've managed to get it to work.

I needed to switch to the --dev branch of PIO you have to use platform = https://github.com/platformio/platform-espressif32.git for the upstream,

and crucially the arduino #defines are missing so I needed to add these

  -D ESP32 
  -D ESP_PLATFORM 
  -D F_CPU=240000000L 
  -D ARDUINO=10805 
  -D ARDUINO_ARCH_ESP32 
  -D ARDUINO_VARIANT=\"d32_pro\" 
  "-D ARDUINO_BOARD=\"WEMOS LOLIN D32 PRO\"" 
  -D ARDUINO_ESP32_VER=\"2.10002.190628\" 

bizarrely not including " around the ARDUINO_BOARD resulted in this error

<command-line>:0:15: warning: missing terminating " character

which is a bit odd.

Since upgrading to --dev -/+ using the above upstream esp32 I can no longer get @atanisoft environment to work.

CONFIGURATION: https://docs.platformio.org/page/boards/espressif32/lolin_d32_pro.html
PLATFORM: Espressif 32 1.9.0 #a33c9fa > WEMOS LOLIN D32 PRO
HARDWARE: ESP32 240MHz, 320KB RAM, 4MB Flash
DEBUG: Current (esp-prog) External (esp-prog, iot-bus-jtag, jlink, minimodule, olimex-arm-usb-ocd, olimex-arm-usb-ocd-h, olimex-arm-usb-tiny-h, olimex-jtag-tiny, tumpa)
PACKAGES: toolchain-xtensa32 2.50200.80 (5.2.0), framework-espidf 3.30202.190627 (3.2.2), tool-esptoolpy 1.20600.0 (2.6.0)
Including Arduino-ESP32 from /Users/amelvin/.platformio/packages/framework-arduinoespressif32
LDF: Library Dependency Finder -> http://bit.ly/configure-pio-ldf
LDF Modes: Finder ~ chain, Compatibility ~ strict

It looks like might be using the wrong framework instead of his branch. It fails trying to find the libs wire + SPIFFS

[atanisoft]

@sticilface set framework to "espidf, arduino" (note there is a space after the comma, without it PIO fails). Also for your defines, did you try just " and not " ?

@valeros these missing defines are used by some libs and would be good to include for the Arduino IDF build.

[atanisoft]

@valeros difference in PIO based on framework line:

espidf, arduino:

xtensa-esp32-elf-g++ -o .pio\build\esp32\src\ConfigurationManager.cpp.o -c -std=c++11 -fno-rtti -fno-exceptions -std=gnu++11 -Werror=all -Wno-error=deprecated-declarations -Wextra -Wno-unused-parameter -Wno-sign-compare -Wno-error=unused-function -g3 -Os -Wall -nostdlib -Wpointer-arith -Wno-error=unused-but-set-variable -Wno-error=unused-variable -mlongcalls -ffunction-sections -fdata-sections -fstrict-volatile-bitfields -DWITH_POSIX -DIDF_VER=\"3.30202.190627\" -DPLATFORMIO=40000 -DARDUINO_ESP32_DEV -DCORE_DEBUG_LEVEL=ARDUHAL_LOG_LEVEL_INFO -DESP32_WIFI_MGR_MDNS_SOCKETPARAMS_LOG_LEVEL=VERBOSE -DESP32_WIFI_MGR_MDNS_QUERY_RES_LOG_LEVEL=VERBOSE -DARDUINOJSON_ENABLE_STD_STRING=1 -DARDUINOJSON_ENABLE_STD_STREAM=1 -D_GLIBCXX_USE_C99 -DLOCKED_LOGGING -DESP32 -DESP_PLATFORM -DF_CPU=240000000L -DHAVE_CONFIG_H -DMBEDTLS_CONFIG_FILE=\"mbedtls/esp_config.h\" -DGCC_NOT_5_2_0=0 @"C:\trains\ESP32CommandStation\.pio\build\esp32\longcmd-ab59af8556baaa0e11ed4fba0e94ddfd" src\ConfigurationManager.cpp

vs (arduino):

xtensa-esp32-elf-g++ -o .pio\build\esp32\src\ConfigurationManager.cpp.o -c -std=c++11 -fno-rtti -fno-exceptions -std=gnu++11 -g3 -Os -Wall -nostdlib -Wpointer-arith -Wno-error=unused-but-set-variable -Wno-error=unused-variable -mlongcalls -ffunction-sections -fdata-sections -fstrict-volatile-bitfields -Wno-error=deprecated-declarations -Wno-error=unused-function -Wno-unused-parameter -Wno-sign-compare -fstack-protector -fexceptions
-DPLATFORMIO=40000 -DARDUINO_ESP32_DEV -DCORE_DEBUG_LEVEL=ARDUHAL_LOG_LEVEL_INFO -DESP32_WIFI_MGR_MDNS_SOCKETPARAMS_LOG_LEVEL=VERBOSE -DESP32_WIFI_MGR_MDNS_QUERY_RES_LOG_LEVEL=VERBOSE -DARDUINOJSON_ENABLE_STD_STRING=1
-DARDUINOJSON_ENABLE_STD_STREAM=1 -D_GLIBCXX_USE_C99 -DLOCKED_LOGGING -DESP32 -DESP_PLATFORM -DF_CPU=240000000L -DHAVE_CONFIG_H -DMBEDTLS_CONFIG_FILE=\"mbedtls/esp_config.h\" -DARDUINO=10805 -DARDUINO_ARCH_ESP32 -DARDUINO_VARIANT=\"esp32\" "-DARDUINO_BOARD=\"Espressif ESP32 Dev Module\"" @"C:\trains\ESP32CommandStation\.pio\build\esp32\longcmd-8c6e2af4ed6f37384be531a108754512" src\ConfigurationManager.cpp

The entries: ARDUINO, ARDUINO_ARCH_ESP32, ARDUINO_VARIANT and ARDUINO_BOARD are missing. In the Arduino as IDF component make approach (not cmake) we can see a couple of these defined here, the cmake version is missing these (issue to be filed). A simple fix for this is below:

diff --git a/builder/frameworks/espidf.py b/builder/frameworks/espidf.py
index 6f3cf45..3a97378 100644
--- a/builder/frameworks/espidf.py
+++ b/builder/frameworks/espidf.py
@@ -391,11 +391,16 @@ def build_arduino_framework():
         ]
     )

-    envsafe = env.Clone()
-    envsafe.Append(
-        CPPDEFINES=[("ARDUINO", 10805), ("ARDUINO_ARCH_ESP32", 1)]
+    env.Append(
+        CPPDEFINES=[
+            ("ARDUINO", 10805),
+            "ARDUINO_ARCH_ESP32",
+            ("ARDUINO_VARIANT", '\\"%s\\"' % env.BoardConfig().get("build.variant").replace('"', "")),
+            ("ARDUINO_BOARD", '\\"%s\\"' % env.BoardConfig().get("name").replace('"', ""))
+        ]
     )

+    envsafe = env.Clone()
     arduino_libs = []
     if "build.variant" in env.BoardConfig():
         envsafe.Append(

By having the CPPDEFINES only in the envsafe clone that would allow these only to be used by the arduino code and not user code (or user libraries under lib)

[atanisoft]

@valeros found one other issue, LDF will need an update as well:

Processing esp32 (framework: espidf, arduino; platform: https://github.com/platformio/platform-espressif32.git; board: esp32dev)
-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
Verbose mode can be enabled via `-v, --verbose` option
CONFIGURATION: https://docs.platformio.org/page/boards/espressif32/esp32dev.html
PLATFORM: Espressif 32 1.9.0 #a186f66 > Espressif ESP32 Dev Module
HARDWARE: ESP32 240MHz, 320KB RAM, 4MB Flash
DEBUG: Current (esp-prog) External (esp-prog, iot-bus-jtag, jlink, minimodule, olimex-arm-usb-ocd, olimex-arm-usb-ocd-h, olimex-arm-usb-tiny-h, olimex-jtag-tiny, tumpa)
PACKAGES: toolchain-xtensa32 2.50200.80 (5.2.0), framework-arduinoespressif32 2.10002.190628 (1.0.2), framework-espidf 3.30202.190627 (3.2.2), tool-esptoolpy 1.20600.0 (2.6.0)
LDF: Library Dependency Finder -> http://bit.ly/configure-pio-ldf
LDF Modes: Finder ~ chain, Compatibility ~ strict
Looking for ESP32 Async UDP library in registry
Warning! Library `{'requirements': None, 'name': 'ESP32 Async UDP'}` has not been found in PlatformIO Registry.
You can ignore this message, if `{'requirements': None, 'name': 'ESP32 Async UDP'}` is a built-in library (included in framework, SDK). E.g., SPI, Wire, etc.
Found 35 compatible libraries

"ESP32 Async UDP" is a built in library from arduino-esp32 (here) and the library.json added by PIO only lists "arduino" as a framework so it is excluded due to not an exact match.

When LDF runs, can it split the framework and test each "framework" to see if the library is "valid" ?

[valeros]

Thanks, @atanisoft! I've updated the build script, now it should look better. As for the LDF, I tried a simple example from ESP32 Async UDP library and everything works as expected. I see there is a warning in your log, probably because you explicitly specified the dependency on the library. Could you please put together a simple project to reproduce this behavior, because LDF should automatically detect your dependency with the next configuration framework = arduino, espidf