env.AddBuildMiddleware() breaks library includes

[maxgerhardt]

What kind of issue is this?

• [X] PlatformIO Core. If you’ve found a bug, please provide an information below.

---

Configuration

Operating system: Windows 10 x64

PlatformIO Version (platformio --version): 6.1.12a1

Description of problem

Per documentation it is possible to use env.AddBuildMiddleware(callback, pattern) to modify the compile settings of specific files (SCons objects).

However, even returning env.Object(node), that is without any other options to modify compiler settings, in the callback to this function breaks PlatformIO's ability to identify the library dependencies or at least add the proper include compiler settings to the build of that specific file. Specifically, creating a minimal library MyTestLib in the lib/ folder with one .h and .cpp file implemeting a simple return 42; function and adding #include "MyTestLib.h" to the src/main.cpp will fail compilation if this extra_scripts = pre:test.py is added:

Import("env")

def pass_through(node):
    print("Node '%s'" % (node.name))
    #print("Type: " + str(type(node)))
    # pass through works fine
    #return node
    # wrapped in env.Object() fails, even when not modfying settings
    return env.Object(node)

env.AddBuildMiddleware(pass_through)

Note that return node itself works fine, but that eliminates any possibility to modify compiler settings for just that file.

Steps to Reproduce

1. Download https://github.com/maxgerhardt/pio-middleware-bug-repro
2. Open in VSCode
3. Attempt to compile (--> should fail)
4. Comment in return node
5. Build again (--> should work)

Actual Results

Scanning dependencies...
Dependency Graph
|-- MyTestLib (License: Unknown, Path: C:\Users\Max\temp\gran\lib\MyTestLib)
Node 'MyTestLib.cpp'
Node 'main.cpp'
Building in release mode
avr-g++ -o .pio\build\controllino_mega\src\main.cpp.o -c -fno-exceptions -fno-threadsafe-statics -fpermissive -std=gnu++11 -mmcu=atmega2560 -Os -Wall -ffunction-sections -fdata-sections -flto -DPLATFORMIO=60112 -DARDUINO_AVR_MEGA2560 -DF_CPU=16000000L -DARDUINO_ARCH_AVR -DARDUINO=10808 -IC:\Users\Max\.platformio\packages\framework-arduino-avr\cores\arduino -IC:\Users\Max\.platformio\packages\framework-arduino-avr\variants\Controllino_mega src\main.cpp
src\main.cpp:2:10: fatal error: MyTestLib.h: No such file or directory

Expected Results

Should compile fine since no compiler settings were modified (yet), node was just wrapped into an env.Object() like the documentation suggests is possible.

If problems with PlatformIO Build System:

The content of platformio.ini:

[env:controllino_mega]
platform = atmelavr
board = controllino_mega
framework = arduino
extra_scripts = pre:test.py

Source file to reproduce issue:

#include <Arduino.h>
#include "MyTestLib.h"

void setup() {
    Serial.begin(9600);
}

void loop() {
    int ret = MyTestLib_Function();
    Serial.println(ret);
    delay(1000);
}

Additional info

Relevant use-case per https://community.platformio.org/t/how-to-use-custom-debug-build-flags-per-file/35593.

[Kaiqgs]

I was magically able to get it working under one specific .pio build. Maybe it has information on why this is happening.

pio-middleware-bug-repro-working.zip

Log Output:

Processing controllino_mega (platform: atmelavr; board: controllino_mega; framew ork: arduino; extra_scripts: pre:test.py) CONFIGURATION: https://docs.platformio.org/page/boards/atmelavr/controllino_mega .html PLATFORM: Atmel AVR (5.0.0) > Controllino Mega HARDWARE: ATMEGA2560 16MHz, 8KB RAM, 248KB Flash DEBUG: Current (avr-stub) External (avr-stub, simavr) PACKAGES:

• framework-arduino-avr @ 5.2.0
• toolchain-atmelavr @ 1.70300.191015 (7.3.0) Node 'CDC.cpp' Node 'HardwareSerial.cpp' Node 'HardwareSerial0.cpp' Node 'HardwareSerial1.cpp' Node 'HardwareSerial2.cpp' Node 'HardwareSerial3.cpp' Node 'IPAddress.cpp' Node 'PluggableUSB.cpp' Node 'Print.cpp' Node 'Stream.cpp' Node 'Tone.cpp' Node 'USBCore.cpp' Node 'WInterrupts.c' Node 'WMath.cpp' Node 'WString.cpp' Node 'abi.cpp' Node 'hooks.c' Node 'main.cpp' Node 'new.cpp' Node 'wiring.c' Node 'wiring_analog.c' Node 'wiring_digital.c' Node 'wiring_pulse.S' Node 'wiring_pulse.c' Node 'wiring_shift.c' LDF: Library Dependency Finder -> https://bit.ly/configure-pio-ldf LDF Modes: Finder ~ chain, Compatibility ~ soft Found 6 compatible libraries Scanning dependencies... Dependency Graph |-- MyTestLib (License: Unknown, Path: /home/kags/Desktop/rogerkarlen/pio-middle ware-bug-repro/lib/MyTestLib) Node 'MyTestLib.cpp' Node 'main.cpp' Building in release mode MethodWrapper(["checkprogsize"], [".pio/build/controllino_mega/firmware.elf"]) Advanced Memory Usage is available via "PlatformIO Home > Project Inspect" RAM: [ ] 2.3% (used 188 bytes from 8192 bytes) Flash: [ ] 0.8% (used 2018 bytes from 253952 bytes) .pio/build/controllino_mega/firmware.elf : section size addr .data 22 8389120 .text 1996 0 .bss 166 8389142 .comment 17 0 .note.gnu.avr.deviceinfo 64 0 .debug_aranges 160 0 .debug_info 3859 0 .debug_abbrev 2942 0 .debug_line 732 0 .debug_str 998 0 Total 10956 ========================= [SUCCESS] Took 0.28 seconds =========================

[asd592]

@Kaiqgs I downloaded your code and tried to run it, but I still get the same error (see log output below). You mention that you got the code to compile under a specific version of PIO? What version did you use? And what IDE and OS are you using? I am running PIO as a plugin in VS Code on a Win11 Machine:

VS Code info: Version: 1.89.1 (user setup) Commit: dc96b837cf6bb4af9cd736aa3af08cf8279f7685 Date: 2024-05-07T05:13:33.891Z Electron: 28.2.8 ElectronBuildId: 27744544 Chromium: 120.0.6099.291 Node.js: 18.18.2 V8: 12.0.267.19-electron.0 OS: Windows_NT x64 10.0.22631

PIO info Plugin in VS Code: v3.3.3 Core: 6.1.15 Home: 3.4.4

OS / hardware info: Manufacturer: Framework Model: 13 OS: Win11 Pro 64-bit (10.0, Build 22631) CPU: 12th Gen Intel(R) Core(TM) i7-1280P RAM: 65536MB

Python info: Version: 3.12.3 64-bit installed from the official downloads page

What seems strange to me is that the script seems to be running with python version 3.11.7 despite me having not installed any 3.11 version on my system. I found this out by adding: import platform to the top of test.py and print("python version: %s" % platform.python_version()) before the line env.AddBuildMiddleware(pass_through)

My modified file: test_py_file.zip

However in the .platformio directory located on my C drive at C:\Users\User_Name.platformio\python3 there seems to be a python 311 dll and a python.exe that opens to a python 3.11.7 terminal. Does yout PIO installation use the same python version to run this file ?

Log output:

Processing controllino_mega (platform: atmelavr; board: controllino_mega; framework: arduino)

Verbose mode can be enabled via -v, --verbose option python version: 3.11.7 CONFIGURATION: https://docs.platformio.org/page/boards/atmelavr/controllino_mega.html PLATFORM: Atmel AVR (5.0.0) > Controllino Mega HARDWARE: ATMEGA2560 16MHz, 8KB RAM, 248KB Flash DEBUG: Current (avr-stub) External (avr-stub, simavr) PACKAGES:

• framework-arduino-avr @ 5.2.0
• toolchain-atmelavr @ 1.70300.191015 (7.3.0) Node 'CDC.cpp' Node 'HardwareSerial.cpp' Node 'HardwareSerial0.cpp' Node 'HardwareSerial1.cpp' Node 'HardwareSerial2.cpp' Node 'HardwareSerial3.cpp' Node 'IPAddress.cpp' Node 'PluggableUSB.cpp' Node 'Print.cpp' Node 'Stream.cpp' Node 'Tone.cpp' Node 'USBCore.cpp' Node 'WInterrupts.c' Node 'WMath.cpp' Node 'WString.cpp' Node 'abi.cpp' Node 'hooks.c' Node 'main.cpp' Node 'new.cpp'

Node 'wiring.c' Node 'wiring_analog.c' Node 'wiring_digital.c' Node 'wiring_pulse.S' Node 'wiring_pulse.c' Node 'wiring_shift.c' LDF: Library Dependency Finder -> https://bit.ly/configure-pio-ldf LDF Modes: Finder ~ chain, Compatibility ~ soft Found 6 compatible libraries Scanning dependencies... Dependency Graph |-- MyTestLib Node 'MyTestLib.cpp' Node 'main.cpp' Building in debug mode Compiling .pio\build\controllino_mega\src\main.cpp.o src\main.cpp:2:10: fatal error: MyTestLib.h: No such file or directory

---

• Looking for MyTestLib.h dependency? Check our library registry!
• CLI > platformio lib search "header:MyTestLib.h"
• Web > https://registry.platformio.org/search?q=header:MyTestLib.h

---

#include "MyTestLib.h" ^~~~~~~~~~~~~ compilation terminated. *** [.pio\build\controllino_mega\src\main.cpp.o] Error 1 ======================================================================================== [FAILED] Took 0.82 seconds ========================================================================================

Running the script again but this time using the -v flag gives the following output:

PS C:\Users\User_Name\Downloads\pio-middleware-bug-repro-working\pio-middleware-bug-repro-working> platformio.exe run -v Processing controllino_mega (platform: atmelavr; board: controllino_mega; framework: arduino; extra_scripts: pre:test.py; build_type: debug)

python version: 3.11.7 CONFIGURATION: https://docs.platformio.org/page/boards/atmelavr/controllino_mega.html PLATFORM: Atmel AVR (5.0.0) > Controllino Mega HARDWARE: ATMEGA2560 16MHz, 8KB RAM, 248KB Flash DEBUG: Current (avr-stub) External (avr-stub, simavr) PACKAGES:

• framework-arduino-avr @ 5.2.0
• toolchain-atmelavr @ 1.70300.191015 (7.3.0) Node 'CDC.cpp' Node 'HardwareSerial.cpp' Node 'HardwareSerial0.cpp' Node 'HardwareSerial1.cpp' Node 'HardwareSerial2.cpp' Node 'HardwareSerial3.cpp' Node 'IPAddress.cpp' Node 'PluggableUSB.cpp' Node 'Print.cpp' Node 'Stream.cpp' Node 'Tone.cpp' Node 'USBCore.cpp' Node 'WInterrupts.c' Node 'WMath.cpp' Node 'WString.cpp' Node 'abi.cpp' Node 'hooks.c' Node 'main.cpp' Node 'new.cpp' Node 'wiring.c' Node 'wiring_analog.c' Node 'wiring_digital.c' Node 'wiring_pulse.S' Node 'wiring_pulse.c' Node 'wiring_shift.c' LDF: Library Dependency Finder -> https://bit.ly/configure-pio-ldf LDF Modes: Finder ~ chain, Compatibility ~ soft Found 6 compatible libraries Scanning dependencies... Dependency Graph |-- MyTestLib (License: Unknown, Path: C:\Users\User_Name\Downloads\pio-middleware-bug-repro-working\pio-middleware-bug-repro-working\lib\MyTestLib) Node 'MyTestLib.cpp' Node 'main.cpp' Building in debug mode avr-g++ -o .pio\build\controllino_mega\src\main.cpp.o -c -fno-exceptions -fno-threadsafe-statics -fpermissive -std=gnu++11 -mmcu=atmega2560 -Wall -ffunction-sections -fdata-sections -flto -Og -g2 -ggdb2 -DPLATFORMIO=60115 -DARDUINO_AVR_MEGA2560 -DF_CPU=16000000L -DARDUINO_ARCH_AVR -DARDUINO=10808 -D__PLATFORMIO_BUILD_DEBUG__ "-IC:\Users\User_Name.platformio\packages\framework-arduino-avr\cores\arduino" *******************************************************************

• Looking for MyTestLib.h dependency? Check our library registry!
• CLI > platformio lib search "header:MyTestLib.h"
• Web > https://registry.platformio.org/search?q=header:MyTestLib.h

---

#include "MyTestLib.h" ^~~~~~~~~~~~~ compilation terminated. *** [.pio\build\controllino_mega\src\main.cpp.o] Error 1 ======================================================================================== [FAILED] Took 0.81 seconds ========================================================================================

[mathemaphysics]

I'm having the same problem. I don't understand why making no changes to a node's flags somehow breaks the includes path a dependency has on its dependencies. This has just taken up too much time to figure out.

[mathemaphysics]

I figured out the difference.

The Object type created by the following SCons code isn't the same thing as the Node that we've been given.

In SCons, Object is an actual object file, i.e. if your node is printed it will show a source file, e.g. main.cpp. But if you do obj = env.Object(node, ...), and print that you'll get ["main.cpp.o"]. It's an array of output objects.

If you return node, it has all the information to build, but if you return the env.Object(node, ...) you lose the information.

I don't know what PIO expects back from this. What you should do is call "build" or something to produce the build file inside of this middleware script. Bottom line is they're different types, returning node versus env.Object(node, ...).

[mathemaphysics]

Basically Arduino has made completely sure that I can't build anything with RTTI since their mbed OS doesn't allow it. And now I have to figure out how to enable it and disable it on a per-file basis. PlatformIO makes it impossible to do that now.

[mathemaphysics]

Also, you might not have noticed this, but the CPPPATH itself is clearly broken because spaces are not escaped.

At the very least this is related. You see a bunch of ['Adafruit', 'BusIO'] instead of ['Adafruit BusIO']. I still have no clue why anyone would put spaces in a file name or path.

[mathemaphysics]

So I've also figured out out that projenv is the only way to get the include paths. You have to generate them yourself if you want to get CPPPATH which is correct to add to your custom build.

Therefore, you cannot use AddBuildMiddleware to customize on a per-file basis and not break the build.

[mathemaphysics]

I got around this by copying the CPPPATH from the post:extra_script.py I wrote to just dump CPPPATH. I copied it into my preprocess.py. It isn't dynamic at all, but it will work. And I have manual control over individual file compile options.

[asd592]

@mathemaphysics you seem to have done quite some homework on this.... Sadly, I must say I don't quite understand how you managed to get it working in the end. Is there any chance you could upload your modified version of the sample project for me? Thanks