Serial monitor clears .pio/build/<env> thus stackdecoder cannot run

[TD-er]

When I start the serial monitor from PlatformIO, it clears the build dir of the active project. It already takes quite a while to start the serial monitor, which is annoying, but it also clears the just built files and thus the exception decoder cannot load the .elf file.

This is really annoying to say the least.

PS C:\GitHub\TD-er\ESPEasy> pio --version
PlatformIO Core, version 5.1.0

N.B. "Upload and Monitor" does work (sometimes), but build/upload and then press the Serial Monitor icon in the bottom bar does always clear the build.

[valeros]

Hi @TD-er ! I coudn't reproduce your problem. Could you please provide more details? What is your project configuration?

[TD-er]

Project configuration is here: https://github.com/letscontrolit/ESPEasy

I run it on Windows 10, VS code 1.52.1 I mainly build my "custom" builds while developing, which use Python scripts in the tools/PIO dir of the ESPEasy repository.

[valeros]

I believe it has nothing to do with the serial monitor, PlatformIO rebuilds the project even when I simply run the build task in a row. It seems the reason for that is because you're messing with the src folder in your extra scripts which implicitly changes the hashsum of the project and automatically triggers the project to rebuild entirely.

[TD-er]

You mean to generate the temp files (concatenated .cpp) to overcome the linking issue in PIO on Windows?

[valeros]

It looks like that temp files are exactly the reason.

[TD-er]

But why is this executed when I start the serial monitor process?

I was planning to update these Python files anyway to generate a test file first and compare it with the temp files to only copy them when it has changed to reduce build times. But still I don't understand why the serial monitor step does try to do all this work when it is starting. (takes a while too)

Probably related is that when I unfold another env when PIO is building, the build fails as it is not able to find setup() and loop() as the temp ESPEasy.ino.cpp file is gone. (this also happens a lot without touching any other env)

[valeros]

The serial monitor is part of platformio-core, so I suspect when you start the monitor, the build system evaluates your platformio.ini for changes and probably at this moment the build directory is already empty. In any case, generating dynamic files in the src folder is fundamentally a bad idea. Instead, you'd better generate that files in the build folder and add them to the build process via PIOBUILDFILES variable in one of your extra scripts.

[TD-er]

Well I had to do something to be able to build my project as I continously hit this max command line limit when linking.

I will look into the PIOBUILDFILES to see what can be done here. A first search doesn't find a lot of documentation on it. I also must make sure all of my code uses the same set of defines and I am not entirely sure it will if this PIOBUILDFILES does what I think it does.

[valeros]

I'm not sure if there is an example with PIOBUILDFILES in the docs as it's a somewhat internal feature, but here is a simple code snippet you can put in your concat_cpp_files.py:

...
tmp_cpp_file = os.path.join(env.subst("$BUILD_DIR"), '__tmpfile.cpp')
...
# Merge files
...
env.Append(PIOBUILDFILES=[tmp_cpp_file])
...

The files added to PIOBUILDFILES will be compiled with the same flags as the files from the src folder. You probably will need to adjust your includes according to new paths.

[TD-er]

Hmm it seems adding this will cause the file to be expected to be found in the platform files also.

*** [C:\users\gijsn\.platformio\platforms\espressif8266@src-eb7495f88eb0afa18fedff98bfb5e40f\builder\src\src\Commands_tmp\__tmpfile.cpp.o] Source `C:\users\gijsn\.platformio\platforms\espressif8266@src-eb7495f88eb0afa18fedff98bfb5e40f\builder\src\src\Commands_tmp\__tmpfile.cpp' not found, needed by target `C:\users\gijsn\.platformio\platforms\espressif8266@src-eb7495f88eb0afa18fedff98bfb5e40f\builder\src\src\Commands_tmp\__tmpfile.cpp.o'.

I also commented this one out in remove_concat_cpp_files.py

#env.AddPostAction("$BUILD_DIR/${PROGNAME}.bin", [clear_all_concat_cpp_files])

So I guess the PIOBUILDFILES should at least have the full path to these temp files? Or can it be specified when this should apply?

[valeros]

I provided an example above, the path passed to PIOBUILDFILES will be full anyway as you operate on that file when merging sources.

[TD-er]

Your suggestion only allows for a single __tmpfile.cpp. So I tried: tmp_cpp_file = os.path.join(env.subst("$BUILD_DIR"), os.path.join(cpp_path_out, '__tmpfile.cpp'))

But that also gives a number of build errors like these:

warning: Two different environments were specified for target C:\GitHub\TD-er\ESPEasy\.pio\build\custom_beta_ESP8266_4M1M\src\src\Commands_tmp\__tmpfile.cpp.o,
        but they appear to have the same action: ${TEMPFILE('$CXX -o $TARGET -c $CXXFLAGS $CCFLAGS $_CCCOMCOM $SOURCES','$CXXCOMSTR')}
File "C:\users\gijsn\.platformio\penv\lib\site-packages\platformio\builder\tools\platformio.py", line 78, in BuildProgram

warning: Two different environments were specified for target C:\GitHub\TD-er\ESPEasy\.pio\build\custom_beta_ESP8266_4M1M\src\src\ControllerQueue_tmp\__tmpfile.cpp.o,
        but they appear to have the same action: ${TEMPFILE('$CXX -o $TARGET -c $CXXFLAGS $CCFLAGS $_CCCOMCOM $SOURCES','$CXXCOMSTR')}
File "C:\users\gijsn\.platformio\penv\lib\site-packages\platformio\builder\tools\platformio.py", line 78, in BuildProgram

warning: Two different environments were specified for target C:\GitHub\TD-er\ESPEasy\.pio\build\custom_beta_ESP8266_4M1M\src\src\PluginStructs_tmp\__tmpfile.cpp.o,
        but they appear to have the same action: ${TEMPFILE('$CXX -o $TARGET -c $CXXFLAGS $CCFLAGS $_CCCOMCOM $SOURCES','$CXXCOMSTR')}
File "C:\users\gijsn\.platformio\penv\lib\site-packages\platformio\builder\tools\platformio.py", line 78, in BuildProgram

Building in release mode
warning: Two different environments were specified for target C:\GitHub\TD-er\ESPEasy\.pio\build\custom_beta_ESP8266_4M1M\src\src\WebServer_tmp\__tmpfile.cpp.o,
        but they appear to have the same action: ${TEMPFILE('$CXX -o $TARGET -c $CXXFLAGS $CCFLAGS $_CCCOMCOM $SOURCES','$CXXCOMSTR')}
File "C:\users\gijsn\.platformio\penv\lib\site-packages\platformio\builder\tools\platformio.py", line 78, in BuildProgram
Compiling .pio\build\custom_beta_ESP8266_4M1M\src\src\Commands_tmp\__tmpfile.cpp.o
Compiling .pio\build\custom_beta_ESP8266_4M1M\src\src\ControllerQueue_tmp\__tmpfile.cpp.o
Compiling .pio\build\custom_beta_ESP8266_4M1M\src\src\PluginStructs_tmp\__tmpfile.cpp.o
Compiling .pio\build\custom_beta_ESP8266_4M1M\src\src\WebServer_tmp\__tmpfile.cpp.o
In file included from src\src\Commands_tmp\../Commands/Blynk.h:4,
                 from src\src\Commands_tmp\__tmpfile.cpp:1:
c:\github\td-er\espeasy\src\espeasy_common.h:82:12: fatal error: ESP8266WiFi.h: No such file or directory

[valeros]

I've slightly modified your concat_cpp_files.py:

import os

Import("env")


def concat_cpp_files(path_to_concat):

    cpp_files = []

    cpp_path = path_to_concat

    cpp_path_out = '{}_tmp'.format(path_to_concat)

    if not os.path.exists(cpp_path_out):
        os.makedirs(cpp_path_out)

    tmp_cpp_file = os.path.join(env.subst("$BUILD_DIR"), '__tmpfile.cpp')

    print("\u001b[32m Concat {}/*.cpp to {} \u001b[0m".format(cpp_path, tmp_cpp_file))

    if os.path.exists(tmp_cpp_file):
        os.remove(tmp_cpp_file)

    for root, dirs, files in os.walk(cpp_path):
        for file in files:
            if file.endswith('.cpp'):
                fullname = os.path.join(root, file)
                cpp_files.append(fullname)
                print("\u001b[33m Add: \u001b[0m  {}".format(fullname))

    with open(tmp_cpp_file, 'wb') as newf:
        for filename in cpp_files:
            with open(filename, 'rb') as hf:
                newf.write(hf.read())
                newf.write(os.linesep.encode())  # append newline to separate files.

    env.Append(PIOBUILDFILES=[tmp_cpp_file])

    print("\u001b[32m ------------------------------- \u001b[0m")


concat_cpp_files('./src/src/Commands')
concat_cpp_files('./src/src/ControllerQueue')
concat_cpp_files('./src/src/PluginStructs')
concat_cpp_files('./src/src/WebServer')

[TD-er]

It still only stores the file in the 'root' of the build dir, so this means I need to change all includes in the files. This makes it impossible to compile it in Arduino IDE.

Compiling .pio\build\custom_beta_ESP8266_4M1M\__tmpfile.cpp.o
.pio\build\custom_beta_ESP8266_4M1M\__tmpfile.cpp:1:10: fatal error: ../WebServer/404.h: No such file or directory
    1 | #include "../WebServer/404.h"
      |          ^~~~~~~~~~~~~~~~~~~~
compilation terminated.
Compiling .pio\build\custom_beta_ESP8266_4M1M\src\src\Helpers\Misc.cpp.o
*** [.pio\build\custom_beta_ESP8266_4M1M\__tmpfile.cpp.o] Error 1

[valeros]

As an alternative you can improve your extra scripts with a bit more sophisticated algorithm that won't create/delete that temp files each time. Unfortunately, that's the best I can offer so your project won't be rebuilt sporadically.

[TD-er]

Yep, was thinking the same. to either store a list of file checksums used on the last tmp file generation, or generate one and check with the existing tmp file and only save it when it has changed.

What are the checks for PIO to decide to remove the built .elf file?

[valeros]

PlatformIO uses an incremental build system behind the scene, so the files (including the final .elf) are rebuilt only if the input files are not up to date. However, if a user changes platformio.ini or adds new files to the lib, include, src dirs then the entire build directory is cleaned and a project builds from scratch.

[TD-er]

I changed the concat_cpp_files.py file to this:

Import("env")
import os
import glob
import filecmp

def do_concat_cpp_files(cpp_files, output_file):
    with open(output_file, 'wb') as newf:
        for filename in cpp_files:
            with open(filename, 'rb') as hf:
                newf.write(hf.read())
                newf.write(os.linesep.encode())  # append newline to separate files.



def concat_cpp_files(path_to_concat):

    cpp_files = []

    cpp_path = path_to_concat

    cpp_path_out = '{}_tmp'.format(path_to_concat)

    if not os.path.exists(cpp_path_out):
        os.makedirs(cpp_path_out)

    tmp_cpp_file = os.path.join(cpp_path_out, '__tmpfile.cpp')
    tmp2_cpp_file = os.path.join(cpp_path_out, '__tmpfile.cpp.test')

    print("\u001b[32m Concat {}/*.cpp to {} \u001b[0m".format(cpp_path, tmp_cpp_file))


    for root, dirs, files in os.walk(cpp_path):
        for file in files:
            if file.endswith('.cpp'):
                fullname = os.path.join(root, file)
                cpp_files.append(fullname)
                print("\u001b[33m Add: \u001b[0m  {}".format(fullname))

    if os.path.exists(tmp_cpp_file):
        do_concat_cpp_files(cpp_files, tmp2_cpp_file)
            
        if filecmp.cmp(tmp_cpp_file, tmp2_cpp_file):
            print("\u001b[32m Files not changed, will not touch __tmpfile.cpp \u001b[0m")
            os.remove(tmp2_cpp_file)
        else:
            os.remove(tmp_cpp_file)
            os.rename(tmp2_cpp_file, tmp_cpp_file)
    else:
        do_concat_cpp_files(cpp_files, tmp_cpp_file)

    print("\u001b[32m ------------------------------- \u001b[0m")


concat_cpp_files('./src/src/Commands')
concat_cpp_files('./src/src/ControllerQueue')
concat_cpp_files('./src/src/PluginStructs')
concat_cpp_files('./src/src/WebServer')

Not the most elegant way, but it does seem to work at least generate the same tmp file and thus not touch the __tmpfile.cpp Let's hope the build process will not check the timestamp of the directory.

[TD-er]

Hmm not entirely sure if this is the way I would keep it. As a test I added an empty line in one of the files that would be included in one of these tmp files. It did indeed generate the file again and compiled just fine.

However, when I uploaded this build to the ESP unit I got all kinds of strange crashes. (reproducible crashes) The crash dump did seem to work and pointed to somewhere in the file that I changed (added a newline) Built it again and flashed and the ESP worked just fine.

So I will for sure need to remove all these temp files when I try to make a clean build. How do I add clear_all_concat_cpp_files to the "clean" job in PlatformIO?

Until these changes I used this: env.AddPostAction("$BUILD_DIR/${PROGNAME}.bin", [clear_all_concat_cpp_files])

But that's only carried out when a bin file is created and that's what we don't want to do right now.

Edit: Hmm I don't think I need to remove those __tmpfile.cpp files as their compiled object files are already deleted on a "clean" target.