platformio-core Hangs while scanning dependencies only in Docker container

[x] PlatformIO Core. If you’ve found a bug, please provide an information below.

Configuration

Operating system: Host machine: Windows 10 Docker container:

FROM python:3.8.7-slim

ENV DEBIAN_FRONTEND=noninteractive
ENV PROJ_DIR /firmware

RUN apt-get update -q \
  && apt-get install --no-install-recommends -yq \
  lcov valgrind git \
  && apt-get clean \
  && rm -rf /var/lib/apt/lists/*

RUN python3 -m pip install --upgrade pip setuptools
RUN python3 -m pip install platformio

# this is a really crappy way to get all the required environment installed in the container
COPY . $PROJ_DIR
RUN pio run --project-dir ${PROJ_DIR} || true
RUN pio system prune -f
RUN rm -r $PROJ_DIR

PlatformIO Version (platformio --version): Host: PlatformIO Core, version 5.2.3 Docker: PlatformIO Core, version 5.2.3

Description of problem

Compiling firmware on Docker takes a much longer time to scan the dependencies than a native Windows run. I know that the hypervisor can cause extra time but the running it with timestamps shows a 2.61s time on Windows vs 77.52 on Linux (~3000% difference).

Steps to Reproduce

If necessary I can try to create a minimal reproducible but you can use the above docker container and add it to a Platformio project and from the project directory:

docker build -t platformio-builder:latest .
docker run -v [pwd]:/firmware platformio run

Actual Results

Windows:

> filter timestamp {"$(Get-Date -Format o): $_"}; pio run -v | timestamp
...
2021-12-14T10:22:03.72: LDF: Library Dependency Finder -> https://bit.ly/configure-pio-ldf
2021-12-14T10:22:03.72: LDF Modes: Finder ~ chain, Compatibility ~ soft
2021-12-14T10:22:03.98: Found 87 compatible libraries
2021-12-14T10:22:03.98: Scanning dependencies...
2021-12-14T10:22:06.64: Dependency Graph
2021-12-14T10:22:06.64: |--

> docker run -v [pwd]:/firmware pio run -v | timestamp
...
10:24:25.59: LDF: Library Dependency Finder -> https://bit.ly/configure-pio-ldf
10:24:25.59: LDF Modes: Finder ~ chain, Compatibility ~ soft
10:24:33.58: Found 87 compatible libraries
10:24:33.58: Scanning dependencies...
10:25:51.10: Dependency Graph
10:25:51.11: |-- ...

Expected Results

The times are closer

If problems with PlatformIO Build System:

The content of platformio.ini: A little minimal since there's a lot in it

[platformio]
description = Firmware

[env] ;Common is done for all environments
monitor_speed = 115200
monitor_flags= --raw

lib_extra_dirs =
  ./shared-config

; ...

;==========ESP32===========
[env:esp32]
platform = espressif32
board = esp32dev
framework = arduino
build_type = debug
debug_tool = esp-prog
debug_init_break = tbreak setup

Source file to reproduce issue:

Insert here...

Dec 14 '21 09:12 fionn-r

Have you resolved this issue?

Feb 09 '22 17:02 ivankravets

Hi, no I haven't. I think it's some sort of issue with the LDF accessing Windows files from the container but even calling with verbose it doesn't give me any more information.

I can look into the problem a bit more but I'm not sure where to start looking

Feb 10 '22 08:02 fionn-r

Do you have any updates? Could you help us to debug this issue?

Apr 02 '22 20:04 ivankravets

Is there any way that I can add in extra logging so I can timestamp between

10:24:33.58: Scanning dependencies...
10:25:51.10: Dependency Graph

Apr 04 '22 08:04 fionn-r

Could you reproduce this issue with the latest PlatformIO Core?

pio upgrade --dev

Apr 04 '22 08:04 ivankravets

Ok, I tried with that and I still don't get any more logging inbetween:

2022-04-04T10:37:25.2087082+02:00: Found 104 compatible libraries
2022-04-04T10:37:25.2097729+02:00: Scanning dependencies...
2022-04-04T10:38:33.1718258+02:00: Dependency Graph

❯ pio --version
PlatformIO Core, version 5.3.0b3

Apr 04 '22 08:04 fionn-r

I've had a look into it and it could be to do with the way that Platformio is accessing the files when scanning the dependencies since it is accessing files local to Windows via the hypervisior.

I have seen that there's a overhead when accessing files via the hypervisior (e.g. docker, WSL) so that could have something to do with it. I guess my problem is that the LDF is scanning through a heap of files so the short term solution is to not use the LDF and manually include all the dependencies

Apr 04 '22 09:04 fionn-r

You can turn off it https://docs.platformio.org/en/latest/projectconf/section_env_library.html#lib-ldf-mode

Yes, LDF scans ALL C/C++ source/header files to resolve dependencies.

Apr 04 '22 09:04 ivankravets

Ok, I will try that workaround for now

Thanks for the help!

Apr 07 '22 08:04 fionn-r

Can you clarify how to bypass the LDF? I have dumped the following:

...
'__PIO_LIB_BUILDERS': [ <class 'piolib.UnknownLibBuilder'>('C:\\Users\\usr\\firmware-main-app\\lib\\ControllablePowerOutputs\\ControllablePowerOutputBase'),
                          <class 'piolib.UnknownLibBuilder'>('C:\\Users\\usr\\firmware-main-app\\lib\\ControllablePowerOutputs\\PowerOutput_24V'),
                          <class 'piolib.UnknownLibBuilder'>('C:\\Users\\usr\\firmware-main-app\\lib\\ControllablePowerOutputs\\PowerOutput_5V'),
                          <class 'piolib.ArduinoLibBuilder'>('C:\\Users\\usr\\firmware-main-app\\.pio\\libdeps\\esp32\\Adafruit ADS1X15'),
                          <class 'piolib.ArduinoLibBuilder'>('C:\\Users\\usr\\firmware-main-app\\.pio\\libdeps\\esp32\\Adafruit BusIO'),
                          <class 'piolib.ArduinoLibBuilder'>('C:\\Users\\usr\\firmware-main-app\\.pio\\libdeps\\esp32\\Adafruit MCP4725'),
                          <class 'piolib.ArduinoLibBuilder'>('C:\\Users\\usr\\firmware-main-app\\.pio\\libdeps\\esp32\\Adafruit MCP4728'),
                          <class 'piolib.ArduinoLibBuilder'>('C:\\Users\\usr\\firmware-main-app\\.pio\\libdeps\\esp32\\Adafruit NeoPixel'),
                          <class 'piolib.PlatformIOLibBuilder'>('C:\\Users\\usr\\firmware-main-app\\.pio\\libdeps\\esp32\\Embedded Template Library'),
                          <class 'piolib.PlatformIOLibBuilder'>('C:\\Users\\usr\\firmware-main-app\\.pio\\libdeps\\esp32\\NTPClient'),
                          <class 'piolib.ArduinoLibBuilder'>('C:\\Users\\usr\\.platformio\\packages\\framework-arduinoespressif32\\libraries\\ArduinoOTA'),
                          <class 'piolib.ArduinoLibBuilder'>('C:\\Users\\usr\\.platformio\\packages\\framework-arduinoespressif32\\libraries\\AsyncUDP'),
                          <class 'piolib.ArduinoLibBuilder'>('C:\\Users\\usr\\.platformio\\packages\\framework-arduinoespressif32\\libraries\\AzureIoT'),
                          <class 'piolib.ArduinoLibBuilder'>('C:\\Users\\usr\\.platformio\\packages\\framework-arduinoespressif32\\libraries\\BLE'),
                          <class 'piolib.ArduinoLibBuilder'>('C:\\Users\\usr\\.platformio\\packages\\framework-arduinoespressif32\\libraries\\BluetoothSerial'),
                          <class 'piolib.ArduinoLibBuilder'>('C:\\Users\\usr\\.platformio\\packages\\framework-arduinoespressif32\\libraries\\DNSServer'),
                          <class 'piolib.ArduinoLibBuilder'>('C:\\Users\\usr\\.platformio\\packages\\framework-arduinoespressif32\\libraries\\EEPROM'),
                          <class 'piolib.ArduinoLibBuilder'>('C:\\Users\\usr\\.platformio\\packages\\framework-arduinoespressif32\\libraries\\ESP32'),
                          <class 'piolib.ArduinoLibBuilder'>('C:\\Users\\usr\\.platformio\\packages\\framework-arduinoespressif32\\libraries\\ESPmDNS'),
                          <class 'piolib.ArduinoLibBuilder'>('C:\\Users\\usr\\.platformio\\packages\\framework-arduinoespressif32\\libraries\\FFat'),
                          <class 'piolib.ArduinoLibBuilder'>('C:\\Users\\usr\\.platformio\\packages\\framework-arduinoespressif32\\libraries\\FS'),
                          <class 'piolib.ArduinoLibBuilder'>('C:\\Users\\usr\\.platformio\\packages\\framework-arduinoespressif32\\libraries\\HTTPClient'),
                          <class 'piolib.ArduinoLibBuilder'>('C:\\Users\\usr\\.platformio\\packages\\framework-arduinoespressif32\\libraries\\HTTPUpdate'),
                          <class 'piolib.ArduinoLibBuilder'>('C:\\Users\\usr\\.platformio\\packages\\framework-arduinoespressif32\\libraries\\HTTPUpdateServer'),
                          <class 'piolib.ArduinoLibBuilder'>('C:\\Users\\usr\\.platformio\\packages\\framework-arduinoespressif32\\libraries\\NetBIOS'),
                          <class 'piolib.ArduinoLibBuilder'>('C:\\Users\\usr\\.platformio\\packages\\framework-arduinoespressif32\\libraries\\Preferences'),
                          <class 'piolib.ArduinoLibBuilder'>('C:\\Users\\usr\\.platformio\\packages\\framework-arduinoespressif32\\libraries\\SD'),
                          <class 'piolib.ArduinoLibBuilder'>('C:\\Users\\usr\\.platformio\\packages\\framework-arduinoespressif32\\libraries\\SD_MMC'),
                          <class 'piolib.ArduinoLibBuilder'>('C:\\Users\\usr\\.platformio\\packages\\framework-arduinoespressif32\\libraries\\SPI'),
                          <class 'piolib.ArduinoLibBuilder'>('C:\\Users\\usr\\.platformio\\packages\\framework-arduinoespressif32\\libraries\\SPIFFS'),
                          <class 'piolib.ArduinoLibBuilder'>('C:\\Users\\usr\\.platformio\\packages\\framework-arduinoespressif32\\libraries\\SimpleBLE'),
                          <class 'piolib.ArduinoLibBuilder'>('C:\\Users\\usr\\.platformio\\packages\\framework-arduinoespressif32\\libraries\\Ticker'),
                          <class 'piolib.ArduinoLibBuilder'>('C:\\Users\\usr\\.platformio\\packages\\framework-arduinoespressif32\\libraries\\Update'),
                          <class 'piolib.ArduinoLibBuilder'>('C:\\Users\\usr\\.platformio\\packages\\framework-arduinoespressif32\\libraries\\WebServer'),
                          <class 'piolib.ArduinoLibBuilder'>('C:\\Users\\usr\\.platformio\\packages\\framework-arduinoespressif32\\libraries\\WiFi'),
                          <class 'piolib.ArduinoLibBuilder'>('C:\\Users\\usr\\.platformio\\packages\\framework-arduinoespressif32\\libraries\\WiFiClientSecure'),
                          <class 'piolib.ArduinoLibBuilder'>('C:\\Users\\usr\\.platformio\\packages\\framework-arduinoespressif32\\libraries\\WiFiProv'),
                          <class 'piolib.ArduinoLibBuilder'>('C:\\Users\\usr\\.platformio\\packages\\framework-arduinoespressif32\\libraries\\Wire')],

And now I'm trying to include the dependencies manually.

I tried in n platformio.ini:

...
lib_ldf_mode = off
lib_deps =
  ${platformio.packages_dir}/framework-arduinoespressif32/libraries/AsyncUDP
  ${platformio.packages_dir}/framework-arduinoespressif32/libraries/WiFi
  ${platformio.packages_dir}/framework-arduinoespressif32/libraries/WiFiClientSecure
  ${platformio.packages_dir}/framework-arduinoespressif32/libraries/HTTPClient
  ${platformio.packages_dir}/framework-arduinoespressif32/libraries/HTTPUpdate
  ${platformio.packages_dir}/framework-arduinoespressif32/libraries/HTTPUpdateServer
  ${platformio.packages_dir}/framework-arduinoespressif32/libraries/ArduinoOTA
  ${platformio.packages_dir}/framework-arduinoespressif32/libraries/AzureIoT
  ${platformio.packages_dir}/framework-arduinoespressif32/libraries/BLE
  ${platformio.packages_dir}/framework-arduinoespressif32/libraries/BluetoothSerial
  ${platformio.packages_dir}/framework-arduinoespressif32/libraries/DNSServer
  ${platformio.packages_dir}/framework-arduinoespressif32/libraries/EEPROM
  ${platformio.packages_dir}/framework-arduinoespressif32/libraries/ESP32
  ${platformio.packages_dir}/framework-arduinoespressif32/libraries/ESPmDNS
  ${platformio.packages_dir}/framework-arduinoespressif32/libraries/FFat
  ${platformio.packages_dir}/framework-arduinoespressif32/libraries/FS
  ${platformio.packages_dir}/framework-arduinoespressif32/libraries/NetBIOS
  ${platformio.packages_dir}/framework-arduinoespressif32/libraries/Preferences
  ${platformio.packages_dir}/framework-arduinoespressif32/libraries/SD
  ${platformio.packages_dir}/framework-arduinoespressif32/libraries/SD_MMC
  ${platformio.packages_dir}/framework-arduinoespressif32/libraries/SPI
  ${platformio.packages_dir}/framework-arduinoespressif32/libraries/SPIFFS
  ${platformio.packages_dir}/framework-arduinoespressif32/libraries/SimpleBLE
  ${platformio.packages_dir}/framework-arduinoespressif32/libraries/Ticker
  ${platformio.packages_dir}/framework-arduinoespressif32/libraries/Update
  ${platformio.packages_dir}/framework-arduinoespressif32/libraries/WebServer
  ${platformio.packages_dir}/framework-arduinoespressif32/libraries/WiFiProv
  ${platformio.packages_dir}/framework-arduinoespressif32/libraries/Wire
  ${extra.lib_deps_base}
  etlcpp/Embedded Template Library @ ^19.3.5
  ./lib/ControllablePowerOutputs/ControllablePowerOutputBase
  ./lib/ControllablePowerOutputs/PowerOutput_24V
  ./lib/ControllablePowerOutputs/PowerOutput_5V
...

I then get issues that source files within ${platformio.packages_dir}/framework-arduinoespressif32/libraries/HTTPUpdate can't find each other. I tried using lib_extra_dirs but then got a different set of errors. What is the correct way to be doing it?

Apr 21 '22 12:04 fionn-r

platformio-core platformio-core copied to clipboard

Hangs while scanning dependencies only in Docker container

Configuration

Description of problem

Steps to Reproduce

Actual Results

Expected Results

If problems with PlatformIO Build System:

platformio-core
platformio-core copied to clipboard