platformio-core
platformio-core copied to clipboard
platformio
PIO Remote builds tests and then executes them all, resulting in only the last-built test executing
Temporary solution
Please use -r, --force-remote option for pio remote test command.
Configuration
Operating system: MacOS X
PlatformIO Version (platformio --version): 4.1.0
Description of problem
When testing on remote platforms with Teensy, if there are many tests specified for a single environment, the remote agent builds all the tests and then runs them. Unfortunately this only results in a
See post here: https://community.platformio.org/t/pio-remote-testing-bug/10496
Steps to Reproduce
- Write some unit tests for the Teensy environment and have an environment run all of them
- Run them on a remote agent:
platformio remote --agent <name> test -e <env>
Actual Results
The code for the tests is built in series, but only the test built last is actually uploaded to the device for testing. See https://travis-ci.org/pathfinder-for-autonomous-navigation/CommonSoftware/jobs/610692665?utm_medium=notification&utm_source=github_status
Expected Results
The tests are executed one-by-one.
If problems with PlatformIO Build System:
The content of platformio.ini:
[env:teensy]
platform = teensy
board = teensy35
framework = arduino
test_build_project_src = true
src_filter = +<*>
test_filter = *
upload_protocol = teensy-cli
Workaround
Use the --force-remote option in the remote test command.
It looks like this issue is still here. I am unable to run more than one unit test using PIO remote..
My test folder contains 3 tests, simple, complex, datatest.
Simple:
#include "unity.h"
#include <Arduino.h>
void setUp(void) {
}
void tearDown(void) {
}
void test_function_simple(void) {
}
int runUnityTests(void) {
UNITY_BEGIN();
RUN_TEST(test_function_simple);
return UNITY_END();
}
int main(void) {
return runUnityTests();
}
void setup() {
Serial.begin(115200);
delay(2000);
runUnityTests();
}
void loop() {}
Complex:
#include "unity.h"
#include <Arduino.h>
void setUp(void) {
}
void tearDown(void) {
}
void test_function_complex(void) {
}
int runUnityTests(void) {
UNITY_BEGIN();
RUN_TEST(test_function_complex);
return UNITY_END();
}
int main(void) {
return runUnityTests();
}
void setup() {
Serial.begin(115200);
delay(2000);
runUnityTests();
}
void loop() {}
Datatest:
- This is my main test, with about 20 functions to be tested
int runUnityTests(void) {
UNITY_BEGIN();
RUN_TEST(test_function_bool);
RUN_TEST(test_function_uint8_t);
RUN_TEST(test_function_uint16_t);
RUN_TEST(test_function_uint32_t);
RUN_TEST(test_function_float);
RUN_TEST(test_function_string);
RUN_TEST(test_function_last);
RUN_TEST(test_function_walkall);
RUN_TEST(test_function_device2_send_bool);
RUN_TEST(test_function_device2_send_uint8_t);
RUN_TEST(test_function_device2_send_uint16_t);
RUN_TEST(test_function_device2_send_uint32_t);
RUN_TEST(test_function_device2_send_float);
RUN_TEST(test_function_device2_send_string);
RUN_TEST(test_function_callback_1);
RUN_TEST(test_function_callback_2);
RUN_TEST(test_function_callback_3);
RUN_TEST(test_function_callback_4);
return UNITY_END();
}
When I run the tests..
* Executing task in folder Test Board: platformio remote test --environment mega_A_LEDS
Building project locally
Verbosity level can be increased via `-v, -vv, or -vvv` option
Collected 3 tests
Processing embedded/test_simple in mega_A_LEDS environment
----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
Building...
------------------------------------------------------------------------------------------------------- mega_A_LEDS:embedded/test_simple [PASSED] Took 0.69 seconds -------------------------------------------------------------------------------------------------------
Processing embedded/test_datatest in mega_A_LEDS environment
----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
Building...
------------------------------------------------------------------------------------------------------ mega_A_LEDS:embedded/test_datatest [PASSED] Took 0.64 seconds ------------------------------------------------------------------------------------------------------
Processing embedded/test_complex in mega_A_LEDS environment
----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
Building...
------------------------------------------------------------------------------------------------------- mega_A_LEDS:embedded/test_complex [PASSED] Took 0.55 seconds -------------------------------------------------------------------------------------------------------
================================================================================================================ 0 test cases: 0 succeeded in 00:00:01.874 ================================================================================================================
Testing project remotely
Verbosity level can be increased via `-v, -vv, or -vvv` option
Collected 3 tests
Processing embedded/test_complex in mega_A_LEDS environment
--------------------------------------------------------------------------------
Building & Uploading...
avrdude: AVR device initialized and ready to accept instructions
avrdude: device signature = 0x1e950f (probably m328p)
avrdude: processing -U flash:w:.pio/build/mega_A_LEDS/firmware.hex:i
avrdude: reading input file .pio/build/mega_A_LEDS/firmware.hex for flash
with 2638 bytes in 1 section within [0, 0xa4d]
using 21 pages and 50 pad bytes
avrdude: preparing flash input for device bootloader
avrdude: writing 2638 bytes flash ...
Writing | ################################################## | 100% 1.01s
avrdude: 2638 bytes of flash written
avrdude: verifying flash memory against .pio/build/mega_A_LEDS/firmware.hex
Reading | ################################################## | 100% 0.67s
avrdude: 2638 bytes of flash verified
avrdude done. Thank you.
Testing...
If you don't see any output for the first 10 secs, please reset board (press reset button)
test/embedded/test_complex/complex.cpp:20: test_function_complex [PASSED]
--------- mega_A_LEDS:embedded/test_complex [PASSED] Took 6.74 seconds ---------
Processing embedded/test_simple in mega_A_LEDS environment
--------------------------------------------------------------------------------
Building & Uploading...
avrdude: AVR device initialized and ready to accept instructions
avrdude: device signature = 0x1e950f (probably m328p)
avrdude: processing -U flash:w:.pio/build/mega_A_LEDS/firmware.hex:i
avrdude: reading input file .pio/build/mega_A_LEDS/firmware.hex for flash
with 2638 bytes in 1 section within [0, 0xa4d]
using 21 pages and 50 pad bytes
avrdude: preparing flash input for device bootloader
avrdude: writing 2638 bytes flash ...
Writing | ################################################## | 100% 1.01s
avrdude: 2638 bytes of flash written
avrdude: verifying flash memory against .pio/build/mega_A_LEDS/firmware.hex
Reading | ################################################## | 100% 0.67s
avrdude: 2638 bytes of flash verified
avrdude done. Thank you.
Testing...
If you don't see any output for the first 10 secs, please reset board (press reset button)
test/embedded/test_complex/complex.cpp:20: test_function_complex [PASSED]
--------- mega_A_LEDS:embedded/test_simple [PASSED] Took 6.48 seconds ---------
Processing embedded/test_datatest in mega_A_LEDS environment
--------------------------------------------------------------------------------
Building & Uploading...
avrdude: AVR device initialized and ready to accept instructions
avrdude: device signature = 0x1e950f (probably m328p)
avrdude: processing -U flash:w:.pio/build/mega_A_LEDS/firmware.hex:i
avrdude: reading input file .pio/build/mega_A_LEDS/firmware.hex for flash
with 2638 bytes in 1 section within [0, 0xa4d]
using 21 pages and 50 pad bytes
avrdude: preparing flash input for device bootloader
avrdude: writing 2638 bytes flash ...
Writing | ################################################## | 100% 1.01s
avrdude: 2638 bytes of flash written
avrdude: verifying flash memory against .pio/build/mega_A_LEDS/firmware.hex
Reading | ################################################## | 100% 0.67s
avrdude: 2638 bytes of flash verified
avrdude done. Thank you.
Testing...
If you don't see any output for the first 10 secs, please reset board (press reset button)
test/embedded/test_complex/complex.cpp:20: test_function_complex [PASSED]
-------- mega_A_LEDS:embedded/test_datatest [PASSED] Took 6.74 seconds --------
=================================== SUMMARY ===================================
Environment Test Status Duration
------------- ---------------------- -------- ------------
mega_A_LEDS embedded/test_complex PASSED 00:00:06.744
mega_A_LEDS embedded/test_simple PASSED 00:00:06.484
mega_A_LEDS embedded/test_datatest PASSED 00:00:06.735
================== 3 test cases: 3 succeeded in 00:00:19.963 ==================
* Terminal will be reused by tasks, press any key to close it.
You can see that it has build all 3 tests but then it uploads the same one (complex) as that is the only test function returned by the test
writing 2638 bytes flash ...
Given that this issue is 5 years old is there any chance of fixing it?
