RIOT
RIOT copied to clipboard
Published
20 hours ago •
RIOT-OS
RIOT-OS
.murdock: test 200 repeats of known flakey test
This is only a test to try to solve the occasional unrelated hash failure problem forcing CI reruns.
@MrKevinWeiss The test failed due to a syntax error. Murdock is idle, so it's good time to do some tests now.
Heh, but I was not idle. I will try to hit this on the weekend when there is not much traffic.
hmmm I will try it again, it looks like we are testing what we want we just didn't get a hit... Note the mobi6 was enabled before running this test...
This must be the only PR which we want to fail and keeps passing...
Maybe ccache is too optimistic, so repeated runs yield same result but cache miss causes hash mismatch?
I wonder if the failed results are getting overwritten with passing ones...
I will wait until the speedup #18423 is in before continuing with this as the feedback cycle is super long.
Theory-> something to do with gitcache (so far I have only seen it on esp bootloaders or packages or things that need to checkout other repos)
stm pkg
From #18433
-- running on worker riotbuild-0 thread 4, build number 544157.
make: Entering directory '/tmp/dwq.0.5632680420550714/c5bcaa217ff54478dfecb5ebe2842b82/tests/pkg_tflite-micro'
rm -rf /tmp/dwq.0.5632680420550714/c5bcaa217ff54478dfecb5ebe2842b82/build/pkg-build/flatbuffers
rm -rf /tmp/dwq.0.5632680420550714/c5bcaa217ff54478dfecb5ebe2842b82/build/pkg-build/gemmlowp
rm -rf /tmp/dwq.0.5632680420550714/c5bcaa217ff54478dfecb5ebe2842b82/build/pkg-build/ruy
rm -rf /tmp/dwq.0.5632680420550714/c5bcaa217ff54478dfecb5ebe2842b82/build/pkg-build/tflite-micro
make: Leaving directory '/tmp/dwq.0.5632680420550714/c5bcaa217ff54478dfecb5ebe2842b82/tests/pkg_tflite-micro'
make: Entering directory '/tmp/dwq.0.5632680420550714/c5bcaa217ff54478dfecb5ebe2842b82/tests/pkg_tflite-micro'
git-cache: cloning from cache. tag=commita9a295fecf3fbd5a4f571f53b01f63202a3e2113-67849
Building application "tests_pkg_tflite-micro" for "nucleo-l552ze-q" with MCU "stm32".
git-cache: cloning from cache. tag=commitfcf61cd086e02c7a1cf7535ffe30622d1f5d36aa-67791
git-cache: cloning from cache. tag=commite844ffd17118c1e17d94e1ba4354c075a4577b88-70078
git-cache: cloning from cache. tag=commit7ef39c5745a61f43071e699c6a96da41701ae59f-71777
git-cache: cloning from cache. tag=commit1501b574b74fd7877aba30aa9d8b667f41b139c3-73447
sha1sum /tmp/dwq.0.5632680420550714/c5bcaa217ff54478dfecb5ebe2842b82/tests/pkg_tflite-micro/tests/01-run.py /tmp/dwq.0.5632680420550714/c5bcaa217ff54478dfecb5ebe2842b82/build/tests_pkg_tflite-micro.bin > /tmp/dwq.0.5632680420550714/c5bcaa217ff54478dfecb5ebe2842b82/build/test-input-hash.sha1
text data bss dec hex filename
261616 632 13592 275840 43580 /tmp/dwq.0.5632680420550714/c5bcaa217ff54478dfecb5ebe2842b82/build/tests_pkg_tflite-micro.elf
make: Leaving directory '/tmp/dwq.0.5632680420550714/c5bcaa217ff54478dfecb5ebe2842b82/tests/pkg_tflite-micro'
make: Entering directory '/tmp/dwq.0.5632680420550714/c5bcaa217ff54478dfecb5ebe2842b82/tests/pkg_tflite-micro'
rm -rf /tmp/dwq.0.5632680420550714/c5bcaa217ff54478dfecb5ebe2842b82/build/pkg-build/flatbuffers
rm -rf /tmp/dwq.0.5632680420550714/c5bcaa217ff54478dfecb5ebe2842b82/build/pkg-build/gemmlowp
rm -rf /tmp/dwq.0.5632680420550714/c5bcaa217ff54478dfecb5ebe2842b82/build/pkg-build/ruy
rm -rf /tmp/dwq.0.5632680420550714/c5bcaa217ff54478dfecb5ebe2842b82/build/pkg-build/tflite-micro
make: Leaving directory '/tmp/dwq.0.5632680420550714/c5bcaa217ff54478dfecb5ebe2842b82/tests/pkg_tflite-micro'
make: Entering directory '/tmp/dwq.0.5632680420550714/c5bcaa217ff54478dfecb5ebe2842b82/tests/pkg_tflite-micro'
Building application "tests_pkg_tflite-micro" for "nucleo-l552ze-q" with MCU "stm32".
git-cache: cloning from cache. tag=commita9a295fecf3fbd5a4f571f53b01f63202a3e2113-97514
git-cache: cloning from cache. tag=commitfcf61cd086e02c7a1cf7535ffe30622d1f5d36aa-97458
git-cache: cloning from cache. tag=commite844ffd17118c1e17d94e1ba4354c075a4577b88-99609
git-cache: cloning from cache. tag=commit7ef39c5745a61f43071e699c6a96da41701ae59f-101711
git-cache: cloning from cache. tag=commit1501b574b74fd7877aba30aa9d8b667f41b139c3-104901
sha1sum /tmp/dwq.0.5632680420550714/c5bcaa217ff54478dfecb5ebe2842b82/tests/pkg_tflite-micro/tests/01-run.py /tmp/dwq.0.5632680420550714/c5bcaa217ff54478dfecb5ebe2842b82/build/tests_pkg_tflite-micro.bin > /tmp/dwq.0.5632680420550714/c5bcaa217ff54478dfecb5ebe2842b82/build/test-input-hash.sha1
text data bss dec hex filename
261728 632 13592 275952 435f0 /tmp/dwq.0.5632680420550714/c5bcaa217ff54478dfecb5ebe2842b82/build/tests_pkg_tflite-micro.elf
make: Leaving directory '/tmp/dwq.0.5632680420550714/c5bcaa217ff54478dfecb5ebe2842b82/tests/pkg_tflite-micro'
Hashes of binaries with and without Kconfig mismatch for tests/pkg_tflite-micro with nucleo-l552ze-q
Please check that all used modules are modelled in Kconfig and enabled
Input without KConfig:
039de8cf4c7c6517d7ef39857552fa60e2689529 /tmp/dwq.0.5632680420550714/c5bcaa217ff54478dfecb5ebe2842b82/tests/pkg_tflite-micro/tests/01-run.py
eaf5f1f9b629c9e76e843a4ea0272fc0ae6f9e79 /tmp/dwq.0.5632680420550714/c5bcaa217ff54478dfecb5ebe2842b82/build/tests_pkg_tflite-micro.bin
Input with KConfig:
039de8cf4c7c6517d7ef39857552fa60e2689529 /tmp/dwq.0.5632680420550714/c5bcaa217ff54478dfecb5ebe2842b82/tests/pkg_tflite-micro/tests/01-run.py
966516307607c5b75fa47b09beed7bc9deca06e9 /tmp/dwq.0.5632680420550714/c5bcaa217ff54478dfecb5ebe2842b82/build/tests_pkg_tflite-micro.bin
rm -rf /tmp/dwq.0.5632680420550714/c5bcaa217ff54478dfecb5ebe2842b82/build/pkg-build/flatbuffers
rm -rf /tmp/dwq.0.5632680420550714/c5bcaa217ff54478dfecb5ebe2842b82/build/pkg-build/gemmlowp
rm -rf /tmp/dwq.0.5632680420550714/c5bcaa217ff54478dfecb5ebe2842b82/build/pkg-build/ruy
rm -rf /tmp/dwq.0.5632680420550714/c5bcaa217ff54478dfecb5ebe2842b82/build/pkg-build/tflite-micro
rm -rf /tmp/dwq.0.5632680420550714/c5bcaa217ff54478dfecb5ebe2842b82/build/pkg-build/flatbuffers
rm -rf /tmp/dwq.0.5632680420550714/c5bcaa217ff54478dfecb5ebe2842b82/build/pkg-build/gemmlowp
rm -rf /tmp/dwq.0.5632680420550714/c5bcaa217ff54478dfecb5ebe2842b82/build/pkg-build/ruy
rm -rf /tmp/dwq.0.5632680420550714/c5bcaa217ff54478dfecb5ebe2842b82/build/pkg-build/tflite-micro
{"build/": 120}
esp bootloader
#18441
-- running on worker comsys thread 11, build number 497953.
make: Entering directory '/tmp/dwq.0.21095954614003543/d57f9454e88821ae64a29a37ebbbfc05/tests/xtimer_now64_continuity'
rm -rf /tmp/dwq.0.21095954614003543/d57f9454e88821ae64a29a37ebbbfc05/build/pkg-build/esp32_sdk
make: Leaving directory '/tmp/dwq.0.21095954614003543/d57f9454e88821ae64a29a37ebbbfc05/tests/xtimer_now64_continuity'
make: Entering directory '/tmp/dwq.0.21095954614003543/d57f9454e88821ae64a29a37ebbbfc05/tests/xtimer_now64_continuity'
Building application "tests_xtimer_now64_continuity" for "esp32-heltec-lora32-v2" with MCU "esp32".
git-cache: cloning from cache. tag=commit1329b19fe494500aeb79d19b27cfd99b40c37aec-1228229
esptool.py v3.2-dev
Merged 1 ELF section
text data bss dec hex filename
59127 10592 7382 77101 12d2d /tmp/dwq.0.21095954614003543/d57f9454e88821ae64a29a37ebbbfc05/build/tests_xtimer_now64_continuity.elf
esptool.py v3.2-dev
Merged 1 ELF section
Parsing CSV input...
sha1sum /tmp/dwq.0.21095954614003543/d57f9454e88821ae64a29a37ebbbfc05/tests/xtimer_now64_continuity/tests/01-run.py /tmp/dwq.0.21095954614003543/d57f9454e88821ae64a29a37ebbbfc05/build/tests_xtimer_now64_continuity.bin /tmp/dwq.0.21095954614003543/d57f9454e88821ae64a29a37ebbbfc05/build/tests_xtimer_now64_continuity.elf.bin /tmp/dwq.0.21095954614003543/d57f9454e88821ae64a29a37ebbbfc05/build/partitions.bin /tmp/dwq.0.21095954614003543/d57f9454e88821ae64a29a37ebbbfc05/build/esp_bootloader/bootloader.bin > /tmp/dwq.0.21095954614003543/d57f9454e88821ae64a29a37ebbbfc05/build/test-input-hash.sha1
make: Leaving directory '/tmp/dwq.0.21095954614003543/d57f9454e88821ae64a29a37ebbbfc05/tests/xtimer_now64_continuity'
make: Entering directory '/tmp/dwq.0.21095954614003543/d57f9454e88821ae64a29a37ebbbfc05/tests/xtimer_now64_continuity'
rm -rf /tmp/dwq.0.21095954614003543/d57f9454e88821ae64a29a37ebbbfc05/build/pkg-build/esp32_sdk
make: Leaving directory '/tmp/dwq.0.21095954614003543/d57f9454e88821ae64a29a37ebbbfc05/tests/xtimer_now64_continuity'
make: Entering directory '/tmp/dwq.0.21095954614003543/d57f9454e88821ae64a29a37ebbbfc05/tests/xtimer_now64_continuity'
Building application "tests_xtimer_now64_continuity" for "esp32-heltec-lora32-v2" with MCU "esp32".
git-cache: cloning from cache. tag=commit1329b19fe494500aeb79d19b27cfd99b40c37aec-1265027
esptool.py v3.2-dev
Merged 1 ELF section
text data bss dec hex filename
59127 10592 7382 77101 12d2d /tmp/dwq.0.21095954614003543/d57f9454e88821ae64a29a37ebbbfc05/build/tests_xtimer_now64_continuity.elf
esptool.py v3.2-dev
Merged 1 ELF section
Parsing CSV input...
sha1sum /tmp/dwq.0.21095954614003543/d57f9454e88821ae64a29a37ebbbfc05/tests/xtimer_now64_continuity/tests/01-run.py /tmp/dwq.0.21095954614003543/d57f9454e88821ae64a29a37ebbbfc05/build/tests_xtimer_now64_continuity.bin /tmp/dwq.0.21095954614003543/d57f9454e88821ae64a29a37ebbbfc05/build/tests_xtimer_now64_continuity.elf.bin /tmp/dwq.0.21095954614003543/d57f9454e88821ae64a29a37ebbbfc05/build/partitions.bin /tmp/dwq.0.21095954614003543/d57f9454e88821ae64a29a37ebbbfc05/build/esp_bootloader/bootloader.bin > /tmp/dwq.0.21095954614003543/d57f9454e88821ae64a29a37ebbbfc05/build/test-input-hash.sha1
make: Leaving directory '/tmp/dwq.0.21095954614003543/d57f9454e88821ae64a29a37ebbbfc05/tests/xtimer_now64_continuity'
Hashes of binaries with and without Kconfig mismatch for tests/xtimer_now64_continuity with esp32-heltec-lora32-v2
Please check that all used modules are modelled in Kconfig and enabled
Input without KConfig:
a6ba00771a57574a1b85ebf59d5e2e590ecbff49 /tmp/dwq.0.21095954614003543/d57f9454e88821ae64a29a37ebbbfc05/tests/xtimer_now64_continuity/tests/01-run.py
f780ae044f7a1dcb213c33c5e7e19106b2e41361 /tmp/dwq.0.21095954614003543/d57f9454e88821ae64a29a37ebbbfc05/build/tests_xtimer_now64_continuity.bin
2f3f588c6fea7a30cbee7fc86e414c2c6908117d /tmp/dwq.0.21095954614003543/d57f9454e88821ae64a29a37ebbbfc05/build/tests_xtimer_now64_continuity.elf.bin
ba8a1058e64c94d36e6747d51de715ea871bacf4 /tmp/dwq.0.21095954614003543/d57f9454e88821ae64a29a37ebbbfc05/build/partitions.bin
818e4f87242de3eeac54329f97b08a3d17f908e9 /tmp/dwq.0.21095954614003543/d57f9454e88821ae64a29a37ebbbfc05/build/esp_bootloader/bootloader.bin
Input with KConfig:
a6ba00771a57574a1b85ebf59d5e2e590ecbff49 /tmp/dwq.0.21095954614003543/d57f9454e88821ae64a29a37ebbbfc05/tests/xtimer_now64_continuity/tests/01-run.py
f780ae044f7a1dcb213c33c5e7e19106b2e41361 /tmp/dwq.0.21095954614003543/d57f9454e88821ae64a29a37ebbbfc05/build/tests_xtimer_now64_continuity.bin
2f3f588c6fea7a30cbee7fc86e414c2c6908117d /tmp/dwq.0.21095954614003543/d57f9454e88821ae64a29a37ebbbfc05/build/tests_xtimer_now64_continuity.elf.bin
ba8a1058e64c94d36e6747d51de715ea871bacf4 /tmp/dwq.0.21095954614003543/d57f9454e88821ae64a29a37ebbbfc05/build/partitions.bin
a74146b40ef7ceebf9ba653f030ae1ad0955a6e8 /tmp/dwq.0.21095954614003543/d57f9454e88821ae64a29a37ebbbfc05/build/esp_bootloader/bootloader.bin
rm -rf /tmp/dwq.0.21095954614003543/d57f9454e88821ae64a29a37ebbbfc05/build/pkg-build/esp32_sdk
rm -rf /tmp/dwq.0.21095954614003543/d57f9454e88821ae64a29a37ebbbfc05/build/pkg-build/esp32_sdk
{"build/": 128}
esp app ?(many mismatches binary + partitions)
#18415
-- running on worker comsys thread 13, build number 464729.
make: Entering directory '/tmp/dwq.0.21095954614003543/fc3b75e81db754106a9cb199e82d9453/tests/driver_at25xxx'
rm -rf /tmp/dwq.0.21095954614003543/fc3b75e81db754106a9cb199e82d9453/build/pkg-build/esp32_sdk
make: Leaving directory '/tmp/dwq.0.21095954614003543/fc3b75e81db754106a9cb199e82d9453/tests/driver_at25xxx'
make: Entering directory '/tmp/dwq.0.21095954614003543/fc3b75e81db754106a9cb199e82d9453/tests/driver_at25xxx'
Building application "tests_driver_at25xxx" for "esp32c3-devkit" with MCU "esp32".
git-cache: cloning from cache. tag=commit1329b19fe494500aeb79d19b27cfd99b40c37aec-1414005
esptool.py v3.2-dev
Merged 1 ELF section
text data bss dec hex filename
62942 73788 90448 227178 3776a /tmp/dwq.0.21095954614003543/fc3b75e81db754106a9cb199e82d9453/build/tests_driver_at25xxx.elf
esptool.py v3.2-dev
Parsing CSV input...
sha1sum /tmp/dwq.0.21095954614003543/fc3b75e81db754106a9cb199e82d9453/build/tests_driver_at25xxx.bin /tmp/dwq.0.21095954614003543/fc3b75e81db754106a9cb199e82d9453/build/tests_driver_at25xxx.elf.bin /tmp/dwq.0.21095954614003543/fc3b75e81db754106a9cb199e82d9453/build/partitions.bin /tmp/dwq.0.21095954614003543/fc3b75e81db754106a9cb199e82d9453/build/esp_bootloader/bootloader.bin > /tmp/dwq.0.21095954614003543/fc3b75e81db754106a9cb199e82d9453/build/test-input-hash.sha1
make: Leaving directory '/tmp/dwq.0.21095954614003543/fc3b75e81db754106a9cb199e82d9453/tests/driver_at25xxx'
make: Entering directory '/tmp/dwq.0.21095954614003543/fc3b75e81db754106a9cb199e82d9453/tests/driver_at25xxx'
rm -rf /tmp/dwq.0.21095954614003543/fc3b75e81db754106a9cb199e82d9453/build/pkg-build/esp32_sdk
make: Leaving directory '/tmp/dwq.0.21095954614003543/fc3b75e81db754106a9cb199e82d9453/tests/driver_at25xxx'
make: Entering directory '/tmp/dwq.0.21095954614003543/fc3b75e81db754106a9cb199e82d9453/tests/driver_at25xxx'
Building application "tests_driver_at25xxx" for "esp32c3-devkit" with MCU "esp32".
git-cache: cloning from cache. tag=commit1329b19fe494500aeb79d19b27cfd99b40c37aec-1450984
esptool.py v3.2-dev
Merged 1 ELF section
text data bss dec hex filename
62942 73836 90448 227226 3779a /tmp/dwq.0.21095954614003543/fc3b75e81db754106a9cb199e82d9453/build/tests_driver_at25xxx.elf
esptool.py v3.2-dev
Parsing CSV input...
sha1sum /tmp/dwq.0.21095954614003543/fc3b75e81db754106a9cb199e82d9453/build/tests_driver_at25xxx.bin /tmp/dwq.0.21095954614003543/fc3b75e81db754106a9cb199e82d9453/build/tests_driver_at25xxx.elf.bin /tmp/dwq.0.21095954614003543/fc3b75e81db754106a9cb199e82d9453/build/partitions.bin /tmp/dwq.0.21095954614003543/fc3b75e81db754106a9cb199e82d9453/build/esp_bootloader/bootloader.bin > /tmp/dwq.0.21095954614003543/fc3b75e81db754106a9cb199e82d9453/build/test-input-hash.sha1
make: Leaving directory '/tmp/dwq.0.21095954614003543/fc3b75e81db754106a9cb199e82d9453/tests/driver_at25xxx'
Hashes of binaries with and without Kconfig mismatch for tests/driver_at25xxx with esp32c3-devkit
Please check that all used modules are modelled in Kconfig and enabled
Input without KConfig:
607a89fa5b2e1cfa8a84c9d15ec92adba956e9f1 /tmp/dwq.0.21095954614003543/fc3b75e81db754106a9cb199e82d9453/build/tests_driver_at25xxx.bin
d85cad23e4ac65e5907ab3cedec3c4e6819a7bcd /tmp/dwq.0.21095954614003543/fc3b75e81db754106a9cb199e82d9453/build/tests_driver_at25xxx.elf.bin
1ef71a9ecedd328d8c43f0a62ab84bc4dce03f35 /tmp/dwq.0.21095954614003543/fc3b75e81db754106a9cb199e82d9453/build/partitions.bin
19ee9b3cf915cddb2c37d6774f1c5b62d1fbabb5 /tmp/dwq.0.21095954614003543/fc3b75e81db754106a9cb199e82d9453/build/esp_bootloader/bootloader.bin
Input with KConfig:
9206817e62236a462ca3410fab94a3b39131b7af /tmp/dwq.0.21095954614003543/fc3b75e81db754106a9cb199e82d9453/build/tests_driver_at25xxx.bin
bdbf88a86a7f61b7d33459fe82c9d339287e25ae /tmp/dwq.0.21095954614003543/fc3b75e81db754106a9cb199e82d9453/build/tests_driver_at25xxx.elf.bin
4a9349573c400e4d4790ae251c5a6639c3b126c0 /tmp/dwq.0.21095954614003543/fc3b75e81db754106a9cb199e82d9453/build/partitions.bin
19ee9b3cf915cddb2c37d6774f1c5b62d1fbabb5 /tmp/dwq.0.21095954614003543/fc3b75e81db754106a9cb199e82d9453/build/esp_bootloader/bootloader.bin
rm -rf /tmp/dwq.0.21095954614003543/fc3b75e81db754106a9cb199e82d9453/build/pkg-build/esp32_sdk
rm -rf /tmp/dwq.0.21095954614003543/fc3b75e81db754106a9cb199e82d9453/build/pkg-build/esp32_sdk
{"build/": 124}
Status update, we still see mismatches in the nightlies (cryptoauth package tests).
This PR has been rebased and we will see if we can make it reproducible.
Ugg, this is still an issue but probably does not make sense to work on it during hacknack.
Hmmm I give up on this. It seems like all the changes are somehow working through it.