vowpal_wabbit icon indicating copy to clipboard operation
vowpal_wabbit copied to clipboard

Published 20 hours ago • verified publisher icon VowpalWabbit

The MSVC team recently test VowpalWabbit project failed to run "vw_unit_test" test on release and debug configuration on Windows_x86 and windows_x64.

Open Vincent9802 opened this issue 2 years ago • 4 comments

Describe the bug

Hi All,

VowpalWabbit failed to run run "vw_unit_test" test on release and debug configuration with MSVC on windows x86 and x64. Could you please help look at this issue or provide some workarounds? Thanks in advance.

Error message: Test project F:/gitP/VowpalWabbit/vowpal_wabbit/build_x86 Start 1: vw_c_wrapper_test 1/6 Test #1: vw_c_wrapper_test ................ Passed 0.06 sec Start 2: vw_config_test 2/6 Test #2: vw_config_test ................... Passed 0.03 sec Start 3: vw_core_test 3/6 Test #3: vw_core_test ..................... Passed 0.09 sec Start 4: vw_explore_test 4/6 Test #4: vw_explore_test .................. Passed 0.04 sec Start 5: vw_io_test 5/6 Test #5: vw_io_test ....................... Passed 0.02 sec Start 6: vw_unit_test 6/6 Test #6: vw_unit_test .....................***Failed 4.69 sec Running 271 test cases... F:/gitP/VowpalWabbit/vowpal_wabbit/test/unit_test/cb_large_actions_test.cc(793): error: in "check_final_truncated_SVD_validity": difference{1.02984e-06} between S(i){0.347264767} and action_space->explore._S(i){0.347265124} exceeds 0.0001% F:/gitP/VowpalWabbit/vowpal_wabbit/test/unit_test/cb_large_actions_test.cc(793): error: in "check_final_truncated_SVD_validity": difference{1.52138e-06} between S(i){0.215479165} and action_space->explore._S(i){0.215478837} exceeds 0.0001% F:/gitP/VowpalWabbit/vowpal_wabbit/test/unit_test/cb_large_actions_test.cc(982): error: in "check_spanner_results_squarecb": difference{0.000107196} between preds[0].score{0.697345734} and 0.697270989{0.69727098899999995} exceeds 0.0001% F:/gitP/VowpalWabbit/vowpal_wabbit/test/unit_test/cb_large_actions_test.cc(985): error: in "check_spanner_results_squarecb": difference{0.000247158} between preds[1].score{0.302654237} and 0.30272904{0.30272904} exceeds 0.0001% F:/gitP/VowpalWabbit/vowpal_wabbit/test/unit_test/cb_large_actions_test.cc(982): error: in "check_spanner_results_squarecb": difference{0.000107196} between preds[0].score{0.697345734} and 0.697270989{0.69727098899999995} exceeds 0.0001% F:/gitP/VowpalWabbit/vowpal_wabbit/test/unit_test/cb_large_actions_test.cc(985): error: in "check_spanner_results_squarecb": difference{0.000247158} between preds[1].score{0.302654237} and 0.30272904{0.30272904} exceeds 0.0001%

*** 6 failures are detected in the test module "Main"

Expected behavior For Vowpal_wabbit x86 and x64, "vw_unit_test" test should be run pass on the windows.

How to reproduce

  1. git clone (https://github.com/VowpalWabbit/vowpal_wabbit) F:\gitP\VowpalWabbit
  2. git clone https://github.com/Microsoft/vcpkg F:\gitP\Microsoft
  3. set VSCMD_SKIP_SENDTELEMETRY=1 & "C:\Program Files (x86)\Microsoft Visual Studio\2019\Enterprise\Common7\Tools\VsDevCmd.bat" -host_arch=amd64 -arch=amd64
  4. cd F:\gitP\Microsoft\vcpkg
  5. bootstrap-vcpkg.bat 2>&1
  6. vcpkg.exe install --recurse zlib boost-system boost-program-options boost-test boost-align boost-foreach boost-math boost-python python3 flatbuffers --triplet x64-windows --clean-after-build 2>&1
  7. mkdir&cd F:\gitP\VowpalWabbit\vowpal_wabbit\build_amd64
  8. cmake -G "Visual Studio 16 2019" -A x64 -DCMAKE_SYSTEM_VERSION=10.0.18362.0 -DCMAKE_BUILD_TYPE=Release -DUSE_LATEST_STD=ON -Dvw_BUILD_NET_FRAMEWORK=OFF -DBUILD_FLATBUFFERS=OFF -DCMAKE_TOOLCHAIN_FILE=F:\gitP\Microsoft\vcpkg\scripts\buildsystems\vcpkg.cmake -DVCPKG_TARGET_TRIPLET=x64-windows .. 2>&1
  9. msbuild /m /p:Platform=x64 /p:Configuration=Release vowpal_wabbit.sln /t:Rebuild /p:BuildInParallel=true 2>&1
  10. ctest --build-config Release --output-on-failure 2>&1

Version

git commit : a38aa0e

OS

Windows

Language

C++

Additional context

Setup VS 2019 + Windows Server 2019 test.log test.log.2.txt

Vincent9802 avatar Jun 16 '22 09:06 Vincent9802

I can't get it to repro locally but just merged a PR into master that will potentially resolve the issue

olgavrou avatar Jun 20 '22 14:06 olgavrou

Hi @olgavrou, I used the newest commit to verify it, found that test still fail, but error message updated. Could you please help look at this issue again? Thank you in advance ~

Error message: 7/7 Test #7: vw_unit_test .....................***Failed 3.56 sec Running 273 test cases... F:/gitP/VowpalWabbit/vowpal_wabbit/test/unit_test/automl_test.cc(78): error: in "clear_configs_one_diff": check score.update_count == 0 has failed [5 != 0] F:/gitP/VowpalWabbit/vowpal_wabbit/test/unit_test/automl_test.cc(80): error: in "clear_configs_one_diff": check score.upper_bound() == 0.f has failed [1.15384626 != 0] F:/gitP/VowpalWabbit/vowpal_wabbit/test/unit_test/automl_test.cc(78): error: in "clear_configs_one_diff": check score.update_count == 0 has failed [5 != 0] F:/gitP/VowpalWabbit/vowpal_wabbit/test/unit_test/automl_test.cc(80): error: in "clear_configs_one_diff": check score.upper_bound() == 0.f has failed [1.15384626 != 0] F:/gitP/VowpalWabbit/vowpal_wabbit/test/unit_test/automl_test.cc(78): error: in "clear_configs_one_diff": check score.update_count == 0 has failed [5 != 0] F:/gitP/VowpalWabbit/vowpal_wabbit/test/unit_test/automl_test.cc(80): error: in "clear_configs_one_diff": check score.upper_bound() == 0.f has failed [1.15384626 != 0] F:/gitP/VowpalWabbit/vowpal_wabbit/test/unit_test/automl_test.cc(84): error: in "clear_configs_one_diff": check score.update_count == 0 has failed [5 != 0] F:/gitP/VowpalWabbit/vowpal_wabbit/test/unit_test/automl_test.cc(86): error: in "clear_configs_one_diff": check score.upper_bound() == 0.f has failed [1.15384626 != 0] F:/gitP/VowpalWabbit/vowpal_wabbit/test/unit_test/automl_test.cc(84): error: in "clear_configs_one_diff": check score.update_count == 0 has failed [5 != 0] F:/gitP/VowpalWabbit/vowpal_wabbit/test/unit_test/automl_test.cc(86): error: in "clear_configs_one_diff": check score.upper_bound() == 0.f has failed [1.15384626 != 0] F:/gitP/VowpalWabbit/vowpal_wabbit/test/unit_test/automl_test.cc(84): error: in "clear_configs_one_diff": check score.update_count == 0 has failed [5 != 0] F:/gitP/VowpalWabbit/vowpal_wabbit/test/unit_test/automl_test.cc(86): error: in "clear_configs_one_diff": check score.upper_bound() == 0.f has failed [1.15384626 != 0] F:/gitP/VowpalWabbit/vowpal_wabbit/test/unit_test/automl_test.cc(400): error: in "clear_configs_one_diff": check aml->cm->scores.size() == 1 has failed [4 != 1]

*** 13 failures are detected in the test module "Main"

86% tests passed, 1 tests failed out of 7

Label Time Summary: VWTestList = 4.14 sec*proc (7 tests)

Total Test time (real) = 4.29 sec

The following tests FAILED: 7 - vw_unit_test (Failed) Errors while running CTest

If you can't repro this issue, please use this new repro steps:

  1. git clone https://github.com/Microsoft/vcpkg F:\gitP\Microsoft

  2. git -C "F:\gitP\Microsoft\vcpkg" reset --hard 20907dc

  3. git -C "F:\gitP\Microsoft\vcpkg" remote prune origin

  4. git clone (https://github.com/VowpalWabbit/vowpal_wabbit) F:\gitP\VowpalWabbit

  5. git -C "F:\gitP\VowpalWabbit\vowpal_wabbit" reset --hard eecdacc

  6. git -C "F:\gitP\VowpalWabbit\vowpal_wabbit" remote prune origin

  7. git -C "F:\gitP\VowpalWabbit\vowpal_wabbit" submodule sync

  8. git -C "F:\gitP\VowpalWabbit\vowpal_wabbit" submodule foreach git reset --hard

  9. git -C "F:\gitP\VowpalWabbit\vowpal_wabbit" submodule foreach git clean -xdf

  10. git -C "F:\gitP\VowpalWabbit\vowpal_wabbit" submodule update --init --recursive

  11. set VSCMD_SKIP_SENDTELEMETRY=1 & "C:\Program Files (x86)\Microsoft Visual Studio\2019\Enterprise\Common7\Tools\VsDevCmd.bat" -host_arch=amd64 -arch=amd64

  12. cd F:\gitP\Microsoft\vcpkg

  13. bootstrap-vcpkg.bat 2>&1

  14. vcpkg.exe install --recurse zlib boost-system boost-program-options boost-test boost-align boost-foreach boost-math boost-python python3 flatbuffers --triplet x64-windows --clean-after-build 2>&1

  15. mkdir F:\gitP\VowpalWabbit\vowpal_wabbit\build_amd64

  16. cd F:\gitP\VowpalWabbit\vowpal_wabbit\build_amd64

  17. cmake -G "Visual Studio 16 2019" -A x64 -DCMAKE_SYSTEM_VERSION=10.0.18362.0 -DCMAKE_BUILD_TYPE=Release -DUSE_LATEST_STD=ON -Dvw_BUILD_NET_FRAMEWORK=OFF -DBUILD_FLATBUFFERS=OFF -DCMAKE_TOOLCHAIN_FILE=F:\gitP\Microsoft\vcpkg\scripts\buildsystems\vcpkg.cmake -DVCPKG_TARGET_TRIPLET=x64-windows .. 2>&1

  18. msbuild /m /p:Platform=x64 /p:Configuration=Release vowpal_wabbit.sln /t:Rebuild /p:BuildInParallel=true 2>&1

  19. ctest --build-config Release --output-on-failure 2>&1

rachelhe123 avatar Jun 21 '22 06:06 rachelhe123

Hi @rachelhe123,

It looks like the issue has to do with decimal precision since the automl tests are very sensitive to small variations. Working on a PR to make these tests less sensitive

bassmang avatar Jun 21 '22 14:06 bassmang

Seems to be some mismatch between float operations in different environments. This test is passing on our windows build CI. See https://github.com/VowpalWabbit/vowpal_wabbit/runs/6968381651?check_suite_focus=true

Is this being tested on some new MSVC version? This is the CI one:

Run cmake --build "D:\a\vowpal_wabbit\vowpal_wabbit/vw/build" --config Release --verbose
Microsoft (R) Build Engine version 16.11.2+f32259642 for .NET Framework
Copyright (C) Microsoft Corporation. All rights reserved.

lalo avatar Jun 21 '22 14:06 lalo

Hi @rachelhe123,

The tests that were causing this issue have been replaced/updated. Could you let us know if you're still having the same issues?

bassmang avatar Sep 26 '22 15:09 bassmang

Please re-open if you are still having issues.

lalo avatar Dec 08 '22 20:12 lalo