llvm icon indicating copy to clipboard operation
llvm copied to clipboard

[CI] Revert w/a for FPGA emu bug

Open alexbatashev opened this issue 3 years ago • 11 comments

Containers were updated with fixed FPGA emulator packages.

alexbatashev avatar Mar 10 '22 08:03 alexbatashev

Timed Out : 8 Failed : 1

Do you know what might cause this issue?

bader avatar Mar 10 '22 09:03 bader

One failed test is a known issue, 8 timed out tests require investigation

alexbatashev avatar Mar 10 '22 09:03 alexbatashev

One failed test is a known issue

Is it related to this change? If no, please, disable the failing test to avoid false alarms.

8 timed out tests require investigation

The same - please, check that these are not related to your patch.

bader avatar Mar 10 '22 10:03 bader

Is it related to this change? If no, please, disable the failing test to avoid false alarms.

See https://github.com/intel/llvm-test-suite/pull/907

The same - please, check that these are not related to your patch

Looking into it

alexbatashev avatar Mar 10 '22 10:03 alexbatashev

FYI. I see something related in pre-commit for https://github.com/intel/llvm/pull/5778.

See https://github.com/intel/llvm/runs/5496313460?check_suite_focus=true.


Timed Out Tests (7): SYCL :: KernelAndProgram/multiple-kernel-linking.cpp SYCL :: Plugin/level_zero_batch_test.cpp SYCL :: Plugin/level_zero_batch_test_copy_with_compute.cpp SYCL :: Plugin/retain_events.cpp SYCL :: Printf/char.cpp SYCL :: Printf/float.cpp SYCL :: USM/copy.cpp

Testing Time: 1654.86s Unsupported : 184 Passed : 578 Expectedly Failed: 38 Timed Out : 7

bader avatar Mar 10 '22 14:03 bader

Another one - https://github.com/intel/llvm/runs/5496822497?check_suite_focus=true. It looks like a regression introduced before your patch.

bader avatar Mar 10 '22 14:03 bader

@againull, we reviewed recent changes we made to the sycl branch and identified https://github.com/intel/llvm/pull/5231 as one of the changes that might cause the timeout issues discussed above. Could you check if #5231 is related to them, please?

bader avatar Mar 11 '22 11:03 bader

Another instance: https://github.com/intel/llvm/runs/5503076361?check_suite_focus=true. Pre-commit for https://github.com/intel/llvm/pull/5772.

bader avatar Mar 11 '22 13:03 bader

A different picture here: https://github.com/intel/llvm/runs/5504737044?check_suite_focus=true

600.02s: SYCL :: ESIMD/kmeans/kmeans.cpp
595.17s: SYCL :: ESIMD/matrix_transpose_glb.cpp
592.98s: SYCL :: ESIMD/matrix_transpose_usm.cpp
591.00s: SYCL :: ESIMD/noinline_args_size/noinline_args_char_int_size_192.cpp
590.90s: SYCL :: ESIMD/noinline_args_size/noinline_args_char_int_size_512.cpp
590.84s: SYCL :: ESIMD/noinline_args_size/noinline_args_char_int_size_256.cpp
589.63s: SYCL :: ESIMD/noinline_args_size/noinline_args_char_int_size_64.cpp
589.43s: SYCL :: ESIMD/noinline_args_size/noinline_args_char_int_size_96.cpp
588.61s: SYCL :: ESIMD/noinline_args_size/noinline_args_int_size_192.cpp
588.12s: SYCL :: ESIMD/noinline_args_size/noinline_args_int_size_512.cpp
587.82s: SYCL :: ESIMD/noinline_args_size/noinline_args_int_size_256.cpp
586.85s: SYCL :: ESIMD/noinline_args_size/noinline_args_int_size_64.cpp

bader avatar Mar 11 '22 18:03 bader

@againull, we reviewed recent changes we made to the sycl branch and identified #5231 as one of the changes that might cause the timeout issues discussed above. Could you check if #5231 is related to them, please?

@bader I am trying to reproduce timeout failures using intel/llvm by running whole llvm-test-suite multiple times with -j 12, but no luck so far. Actually I was doing this during the last week to test this patch and didn't face timeout issues.

I don't have information about how many testing jobs we run on the same machine, how many threads do we use (-j threads]) and if smth changed recently in the why we run testing jobs in CI. Also I need an access to one of the CI machines where problem is reproduced. Who can provide me this information?

againull avatar Mar 11 '22 23:03 againull

@againull, we reviewed recent changes we made to the sycl branch and identified #5231 as one of the changes that might cause the timeout issues discussed above. Could you check if #5231 is related to them, please?

@bader I've created a PR https://github.com/intel/llvm/pull/5794 which reverts #5231. 5 tests failed wih timeout: https://github.com/intel/llvm/runs/5518337723?check_suite_focus=true

I hope this proves that #5231 is not a guilty commit for timeout failures.

againull avatar Mar 12 '22 00:03 againull