OpenBLAS
OpenBLAS copied to clipboard
ThreadSanitizer detects warnings in cblas_dgemm
#include <cblas.h>
#include <stdio.h>
int main()
{
int i=0;
double A[6] = {1.0,2.0,1.0,-3.0,4.0,-1.0};
double B[6] = {1.0,2.0,1.0,-3.0,4.0,-1.0};
double C[9] = {.5,.5,.5,.5,.5,.5,.5,.5,.5};
cblas_dgemm(CblasColMajor, CblasNoTrans, CblasTrans,3,3,2,1,A, 3, B, 3,2,C,3);
for(i=0; i<9; i++)
printf("%lf ", C[i]);
printf("\n");
}
results in
Output of running example compiler with tsan
==================
WARNING: ThreadSanitizer: data race (pid=13857)
Write of size 4 at 0x55ad043e2048 by thread T6:
#0 alloc_mmap /root/.conan/data/openblas/0.3.20/aborzunov/testing/build/af70c39ddad7b91ea16b95b5cceba7ed41791f1e/source_subfolder/driver/others/memory.c:2301 (test_package+0xe08e)
#1 blas_memory_alloc /root/.conan/data/openblas/0.3.20/aborzunov/testing/build/af70c39ddad7b91ea16b95b5cceba7ed41791f1e/source_subfolder/driver/others/memory.c:2925 (test_package+0xe807)
#2 blas_thread_server /root/.conan/data/openblas/0.3.20/aborzunov/testing/build/af70c39ddad7b91ea16b95b5cceba7ed41791f1e/source_subfolder/driver/others/blas_server.c:382 (test_package+0xf27a)
Previous read of size 4 at 0x55ad043e2048 by thread T9:
#0 alloc_mmap /root/.conan/data/openblas/0.3.20/aborzunov/testing/build/af70c39ddad7b91ea16b95b5cceba7ed41791f1e/source_subfolder/driver/others/memory.c:2225 (test_package+0xdeb0)
#1 blas_memory_alloc /root/.conan/data/openblas/0.3.20/aborzunov/testing/build/af70c39ddad7b91ea16b95b5cceba7ed41791f1e/source_subfolder/driver/others/memory.c:2925 (test_package+0xe807)
#2 blas_thread_server /root/.conan/data/openblas/0.3.20/aborzunov/testing/build/af70c39ddad7b91ea16b95b5cceba7ed41791f1e/source_subfolder/driver/others/blas_server.c:382 (test_package+0xf27a)
Location is global 'hot_alloc' of size 4 at 0x55ad043e2048 (test_package+0x000000024048)
Thread T6 (tid=13864, running) created by main thread at:
#0 pthread_create ../../../../src/libsanitizer/tsan/tsan_interceptors_posix.cpp:969 (libtsan.so.0+0x605b8)
#1 blas_thread_init /root/.conan/data/openblas/0.3.20/aborzunov/testing/build/af70c39ddad7b91ea16b95b5cceba7ed41791f1e/source_subfolder/driver/others/blas_server.c:667 (test_package+0xf771)
#2 blas_thread_init /root/.conan/data/openblas/0.3.20/aborzunov/testing/build/af70c39ddad7b91ea16b95b5cceba7ed41791f1e/source_subfolder/driver/others/blas_server.c:628 (test_package+0xf771)
#3 gotoblas_memory_init /root/.conan/data/openblas/0.3.20/aborzunov/testing/build/af70c39ddad7b91ea16b95b5cceba7ed41791f1e/source_subfolder/driver/others/memory.c:3314 (test_package+0x3e62)
#4 gotoblas_init /root/.conan/data/openblas/0.3.20/aborzunov/testing/build/af70c39ddad7b91ea16b95b5cceba7ed41791f1e/source_subfolder/driver/others/memory.c:3357 (test_package+0x3e62)
#5 __libc_start_main <null> (libc.so.6+0x29eba)
Thread T9 (tid=13867, running) created by main thread at:
#0 pthread_create ../../../../src/libsanitizer/tsan/tsan_interceptors_posix.cpp:969 (libtsan.so.0+0x605b8)
#1 blas_thread_init /root/.conan/data/openblas/0.3.20/aborzunov/testing/build/af70c39ddad7b91ea16b95b5cceba7ed41791f1e/source_subfolder/driver/others/blas_server.c:667 (test_package+0xf771)
#2 blas_thread_init /root/.conan/data/openblas/0.3.20/aborzunov/testing/build/af70c39ddad7b91ea16b95b5cceba7ed41791f1e/source_subfolder/driver/others/blas_server.c:628 (test_package+0xf771)
#3 gotoblas_memory_init /root/.conan/data/openblas/0.3.20/aborzunov/testing/build/af70c39ddad7b91ea16b95b5cceba7ed41791f1e/source_subfolder/driver/others/memory.c:3314 (test_package+0x3e62)
#4 gotoblas_init /root/.conan/data/openblas/0.3.20/aborzunov/testing/build/af70c39ddad7b91ea16b95b5cceba7ed41791f1e/source_subfolder/driver/others/memory.c:3357 (test_package+0x3e62)
#5 __libc_start_main <null> (libc.so.6+0x29eba)
SUMMARY: ThreadSanitizer: data race /root/.conan/data/openblas/0.3.20/aborzunov/testing/build/af70c39ddad7b91ea16b95b5cceba7ed41791f1e/source_subfolder/driver/others/memory.c:2301 in alloc_mmap
==================
==================
WARNING: ThreadSanitizer: data race (pid=13857)
Read of size 8 at 0x7ffc55e63140 by thread T6:
#0 blas_thread_server /root/.conan/data/openblas/0.3.20/aborzunov/testing/build/af70c39ddad7b91ea16b95b5cceba7ed41791f1e/source_subfolder/driver/others/blas_server.c:444 (test_package+0xf34a)
Previous write of size 8 at 0x7ffc55e63140 by main thread:
[failed to restore the stack]
Location is stack of main thread.
Location is global '<null>' at 0x000000000000 ([stack]+0x00000001e140)
Thread T6 (tid=13864, running) created by main thread at:
#0 pthread_create ../../../../src/libsanitizer/tsan/tsan_interceptors_posix.cpp:969 (libtsan.so.0+0x605b8)
#1 blas_thread_init /root/.conan/data/openblas/0.3.20/aborzunov/testing/build/af70c39ddad7b91ea16b95b5cceba7ed41791f1e/source_subfolder/driver/others/blas_server.c:667 (test_package+0xf771)
#2 blas_thread_init /root/.conan/data/openblas/0.3.20/aborzunov/testing/build/af70c39ddad7b91ea16b95b5cceba7ed41791f1e/source_subfolder/driver/others/blas_server.c:628 (test_package+0xf771)
#3 gotoblas_memory_init /root/.conan/data/openblas/0.3.20/aborzunov/testing/build/af70c39ddad7b91ea16b95b5cceba7ed41791f1e/source_subfolder/driver/others/memory.c:3314 (test_package+0x3e62)
#4 gotoblas_init /root/.conan/data/openblas/0.3.20/aborzunov/testing/build/af70c39ddad7b91ea16b95b5cceba7ed41791f1e/source_subfolder/driver/others/memory.c:3357 (test_package+0x3e62)
#5 __libc_start_main <null> (libc.so.6+0x29eba)
SUMMARY: ThreadSanitizer: data race /root/.conan/data/openblas/0.3.20/aborzunov/testing/build/af70c39ddad7b91ea16b95b5cceba7ed41791f1e/source_subfolder/driver/others/blas_server.c:444 in blas_thread_server
==================
==================
WARNING: ThreadSanitizer: data race (pid=13857)
Read of size 8 at 0x7ffc55e63170 by thread T6:
#0 blas_thread_server /root/.conan/data/openblas/0.3.20/aborzunov/testing/build/af70c39ddad7b91ea16b95b5cceba7ed41791f1e/source_subfolder/driver/others/blas_server.c:448 (test_package+0xf369)
Previous write of size 8 at 0x7ffc55e63170 by main thread:
[failed to restore the stack]
Location is stack of main thread.
Location is global '<null>' at 0x000000000000 ([stack]+0x00000001e170)
Thread T6 (tid=13864, running) created by main thread at:
#0 pthread_create ../../../../src/libsanitizer/tsan/tsan_interceptors_posix.cpp:969 (libtsan.so.0+0x605b8)
#1 blas_thread_init /root/.conan/data/openblas/0.3.20/aborzunov/testing/build/af70c39ddad7b91ea16b95b5cceba7ed41791f1e/source_subfolder/driver/others/blas_server.c:667 (test_package+0xf771)
#2 blas_thread_init /root/.conan/data/openblas/0.3.20/aborzunov/testing/build/af70c39ddad7b91ea16b95b5cceba7ed41791f1e/source_subfolder/driver/others/blas_server.c:628 (test_package+0xf771)
#3 gotoblas_memory_init /root/.conan/data/openblas/0.3.20/aborzunov/testing/build/af70c39ddad7b91ea16b95b5cceba7ed41791f1e/source_subfolder/driver/others/memory.c:3314 (test_package+0x3e62)
#4 gotoblas_init /root/.conan/data/openblas/0.3.20/aborzunov/testing/build/af70c39ddad7b91ea16b95b5cceba7ed41791f1e/source_subfolder/driver/others/memory.c:3357 (test_package+0x3e62)
#5 __libc_start_main <null> (libc.so.6+0x29eba)
SUMMARY: ThreadSanitizer: data race /root/.conan/data/openblas/0.3.20/aborzunov/testing/build/af70c39ddad7b91ea16b95b5cceba7ed41791f1e/source_subfolder/driver/others/blas_server.c:448 in blas_thread_server
==================
==================
WARNING: ThreadSanitizer: data race (pid=13857)
Read of size 8 at 0x7ffc55e63178 by thread T6:
#0 blas_thread_server /root/.conan/data/openblas/0.3.20/aborzunov/testing/build/af70c39ddad7b91ea16b95b5cceba7ed41791f1e/source_subfolder/driver/others/blas_server.c:449 (test_package+0xf37d)
Previous write of size 8 at 0x7ffc55e63178 by main thread:
[failed to restore the stack]
Location is stack of main thread.
Location is global '<null>' at 0x000000000000 ([stack]+0x00000001e178)
Thread T6 (tid=13864, running) created by main thread at:
#0 pthread_create ../../../../src/libsanitizer/tsan/tsan_interceptors_posix.cpp:969 (libtsan.so.0+0x605b8)
#1 blas_thread_init /root/.conan/data/openblas/0.3.20/aborzunov/testing/build/af70c39ddad7b91ea16b95b5cceba7ed41791f1e/source_subfolder/driver/others/blas_server.c:667 (test_package+0xf771)
#2 blas_thread_init /root/.conan/data/openblas/0.3.20/aborzunov/testing/build/af70c39ddad7b91ea16b95b5cceba7ed41791f1e/source_subfolder/driver/others/blas_server.c:628 (test_package+0xf771)
#3 gotoblas_memory_init /root/.conan/data/openblas/0.3.20/aborzunov/testing/build/af70c39ddad7b91ea16b95b5cceba7ed41791f1e/source_subfolder/driver/others/memory.c:3314 (test_package+0x3e62)
#4 gotoblas_init /root/.conan/data/openblas/0.3.20/aborzunov/testing/build/af70c39ddad7b91ea16b95b5cceba7ed41791f1e/source_subfolder/driver/others/memory.c:3357 (test_package+0x3e62)
#5 __libc_start_main <null> (libc.so.6+0x29eba)
SUMMARY: ThreadSanitizer: data race /root/.conan/data/openblas/0.3.20/aborzunov/testing/build/af70c39ddad7b91ea16b95b5cceba7ed41791f1e/source_subfolder/driver/others/blas_server.c:449 in blas_thread_server
==================
==================
WARNING: ThreadSanitizer: data race (pid=13857)
Read of size 4 at 0x7ffc55e631e0 by thread T6:
#0 blas_thread_server /root/.conan/data/openblas/0.3.20/aborzunov/testing/build/af70c39ddad7b91ea16b95b5cceba7ed41791f1e/source_subfolder/driver/others/blas_server.c:470 (test_package+0xf395)
Previous write of size 4 at 0x7ffc55e631e0 by main thread:
[failed to restore the stack]
Location is stack of main thread.
Location is global '<null>' at 0x000000000000 ([stack]+0x00000001e1e0)
Thread T6 (tid=13864, running) created by main thread at:
#0 pthread_create ../../../../src/libsanitizer/tsan/tsan_interceptors_posix.cpp:969 (libtsan.so.0+0x605b8)
#1 blas_thread_init /root/.conan/data/openblas/0.3.20/aborzunov/testing/build/af70c39ddad7b91ea16b95b5cceba7ed41791f1e/source_subfolder/driver/others/blas_server.c:667 (test_package+0xf771)
#2 blas_thread_init /root/.conan/data/openblas/0.3.20/aborzunov/testing/build/af70c39ddad7b91ea16b95b5cceba7ed41791f1e/source_subfolder/driver/others/blas_server.c:628 (test_package+0xf771)
#3 gotoblas_memory_init /root/.conan/data/openblas/0.3.20/aborzunov/testing/build/af70c39ddad7b91ea16b95b5cceba7ed41791f1e/source_subfolder/driver/others/memory.c:3314 (test_package+0x3e62)
#4 gotoblas_init /root/.conan/data/openblas/0.3.20/aborzunov/testing/build/af70c39ddad7b91ea16b95b5cceba7ed41791f1e/source_subfolder/driver/others/memory.c:3357 (test_package+0x3e62)
#5 __libc_start_main <null> (libc.so.6+0x29eba)
SUMMARY: ThreadSanitizer: data race /root/.conan/data/openblas/0.3.20/aborzunov/testing/build/af70c39ddad7b91ea16b95b5cceba7ed41791f1e/source_subfolder/driver/others/blas_server.c:470 in blas_thread_server
==================
==================
WARNING: ThreadSanitizer: data race (pid=13857)
Read of size 8 at 0x7ffc55e63158 by thread T6:
#0 blas_thread_server /root/.conan/data/openblas/0.3.20/aborzunov/testing/build/af70c39ddad7b91ea16b95b5cceba7ed41791f1e/source_subfolder/driver/others/blas_server.c:519 (test_package+0xf3b4)
Previous write of size 8 at 0x7ffc55e63158 by main thread:
[failed to restore the stack]
Location is stack of main thread.
Location is global '<null>' at 0x000000000000 ([stack]+0x00000001e158)
Thread T6 (tid=13864, running) created by main thread at:
#0 pthread_create ../../../../src/libsanitizer/tsan/tsan_interceptors_posix.cpp:969 (libtsan.so.0+0x605b8)
#1 blas_thread_init /root/.conan/data/openblas/0.3.20/aborzunov/testing/build/af70c39ddad7b91ea16b95b5cceba7ed41791f1e/source_subfolder/driver/others/blas_server.c:667 (test_package+0xf771)
#2 blas_thread_init /root/.conan/data/openblas/0.3.20/aborzunov/testing/build/af70c39ddad7b91ea16b95b5cceba7ed41791f1e/source_subfolder/driver/others/blas_server.c:628 (test_package+0xf771)
#3 gotoblas_memory_init /root/.conan/data/openblas/0.3.20/aborzunov/testing/build/af70c39ddad7b91ea16b95b5cceba7ed41791f1e/source_subfolder/driver/others/memory.c:3314 (test_package+0x3e62)
#4 gotoblas_init /root/.conan/data/openblas/0.3.20/aborzunov/testing/build/af70c39ddad7b91ea16b95b5cceba7ed41791f1e/source_subfolder/driver/others/memory.c:3357 (test_package+0x3e62)
#5 __libc_start_main <null> (libc.so.6+0x29eba)
SUMMARY: ThreadSanitizer: data race /root/.conan/data/openblas/0.3.20/aborzunov/testing/build/af70c39ddad7b91ea16b95b5cceba7ed41791f1e/source_subfolder/driver/others/blas_server.c:519 in blas_thread_server
==================
==================
WARNING: ThreadSanitizer: data race (pid=13857)
Read of size 8 at 0x7ffc55e63148 by thread T6:
#0 blas_thread_server /root/.conan/data/openblas/0.3.20/aborzunov/testing/build/af70c39ddad7b91ea16b95b5cceba7ed41791f1e/source_subfolder/driver/others/blas_server.c:525 (test_package+0xf3e0)
Previous write of size 8 at 0x7ffc55e63148 by main thread:
[failed to restore the stack]
Location is stack of main thread.
Location is global '<null>' at 0x000000000000 ([stack]+0x00000001e148)
Thread T6 (tid=13864, running) created by main thread at:
#0 pthread_create ../../../../src/libsanitizer/tsan/tsan_interceptors_posix.cpp:969 (libtsan.so.0+0x605b8)
#1 blas_thread_init /root/.conan/data/openblas/0.3.20/aborzunov/testing/build/af70c39ddad7b91ea16b95b5cceba7ed41791f1e/source_subfolder/driver/others/blas_server.c:667 (test_package+0xf771)
#2 blas_thread_init /root/.conan/data/openblas/0.3.20/aborzunov/testing/build/af70c39ddad7b91ea16b95b5cceba7ed41791f1e/source_subfolder/driver/others/blas_server.c:628 (test_package+0xf771)
#3 gotoblas_memory_init /root/.conan/data/openblas/0.3.20/aborzunov/testing/build/af70c39ddad7b91ea16b95b5cceba7ed41791f1e/source_subfolder/driver/others/memory.c:3314 (test_package+0x3e62)
#4 gotoblas_init /root/.conan/data/openblas/0.3.20/aborzunov/testing/build/af70c39ddad7b91ea16b95b5cceba7ed41791f1e/source_subfolder/driver/others/memory.c:3357 (test_package+0x3e62)
#5 __libc_start_main <null> (libc.so.6+0x29eba)
SUMMARY: ThreadSanitizer: data race /root/.conan/data/openblas/0.3.20/aborzunov/testing/build/af70c39ddad7b91ea16b95b5cceba7ed41791f1e/source_subfolder/driver/others/blas_server.c:525 in blas_thread_server
==================
==================
WARNING: ThreadSanitizer: data race (pid=13857)
Write of size 8 at 0x7ffc55e634a8 by main thread:
#0 main /root/conan-center-index/recipes/openblas/all/test_package/test_package.cpp:9 (test_package+0x3a15)
Previous read of size 8 at 0x7ffc55e634a8 by thread T11:
[failed to restore the stack]
Location is stack of main thread.
Location is global '<null>' at 0x000000000000 ([stack]+0x00000001e4a8)
Thread T11 (tid=13869, running) created by main thread at:
#0 pthread_create ../../../../src/libsanitizer/tsan/tsan_interceptors_posix.cpp:969 (libtsan.so.0+0x605b8)
#1 blas_thread_init /root/.conan/data/openblas/0.3.20/aborzunov/testing/build/af70c39ddad7b91ea16b95b5cceba7ed41791f1e/source_subfolder/driver/others/blas_server.c:667 (test_package+0xf771)
#2 blas_thread_init /root/.conan/data/openblas/0.3.20/aborzunov/testing/build/af70c39ddad7b91ea16b95b5cceba7ed41791f1e/source_subfolder/driver/others/blas_server.c:628 (test_package+0xf771)
#3 gotoblas_memory_init /root/.conan/data/openblas/0.3.20/aborzunov/testing/build/af70c39ddad7b91ea16b95b5cceba7ed41791f1e/source_subfolder/driver/others/memory.c:3314 (test_package+0x3e62)
#4 gotoblas_init /root/.conan/data/openblas/0.3.20/aborzunov/testing/build/af70c39ddad7b91ea16b95b5cceba7ed41791f1e/source_subfolder/driver/others/memory.c:3357 (test_package+0x3e62)
#5 __libc_start_main <null> (libc.so.6+0x29eba)
SUMMARY: ThreadSanitizer: data race /root/conan-center-index/recipes/openblas/all/test_package/test_package.cpp:9 in main
==================
==================
WARNING: ThreadSanitizer: data race (pid=13857)
Write of size 8 at 0x7ffc55e634b0 by main thread:
#0 main /root/conan-center-index/recipes/openblas/all/test_package/test_package.cpp:9 (test_package+0x3a29)
Previous read of size 8 at 0x7ffc55e634b0 by thread T11:
[failed to restore the stack]
Location is stack of main thread.
Location is global '<null>' at 0x000000000000 ([stack]+0x00000001e4b0)
Thread T11 (tid=13869, running) created by main thread at:
#0 pthread_create ../../../../src/libsanitizer/tsan/tsan_interceptors_posix.cpp:969 (libtsan.so.0+0x605b8)
#1 blas_thread_init /root/.conan/data/openblas/0.3.20/aborzunov/testing/build/af70c39ddad7b91ea16b95b5cceba7ed41791f1e/source_subfolder/driver/others/blas_server.c:667 (test_package+0xf771)
#2 blas_thread_init /root/.conan/data/openblas/0.3.20/aborzunov/testing/build/af70c39ddad7b91ea16b95b5cceba7ed41791f1e/source_subfolder/driver/others/blas_server.c:628 (test_package+0xf771)
#3 gotoblas_memory_init /root/.conan/data/openblas/0.3.20/aborzunov/testing/build/af70c39ddad7b91ea16b95b5cceba7ed41791f1e/source_subfolder/driver/others/memory.c:3314 (test_package+0x3e62)
#4 gotoblas_init /root/.conan/data/openblas/0.3.20/aborzunov/testing/build/af70c39ddad7b91ea16b95b5cceba7ed41791f1e/source_subfolder/driver/others/memory.c:3357 (test_package+0x3e62)
#5 __libc_start_main <null> (libc.so.6+0x29eba)
SUMMARY: ThreadSanitizer: data race /root/conan-center-index/recipes/openblas/all/test_package/test_package.cpp:9 in main
==================
==================
WARNING: ThreadSanitizer: data race (pid=13857)
Write of size 8 at 0x7ffc55e63360 by main thread:
#0 cblas_dgemm /root/.conan/data/openblas/0.3.20/aborzunov/testing/build/af70c39ddad7b91ea16b95b5cceba7ed41791f1e/source_subfolder/interface/gemm.c:341 (test_package+0x4149)
#1 main /root/conan-center-index/recipes/openblas/all/test_package/test_package.cpp:10 (test_package+0x3acc)
Previous read of size 8 at 0x7ffc55e63360 by thread T9:
[failed to restore the stack]
Location is stack of main thread.
Location is global '<null>' at 0x000000000000 ([stack]+0x00000001e360)
Thread T9 (tid=13867, running) created by main thread at:
#0 pthread_create ../../../../src/libsanitizer/tsan/tsan_interceptors_posix.cpp:969 (libtsan.so.0+0x605b8)
#1 blas_thread_init /root/.conan/data/openblas/0.3.20/aborzunov/testing/build/af70c39ddad7b91ea16b95b5cceba7ed41791f1e/source_subfolder/driver/others/blas_server.c:667 (test_package+0xf771)
#2 blas_thread_init /root/.conan/data/openblas/0.3.20/aborzunov/testing/build/af70c39ddad7b91ea16b95b5cceba7ed41791f1e/source_subfolder/driver/others/blas_server.c:628 (test_package+0xf771)
#3 gotoblas_memory_init /root/.conan/data/openblas/0.3.20/aborzunov/testing/build/af70c39ddad7b91ea16b95b5cceba7ed41791f1e/source_subfolder/driver/others/memory.c:3314 (test_package+0x3e62)
#4 gotoblas_init /root/.conan/data/openblas/0.3.20/aborzunov/testing/build/af70c39ddad7b91ea16b95b5cceba7ed41791f1e/source_subfolder/driver/others/memory.c:3357 (test_package+0x3e62)
#5 __libc_start_main <null> (libc.so.6+0x29eba)
SUMMARY: ThreadSanitizer: data race /root/.conan/data/openblas/0.3.20/aborzunov/testing/build/af70c39ddad7b91ea16b95b5cceba7ed41791f1e/source_subfolder/interface/gemm.c:341 in cblas_dgemm
==================
11.000000 -9.000000 5.000000 -9.000000 21.000000 -1.000000 5.000000 -1.000000 3.000000
ThreadSanitizer: reported 10 warnings
ERROR: openblas/0.3.20@aborzunov/testing (test package): Error in test() method, line 17
self.run(bin_path, run_environment=True)
ConanException: Error 66 while executing bin/test_package
Steps to reproduce with docker:
(dockerfile and accompanying files)
❯ cat Dockerfile
FROM ubuntu:22.04
RUN apt-get update; \
apt-get install -y software-properties-common apt-utils; \
add-apt-repository -y universe; \
add-apt-repository -y ppa:ubuntu-toolchain-r/ppa; \
apt-get update; \
apt-get -y install python3-pip gcc-11 g++-11 cmake build-essential git; \
pip3 install conan; \
cd; \
mkdir -p .conan/profiles;
COPY default /root/.conan/profiles
COPY tsan /root/.conan/profiles
COPY settings.yml /root/.conan/
❯ cat default
[settings]
os=Linux
os_build=Linux
arch=x86_64
arch_build=x86_64
compiler=gcc
compiler.version=11.3
compiler.libcxx=libstdc++11
compiler.cppstd=20
build_type=Release
[options]
[conf]
[build_requires]
[env]
CC=gcc-11
CXX=g++-11
CUDAHOSTCXX=g++-11
❯ cat tsan
[settings]
build_type=RelWithDebInfo
compiler.sanitizer=Thread
[env]
CFLAGS=-fno-omit-frame-pointer -fsanitize=thread
CXXFLAGS=-fno-omit-frame-pointer -fsanitize=thread
LDFLAGS=-fsanitize=thread
docker build -t ubuntu-22.04-gcc-11-conan ./
docker run -it --rm --name gcc11 ubuntu-22.04-gcc-11-conan /bin/bash
cd
git clone https://github.com/conan-io/conan-center-index
cd conan-center-index/recipes/openblas
conan create all openblas/0.3.20@aborzunov/testing --profile=default --profile=tsan
Each race pair has "failed to restore stack" on one side. Could you compile with gcc -O0 -g or something that gives proper backtraces?
It is miscompiled if it uses threads for such a small sample!!!! It does not relieve thread safety issue if any BUT.
could be a side effect of building with tsan support - not sure if these are genuine or just false positives
Example compiled with USE_THREADS = true
.
Building and running in Debug
does not produce more valuable output:
tsan output
==================
WARNING: ThreadSanitizer: data race (pid=13691)
Write of size 4 at 0x56314c13197c by thread T10:
#0 alloc_mmap /root/.conan/data/openblas/0.3.20/aborzunov/testing/build/626b9609ed430a79075b34763493768e66964159/source_subfolder/driver/others/memory.c:2301 (test_package+0x111f3)
#1 blas_memory_alloc /root/.conan/data/openblas/0.3.20/aborzunov/testing/build/626b9609ed430a79075b34763493768e66964159/source_subfolder/driver/others/memory.c:2925 (test_package+0x11931)
#2 blas_thread_server /root/.conan/data/openblas/0.3.20/aborzunov/testing/build/626b9609ed430a79075b34763493768e66964159/source_subfolder/driver/others/blas_server.c:382 (test_package+0x13150)
Previous read of size 4 at 0x56314c13197c by thread T9:
#0 alloc_mmap /root/.conan/data/openblas/0.3.20/aborzunov/testing/build/626b9609ed430a79075b34763493768e66964159/source_subfolder/driver/others/memory.c:2225 (test_package+0x10ffc)
#1 blas_memory_alloc /root/.conan/data/openblas/0.3.20/aborzunov/testing/build/626b9609ed430a79075b34763493768e66964159/source_subfolder/driver/others/memory.c:2925 (test_package+0x11931)
#2 blas_thread_server /root/.conan/data/openblas/0.3.20/aborzunov/testing/build/626b9609ed430a79075b34763493768e66964159/source_subfolder/driver/others/blas_server.c:382 (test_package+0x13150)
Location is global 'hot_alloc' of size 4 at 0x56314c13197c (test_package+0x00000002a97c)
Thread T10 (tid=13702, running) created by main thread at:
#0 pthread_create ../../../../src/libsanitizer/tsan/tsan_interceptors_posix.cpp:969 (libtsan.so.0+0x605b8)
#1 blas_thread_init /root/.conan/data/openblas/0.3.20/aborzunov/testing/build/626b9609ed430a79075b34763493768e66964159/source_subfolder/driver/others/blas_server.c:667 (test_package+0x13881)
#2 gotoblas_memory_init /root/.conan/data/openblas/0.3.20/aborzunov/testing/build/626b9609ed430a79075b34763493768e66964159/source_subfolder/driver/others/memory.c:3314 (test_package+0x12840)
#3 gotoblas_init /root/.conan/data/openblas/0.3.20/aborzunov/testing/build/626b9609ed430a79075b34763493768e66964159/source_subfolder/driver/others/memory.c:3357 (test_package+0x128a6)
#4 __libc_start_main <null> (libc.so.6+0x29eba)
Thread T9 (tid=13701, running) created by main thread at:
#0 pthread_create ../../../../src/libsanitizer/tsan/tsan_interceptors_posix.cpp:969 (libtsan.so.0+0x605b8)
#1 blas_thread_init /root/.conan/data/openblas/0.3.20/aborzunov/testing/build/626b9609ed430a79075b34763493768e66964159/source_subfolder/driver/others/blas_server.c:667 (test_package+0x13881)
#2 gotoblas_memory_init /root/.conan/data/openblas/0.3.20/aborzunov/testing/build/626b9609ed430a79075b34763493768e66964159/source_subfolder/driver/others/memory.c:3314 (test_package+0x12840)
#3 gotoblas_init /root/.conan/data/openblas/0.3.20/aborzunov/testing/build/626b9609ed430a79075b34763493768e66964159/source_subfolder/driver/others/memory.c:3357 (test_package+0x128a6)
#4 __libc_start_main <null> (libc.so.6+0x29eba)
SUMMARY: ThreadSanitizer: data race /root/.conan/data/openblas/0.3.20/aborzunov/testing/build/626b9609ed430a79075b34763493768e66964159/source_subfolder/driver/others/memory.c:2301 in alloc_mmap
==================
==================
WARNING: ThreadSanitizer: data race (pid=13691)
Read of size 8 at 0x7ffd7b969040 by thread T10:
#0 blas_thread_server /root/.conan/data/openblas/0.3.20/aborzunov/testing/build/626b9609ed430a79075b34763493768e66964159/source_subfolder/driver/others/blas_server.c:448 (test_package+0x133c6)
Previous write of size 8 at 0x7ffd7b969040 by main thread:
[failed to restore the stack]
Location is stack of main thread.
Location is global '<null>' at 0x000000000000 ([stack]+0x00000001f040)
Thread T10 (tid=13702, running) created by main thread at:
#0 pthread_create ../../../../src/libsanitizer/tsan/tsan_interceptors_posix.cpp:969 (libtsan.so.0+0x605b8)
#1 blas_thread_init /root/.conan/data/openblas/0.3.20/aborzunov/testing/build/626b9609ed430a79075b34763493768e66964159/source_subfolder/driver/others/blas_server.c:667 (test_package+0x13881)
#2 gotoblas_memory_init /root/.conan/data/openblas/0.3.20/aborzunov/testing/build/626b9609ed430a79075b34763493768e66964159/source_subfolder/driver/others/memory.c:3314 (test_package+0x12840)
#3 gotoblas_init /root/.conan/data/openblas/0.3.20/aborzunov/testing/build/626b9609ed430a79075b34763493768e66964159/source_subfolder/driver/others/memory.c:3357 (test_package+0x128a6)
#4 __libc_start_main <null> (libc.so.6+0x29eba)
SUMMARY: ThreadSanitizer: data race /root/.conan/data/openblas/0.3.20/aborzunov/testing/build/626b9609ed430a79075b34763493768e66964159/source_subfolder/driver/others/blas_server.c:448 in blas_thread_server
==================
==================
WARNING: ThreadSanitizer: data race (pid=13691)
Read of size 8 at 0x7ffd7b969048 by thread T10:
#0 blas_thread_server /root/.conan/data/openblas/0.3.20/aborzunov/testing/build/626b9609ed430a79075b34763493768e66964159/source_subfolder/driver/others/blas_server.c:449 (test_package+0x133e2)
Previous write of size 8 at 0x7ffd7b969048 by main thread:
[failed to restore the stack]
Location is stack of main thread.
Location is global '<null>' at 0x000000000000 ([stack]+0x00000001f048)
Thread T10 (tid=13702, running) created by main thread at:
#0 pthread_create ../../../../src/libsanitizer/tsan/tsan_interceptors_posix.cpp:969 (libtsan.so.0+0x605b8)
#1 blas_thread_init /root/.conan/data/openblas/0.3.20/aborzunov/testing/build/626b9609ed430a79075b34763493768e66964159/source_subfolder/driver/others/blas_server.c:667 (test_package+0x13881)
#2 gotoblas_memory_init /root/.conan/data/openblas/0.3.20/aborzunov/testing/build/626b9609ed430a79075b34763493768e66964159/source_subfolder/driver/others/memory.c:3314 (test_package+0x12840)
#3 gotoblas_init /root/.conan/data/openblas/0.3.20/aborzunov/testing/build/626b9609ed430a79075b34763493768e66964159/source_subfolder/driver/others/memory.c:3357 (test_package+0x128a6)
#4 __libc_start_main <null> (libc.so.6+0x29eba)
SUMMARY: ThreadSanitizer: data race /root/.conan/data/openblas/0.3.20/aborzunov/testing/build/626b9609ed430a79075b34763493768e66964159/source_subfolder/driver/others/blas_server.c:449 in blas_thread_server
==================
==================
WARNING: ThreadSanitizer: data race (pid=13691)
Read of size 4 at 0x7ffd7b9690b0 by thread T10:
#0 blas_thread_server /root/.conan/data/openblas/0.3.20/aborzunov/testing/build/626b9609ed430a79075b34763493768e66964159/source_subfolder/driver/others/blas_server.c:470 (test_package+0x1341a)
Previous write of size 4 at 0x7ffd7b9690b0 by main thread:
[failed to restore the stack]
Location is stack of main thread.
Location is global '<null>' at 0x000000000000 ([stack]+0x00000001f0b0)
Thread T10 (tid=13702, running) created by main thread at:
#0 pthread_create ../../../../src/libsanitizer/tsan/tsan_interceptors_posix.cpp:969 (libtsan.so.0+0x605b8)
#1 blas_thread_init /root/.conan/data/openblas/0.3.20/aborzunov/testing/build/626b9609ed430a79075b34763493768e66964159/source_subfolder/driver/others/blas_server.c:667 (test_package+0x13881)
#2 gotoblas_memory_init /root/.conan/data/openblas/0.3.20/aborzunov/testing/build/626b9609ed430a79075b34763493768e66964159/source_subfolder/driver/others/memory.c:3314 (test_package+0x12840)
#3 gotoblas_init /root/.conan/data/openblas/0.3.20/aborzunov/testing/build/626b9609ed430a79075b34763493768e66964159/source_subfolder/driver/others/memory.c:3357 (test_package+0x128a6)
#4 __libc_start_main <null> (libc.so.6+0x29eba)
SUMMARY: ThreadSanitizer: data race /root/.conan/data/openblas/0.3.20/aborzunov/testing/build/626b9609ed430a79075b34763493768e66964159/source_subfolder/driver/others/blas_server.c:470 in blas_thread_server
==================
==================
WARNING: ThreadSanitizer: data race (pid=13691)
Read of size 8 at 0x7ffd7b968a28 by thread T1:
#0 blas_thread_server /root/.conan/data/openblas/0.3.20/aborzunov/testing/build/626b9609ed430a79075b34763493768e66964159/source_subfolder/driver/others/blas_server.c:444 (test_package+0x13384)
Previous write of size 8 at 0x7ffd7b968a28 by main thread:
[failed to restore the stack]
Location is stack of main thread.
Location is global '<null>' at 0x000000000000 ([stack]+0x00000001ea28)
Thread T1 (tid=13693, running) created by main thread at:
#0 pthread_create ../../../../src/libsanitizer/tsan/tsan_interceptors_posix.cpp:969 (libtsan.so.0+0x605b8)
#1 blas_thread_init /root/.conan/data/openblas/0.3.20/aborzunov/testing/build/626b9609ed430a79075b34763493768e66964159/source_subfolder/driver/others/blas_server.c:667 (test_package+0x13881)
#2 gotoblas_memory_init /root/.conan/data/openblas/0.3.20/aborzunov/testing/build/626b9609ed430a79075b34763493768e66964159/source_subfolder/driver/others/memory.c:3314 (test_package+0x12840)
#3 gotoblas_init /root/.conan/data/openblas/0.3.20/aborzunov/testing/build/626b9609ed430a79075b34763493768e66964159/source_subfolder/driver/others/memory.c:3357 (test_package+0x128a6)
#4 __libc_start_main <null> (libc.so.6+0x29eba)
SUMMARY: ThreadSanitizer: data race /root/.conan/data/openblas/0.3.20/aborzunov/testing/build/626b9609ed430a79075b34763493768e66964159/source_subfolder/driver/others/blas_server.c:444 in blas_thread_server
==================
==================
WARNING: ThreadSanitizer: data race (pid=13691)
Read of size 8 at 0x7ffd7b968a40 by thread T1:
#0 blas_thread_server /root/.conan/data/openblas/0.3.20/aborzunov/testing/build/626b9609ed430a79075b34763493768e66964159/source_subfolder/driver/others/blas_server.c:525 (test_package+0x13648)
Previous write of size 8 at 0x7ffd7b968a40 by main thread:
[failed to restore the stack]
Location is stack of main thread.
Location is global '<null>' at 0x000000000000 ([stack]+0x00000001ea40)
Thread T1 (tid=13693, running) created by main thread at:
#0 pthread_create ../../../../src/libsanitizer/tsan/tsan_interceptors_posix.cpp:969 (libtsan.so.0+0x605b8)
#1 blas_thread_init /root/.conan/data/openblas/0.3.20/aborzunov/testing/build/626b9609ed430a79075b34763493768e66964159/source_subfolder/driver/others/blas_server.c:667 (test_package+0x13881)
#2 gotoblas_memory_init /root/.conan/data/openblas/0.3.20/aborzunov/testing/build/626b9609ed430a79075b34763493768e66964159/source_subfolder/driver/others/memory.c:3314 (test_package+0x12840)
#3 gotoblas_init /root/.conan/data/openblas/0.3.20/aborzunov/testing/build/626b9609ed430a79075b34763493768e66964159/source_subfolder/driver/others/memory.c:3357 (test_package+0x128a6)
#4 __libc_start_main <null> (libc.so.6+0x29eba)
SUMMARY: ThreadSanitizer: data race /root/.conan/data/openblas/0.3.20/aborzunov/testing/build/626b9609ed430a79075b34763493768e66964159/source_subfolder/driver/others/blas_server.c:525 in blas_thread_server
==================
==================
WARNING: ThreadSanitizer: data race (pid=13691)
Read of size 8 at 0x7ffd7b969018 by thread T10:
#0 blas_thread_server /root/.conan/data/openblas/0.3.20/aborzunov/testing/build/626b9609ed430a79075b34763493768e66964159/source_subfolder/driver/others/blas_server.c:525 (test_package+0x13600)
Previous write of size 8 at 0x7ffd7b969018 by main thread:
[failed to restore the stack]
Location is stack of main thread.
Location is global '<null>' at 0x000000000000 ([stack]+0x00000001f018)
Thread T10 (tid=13702, running) created by main thread at:
#0 pthread_create ../../../../src/libsanitizer/tsan/tsan_interceptors_posix.cpp:969 (libtsan.so.0+0x605b8)
#1 blas_thread_init /root/.conan/data/openblas/0.3.20/aborzunov/testing/build/626b9609ed430a79075b34763493768e66964159/source_subfolder/driver/others/blas_server.c:667 (test_package+0x13881)
#2 gotoblas_memory_init /root/.conan/data/openblas/0.3.20/aborzunov/testing/build/626b9609ed430a79075b34763493768e66964159/source_subfolder/driver/others/memory.c:3314 (test_package+0x12840)
#3 gotoblas_init /root/.conan/data/openblas/0.3.20/aborzunov/testing/build/626b9609ed430a79075b34763493768e66964159/source_subfolder/driver/others/memory.c:3357 (test_package+0x128a6)
#4 __libc_start_main <null> (libc.so.6+0x29eba)
SUMMARY: ThreadSanitizer: data race /root/.conan/data/openblas/0.3.20/aborzunov/testing/build/626b9609ed430a79075b34763493768e66964159/source_subfolder/driver/others/blas_server.c:525 in blas_thread_server
==================
==================
WARNING: ThreadSanitizer: data race (pid=13691)
Write of size 8 at 0x7ffd7b9690d8 by main thread:
#0 main /root/conan-center-index/recipes/openblas/all/test_package/test_package.cpp:9 (test_package+0x3b4a)
Previous read of size 8 at 0x7ffd7b9690d8 by thread T11:
[failed to restore the stack]
Location is stack of main thread.
Location is global '<null>' at 0x000000000000 ([stack]+0x00000001f0d8)
Thread T11 (tid=13703, running) created by main thread at:
#0 pthread_create ../../../../src/libsanitizer/tsan/tsan_interceptors_posix.cpp:969 (libtsan.so.0+0x605b8)
#1 blas_thread_init /root/.conan/data/openblas/0.3.20/aborzunov/testing/build/626b9609ed430a79075b34763493768e66964159/source_subfolder/driver/others/blas_server.c:667 (test_package+0x13881)
#2 gotoblas_memory_init /root/.conan/data/openblas/0.3.20/aborzunov/testing/build/626b9609ed430a79075b34763493768e66964159/source_subfolder/driver/others/memory.c:3314 (test_package+0x12840)
#3 gotoblas_init /root/.conan/data/openblas/0.3.20/aborzunov/testing/build/626b9609ed430a79075b34763493768e66964159/source_subfolder/driver/others/memory.c:3357 (test_package+0x128a6)
#4 __libc_start_main <null> (libc.so.6+0x29eba)
SUMMARY: ThreadSanitizer: data race /root/conan-center-index/recipes/openblas/all/test_package/test_package.cpp:9 in main
==================
==================
WARNING: ThreadSanitizer: data race (pid=13691)
Write of size 8 at 0x7ffd7b9690e0 by main thread:
#0 main /root/conan-center-index/recipes/openblas/all/test_package/test_package.cpp:9 (test_package+0x3b67)
Previous read of size 8 at 0x7ffd7b9690e0 by thread T11:
[failed to restore the stack]
Location is stack of main thread.
Location is global '<null>' at 0x000000000000 ([stack]+0x00000001f0e0)
Thread T11 (tid=13703, running) created by main thread at:
#0 pthread_create ../../../../src/libsanitizer/tsan/tsan_interceptors_posix.cpp:969 (libtsan.so.0+0x605b8)
#1 blas_thread_init /root/.conan/data/openblas/0.3.20/aborzunov/testing/build/626b9609ed430a79075b34763493768e66964159/source_subfolder/driver/others/blas_server.c:667 (test_package+0x13881)
#2 gotoblas_memory_init /root/.conan/data/openblas/0.3.20/aborzunov/testing/build/626b9609ed430a79075b34763493768e66964159/source_subfolder/driver/others/memory.c:3314 (test_package+0x12840)
#3 gotoblas_init /root/.conan/data/openblas/0.3.20/aborzunov/testing/build/626b9609ed430a79075b34763493768e66964159/source_subfolder/driver/others/memory.c:3357 (test_package+0x128a6)
#4 __libc_start_main <null> (libc.so.6+0x29eba)
SUMMARY: ThreadSanitizer: data race /root/conan-center-index/recipes/openblas/all/test_package/test_package.cpp:9 in main
==================
==================
WARNING: ThreadSanitizer: data race (pid=13691)
Write of size 8 at 0x7ffd7b968f88 by main thread:
#0 cblas_dgemm /root/.conan/data/openblas/0.3.20/aborzunov/testing/build/626b9609ed430a79075b34763493768e66964159/source_subfolder/interface/gemm.c:354 (test_package+0x3e62)
#1 main /root/conan-center-index/recipes/openblas/all/test_package/test_package.cpp:10 (test_package+0x3c06)
Previous read of size 8 at 0x7ffd7b968f88 by thread T9:
#0 blas_thread_server /root/.conan/data/openblas/0.3.20/aborzunov/testing/build/626b9609ed430a79075b34763493768e66964159/source_subfolder/driver/others/blas_server.c:525 (test_package+0x13630)
Location is stack of main thread.
Location is global '<null>' at 0x000000000000 ([stack]+0x00000001ef88)
Thread T9 (tid=13701, running) created by main thread at:
#0 pthread_create ../../../../src/libsanitizer/tsan/tsan_interceptors_posix.cpp:969 (libtsan.so.0+0x605b8)
#1 blas_thread_init /root/.conan/data/openblas/0.3.20/aborzunov/testing/build/626b9609ed430a79075b34763493768e66964159/source_subfolder/driver/others/blas_server.c:667 (test_package+0x13881)
#2 gotoblas_memory_init /root/.conan/data/openblas/0.3.20/aborzunov/testing/build/626b9609ed430a79075b34763493768e66964159/source_subfolder/driver/others/memory.c:3314 (test_package+0x12840)
#3 gotoblas_init /root/.conan/data/openblas/0.3.20/aborzunov/testing/build/626b9609ed430a79075b34763493768e66964159/source_subfolder/driver/others/memory.c:3357 (test_package+0x128a6)
#4 __libc_start_main <null> (libc.so.6+0x29eba)
SUMMARY: ThreadSanitizer: data race /root/.conan/data/openblas/0.3.20/aborzunov/testing/build/626b9609ed430a79075b34763493768e66964159/source_subfolder/interface/gemm.c:354 in cblas_dgemm
==================
==================
WARNING: ThreadSanitizer: data race (pid=13691)
Write of size 8 at 0x7ffd7b968f90 by main thread:
#0 cblas_dgemm /root/.conan/data/openblas/0.3.20/aborzunov/testing/build/626b9609ed430a79075b34763493768e66964159/source_subfolder/interface/gemm.c:355 (test_package+0x3e82)
#1 main /root/conan-center-index/recipes/openblas/all/test_package/test_package.cpp:10 (test_package+0x3c06)
Previous read of size 8 at 0x7ffd7b968f90 by thread T9:
#0 blas_thread_server /root/.conan/data/openblas/0.3.20/aborzunov/testing/build/626b9609ed430a79075b34763493768e66964159/source_subfolder/driver/others/blas_server.c:525 (test_package+0x13618)
Location is stack of main thread.
Location is global '<null>' at 0x000000000000 ([stack]+0x00000001ef90)
Thread T9 (tid=13701, running) created by main thread at:
#0 pthread_create ../../../../src/libsanitizer/tsan/tsan_interceptors_posix.cpp:969 (libtsan.so.0+0x605b8)
#1 blas_thread_init /root/.conan/data/openblas/0.3.20/aborzunov/testing/build/626b9609ed430a79075b34763493768e66964159/source_subfolder/driver/others/blas_server.c:667 (test_package+0x13881)
#2 gotoblas_memory_init /root/.conan/data/openblas/0.3.20/aborzunov/testing/build/626b9609ed430a79075b34763493768e66964159/source_subfolder/driver/others/memory.c:3314 (test_package+0x12840)
#3 gotoblas_init /root/.conan/data/openblas/0.3.20/aborzunov/testing/build/626b9609ed430a79075b34763493768e66964159/source_subfolder/driver/others/memory.c:3357 (test_package+0x128a6)
#4 __libc_start_main <null> (libc.so.6+0x29eba)
SUMMARY: ThreadSanitizer: data race /root/.conan/data/openblas/0.3.20/aborzunov/testing/build/626b9609ed430a79075b34763493768e66964159/source_subfolder/interface/gemm.c:355 in cblas_dgemm
==================
11.000000 -9.000000 5.000000 -9.000000 21.000000 -1.000000 5.000000 -1.000000 3.000000
ThreadSanitizer: reported 11 warnings
ERROR: openblas/0.3.20@aborzunov/testing (test package): Error in test() method, line 17
self.run(bin_path, run_environment=True)
ConanException: Error 66 while executing bin/test_package
Simple explanation: tsan breaks wildly if mixing instrumented and non-instrumented objects in same file I fail to repeat claim gcc4(helgrind...) gcc10 .. gcc12
git clone https://github.com/xianyi/openblas
cd openblas
make clean
make COMMON_OPT="-O2 -fsanitize=thread" ONLY_CBLAS=1
cc -L`pwd` -l openblas -fsanitize=thread -Wl,-rpath=. 3787.c
./a.out
EDIT: I added sleep and abort, within gdb - actually !main threads are plainly initialized after gemm exits, without any work ever being sent to them. EDIT2: If number of CPUs exceeds x in blas_quickdivide strides get assigned past input or output (?) I have VM with 2 cores, seems samples have them ten.
blas_quickdivide is very simplistic (and it has little information to work with) but there is supposed to be bounds checking in place on its results. can you provide more context on your "EDIT2" comment ?
Just that work is not divided N.N.N.N.M but around and about like N,N+1,N-1,M but correctly in every case, including marginal or 1.1.0.0.0.0 case. No problem at all, not at least in L1 and L2, though L3 exceeds my screen size, so hard to tell.
Regarding optimizing threshold away - initializing variable to 1 at declaration will not be blown away.
@aborzunov you must use -fsanitize=thread in your project too, and check with ldd
that target binary does not import libpthread or other version of sanitizersomewhere besides thread sanitizer emulation. What it seems to me that normal pthreads activity is recored there as some conflicts recored have just started thread vs main, and backtraces absent i would bet on main thread stack portion used for IPC there.