CPUBBoxUtilTest fail on NVIDIA TX2

[falconray0704]

Please use the caffe-users list for usage, installation, or modeling questions, or other requests for help. Do not post such requests to Issues. Doing so interferes with the development of Caffe.

Please read the guidelines for contributing before submitting this issue.

Issue summary

CPUBBoxUtilTest fail on NVIDIA TX2

[----------] 30 tests from CPUBBoxUtilTest [ RUN ] CPUBBoxUtilTest.TestBBoxSize [ OK ] CPUBBoxUtilTest.TestBBoxSize (0 ms) [ RUN ] CPUBBoxUtilTest.TestOutputBBox F1027 11:26:41.299537 28483 test_bbox_util.cpp:279] Check failed: out_bbox.xmax() == 50. (50 vs. 50) *** Check failure stack trace: *** @ 0x7faf5f6718 google::LogMessage::Fail() @ 0x7faf5f8614 google::LogMessage::SendToLog() @ 0x7faf5f6290 google::LogMessage::Flush() @ 0x7faf5f8eb4 google::LogMessageFatal::~LogMessageFatal() @ 0x751018 caffe::CPUBBoxUtilTest_TestOutputBBox_Test::TestBody() @ 0xab2144 testing::internal::HandleExceptionsInMethodIfSupported<>() @ 0xaab3ec testing::Test::Run() @ 0xaab528 testing::TestInfo::Run() @ 0xaab5e8 testing::TestCase::Run() @ 0xaac748 testing::internal::UnitTestImpl::RunAllTests() @ 0xaaca5c testing::UnitTest::Run() @ 0x595600 main @ 0x7fadc438a0 __libc_start_main Makefile:528: recipe for target 'runtest' failed make: *** [runtest] Aborted (core dumped) nvidia@tegra-ubuntu:/md/ray/github/weiliu89/caffe$

Steps to reproduce

< 1 > cp Makefile.config.example Makefile.config < 2 > vim Makefile.config to make some changes:

nvidia@tegra-ubuntu:/md/ray/github/weiliu89/caffe$ diff Makefile.config Makefile.config.example 5c5 < USE_CUDNN := 1

USE_CUDNN := 1

35,46c35,41 < #CUDA_ARCH := -gencode arch=compute_20,code=sm_20
< # -gencode arch=compute_20,code=sm_21
< # -gencode arch=compute_30,code=sm_30
< # -gencode arch=compute_35,code=sm_35
< # -gencode arch=compute_50,code=sm_50
< # -gencode arch=compute_52,code=sm_52
< # -gencode arch=compute_61,code=sm_61 < < #CUDA_ARCH := -gencode arch=compute_52,code=sm_52
< # -gencode arch=compute_52,code=compute_52 < CUDA_ARCH := -gencode arch=compute_62,code=sm_62 <

CUDA_ARCH := -gencode arch=compute_20,code=sm_20
-gencode arch=compute_20,code=sm_21
-gencode arch=compute_30,code=sm_30
-gencode arch=compute_35,code=sm_35
-gencode arch=compute_50,code=sm_50
-gencode arch=compute_52,code=sm_52
-gencode arch=compute_61,code=sm_61 55,56c50,53 < BLAS_INCLUDE := /opt/OpenBLAS/include/ < BLAS_LIB := /opt/OpenBLAS/lib/

---

Leave commented to accept the defaults for your choice of BLAS

(which should work)!

BLAS_INCLUDE := /path/to/your/blas

BLAS_LIB := /path/to/your/blas

95,96c92,93 < INCLUDE_DIRS := $(PYTHON_INCLUDE) /usr/local/include /usr/include/hdf5/serial < LIBRARY_DIRS := $(PYTHON_LIB) /usr/local/lib /usr/lib /usr/lib/aarch64-linux-gnu/hdf5/serial

INCLUDE_DIRS := $(PYTHON_INCLUDE) /usr/local/include LIBRARY_DIRS := $(PYTHON_LIB) /usr/local/lib /usr/lib nvidia@tegra-ubuntu:/md/ray/github/weiliu89/caffe$ ls

< 3 > vim Makefile to make some change: nvidia@tegra-ubuntu:/md/ray/github/weiliu89/caffe$ git diff Makefile diff --git a/Makefile b/Makefile index 3fd68d1..ae499a7 100644 --- a/Makefile +++ b/Makefile @@ -178,7 +178,9 @@ ifneq ($(CPU_ONLY), 1) LIBRARIES := cudart cublas curand endif

-LIBRARIES += glog gflags protobuf boost_system boost_filesystem boost_regex m hdf5_hl hdf5 +LIBRARIES += glog gflags protobuf boost_system boost_filesystem boost_regex m hdf5_serial_hl hdf5_serial +#LIBRARIES += glog gflags protobuf boost_system boost_filesystem boost_regex m hdf5_hl hdf5 hdf5_serial_hl hdf5_serial +

handle IO dependencies

USE_LEVELDB ?= 1 nvidia@tegra-ubuntu:/md/ray/github/weiliu89/caffe$

< 4 > make < 5 > make test < 6 > make runtest

If you are having difficulty building Caffe or training a model, please ask the caffe-users mailing list. If you are reporting a build error that seems to be due to a bug in Caffe, please attach your build configuration (either Makefile.config or CMakeCache.txt) and the output of the make (or cmake) command.

Your system configuration

Operating system: nvidia@tegra-ubuntu:/md/ray/github/weiliu89/caffe$ lsb_release -a No LSB modules are available. Distributor ID: Ubuntu Description: Ubuntu 16.04.3 LTS Release: 16.04 Codename: xenial nvidia@tegra-ubuntu:/md/ray/github/weiliu89/caffe$

Compiler: nvidia@tegra-ubuntu:/md/ray/github/weiliu89/caffe$ gcc -v Using built-in specs. COLLECT_GCC=gcc COLLECT_LTO_WRAPPER=/usr/lib/gcc/aarch64-linux-gnu/5/lto-wrapper Target: aarch64-linux-gnu Configured with: ../src/configure -v --with-pkgversion='Ubuntu/Linaro 5.4.0-6ubuntu1~16.04.5' --with-bugurl=file:///usr/share/doc/gcc-5/README.Bugs --enable-languages=c,ada,c++,java,go,d,fortran,objc,obj-c++ --prefix=/usr --program-suffix=-5 --enable-shared --enable-linker-build-id --libexecdir=/usr/lib --without-included-gettext --enable-threads=posix --libdir=/usr/lib --enable-nls --with-sysroot=/ --enable-clocale=gnu --enable-libstdcxx-debug --enable-libstdcxx-time=yes --with-default-libstdcxx-abi=new --enable-gnu-unique-object --disable-libquadmath --enable-plugin --with-system-zlib --disable-browser-plugin --enable-java-awt=gtk --enable-gtk-cairo --with-java-home=/usr/lib/jvm/java-1.5.0-gcj-5-arm64/jre --enable-java-home --with-jvm-root-dir=/usr/lib/jvm/java-1.5.0-gcj-5-arm64 --with-jvm-jar-dir=/usr/lib/jvm-exports/java-1.5.0-gcj-5-arm64 --with-arch-directory=aarch64 --with-ecj-jar=/usr/share/java/eclipse-ecj.jar --enable-multiarch --enable-fix-cortex-a53-843419 --disable-werror --enable-checking=release --build=aarch64-linux-gnu --host=aarch64-linux-gnu --target=aarch64-linux-gnu Thread model: posix gcc version 5.4.0 20160609 (Ubuntu/Linaro 5.4.0-6ubuntu1~16.04.5) nvidia@tegra-ubuntu:/md/ray/github/weiliu89/caffe$

CUDA version (if applicable): nvidia@tegra-ubuntu:/md/ray/github/weiliu89/caffe$ nvcc --version nvcc: NVIDIA (R) Cuda compiler driver Copyright (c) 2005-2016 NVIDIA Corporation Built on Mon_Mar_20_17:07:33_CDT_2017 Cuda compilation tools, release 8.0, V8.0.72 nvidia@tegra-ubuntu:/md/ray/github/weiliu89/caffe$

CUDNN version (if applicable): nvidia@tegra-ubuntu:~$ ll cudnn/ total 115024 drwxrwxr-x 2 nvidia nvidia 4096 Oct 24 14:19 ./ drwxr-xr-x 33 nvidia nvidia 4096 Oct 27 11:52 ../ -rw-rw-r-- 1 nvidia nvidia 60507056 Oct 24 14:19 libcudnn6_6.0.21-1+cuda8.0_arm64.deb -rw-rw-r-- 1 nvidia nvidia 51324898 Oct 24 14:19 libcudnn6-dev_6.0.21-1+cuda8.0_arm64.deb -rw-rw-r-- 1 nvidia nvidia 5936144 Oct 24 14:19 libcudnn6-doc_6.0.21-1+cuda8.0_arm64.deb nvidia@tegra-ubuntu:~$

BLAS: nvidia@tegra-ubuntu:/opt/OpenBLAS$ ll lib/ total 30996 drwxrwxr-x 4 nvidia nvidia 4096 Oct 27 09:56 ./ drwxrwxr-x 5 nvidia nvidia 4096 Oct 27 09:56 ../ drwxrwxr-x 3 nvidia nvidia 4096 Oct 27 09:56 cmake/ lrwxrwxrwx 1 nvidia nvidia 35 Oct 27 09:56 libopenblas.a -> libopenblas_cortexa57p-r0.3.0.dev.a -rw-r--r-- 1 nvidia nvidia 21325906 Oct 27 09:55 libopenblas_cortexa57p-r0.3.0.dev.a -rwxr-xr-x 1 nvidia nvidia 10391576 Oct 27 09:55 libopenblas_cortexa57p-r0.3.0.dev.so* lrwxrwxrwx 1 nvidia nvidia 36 Oct 27 09:56 libopenblas.so -> libopenblas_cortexa57p-r0.3.0.dev.so* lrwxrwxrwx 1 nvidia nvidia 36 Oct 27 09:56 libopenblas.so.0 -> libopenblas_cortexa57p-r0.3.0.dev.so* drwxrwxr-x 2 nvidia nvidia 4096 Oct 27 09:56 pkgconfig/ nvidia@tegra-ubuntu:/opt/OpenBLAS$

Python or MATLAB version (for pycaffe and matcaffe respectively):

[loulansuiye]

I met the same problem ,but I'm not sure which item should be commentted

[loulansuiye]

is necessary to set USE_LEVELDB ?= 1

[loulansuiye]

[----------] 1 test from HDF5DataLayerTest/1, where TypeParam = caffe::CPUDevice [ RUN ] HDF5DataLayerTest/1.TestRead [ OK ] HDF5DataLayerTest/1.TestRead (9 ms) [----------] 1 test from HDF5DataLayerTest/1 (10 ms total)

[----------] 3 tests from MSRAFillerTest/1, where TypeParam = double [ RUN ] MSRAFillerTest/1.TestFillFanIn [ OK ] MSRAFillerTest/1.TestFillFanIn (2 ms) [ RUN ] MSRAFillerTest/1.TestFillFanOut [ OK ] MSRAFillerTest/1.TestFillFanOut (2 ms) [ RUN ] MSRAFillerTest/1.TestFillAverage [ OK ] MSRAFillerTest/1.TestFillAverage (2 ms) [----------] 3 tests from MSRAFillerTest/1 (6 ms total)

[----------] 30 tests from CPUBBoxUtilTest [ RUN ] CPUBBoxUtilTest.TestDecodeBBoxCorner [ OK ] CPUBBoxUtilTest.TestDecodeBBoxCorner (2 ms) [ RUN ] CPUBBoxUtilTest.TestGetConfidenceScores [ OK ] CPUBBoxUtilTest.TestGetConfidenceScores (0 ms) [ RUN ] CPUBBoxUtilTest.TestScaleBBox [ OK ] CPUBBoxUtilTest.TestScaleBBox (0 ms) [ RUN ] CPUBBoxUtilTest.TestEncodeBBoxCenterSize [ OK ] CPUBBoxUtilTest.TestEncodeBBoxCenterSize (0 ms) [ RUN ] CPUBBoxUtilTest.TestApplyNMS [ OK ] CPUBBoxUtilTest.TestApplyNMS (0 ms) [ RUN ] CPUBBoxUtilTest.TestBBoxSize [ OK ] CPUBBoxUtilTest.TestBBoxSize (0 ms) [ RUN ] CPUBBoxUtilTest.TestEncodeBBoxCorner [ OK ] CPUBBoxUtilTest.TestEncodeBBoxCorner (0 ms) [ RUN ] CPUBBoxUtilTest.TestApplyNMSFast [ OK ] CPUBBoxUtilTest.TestApplyNMSFast (0 ms) [ RUN ] CPUBBoxUtilTest.TestMatchBBoxLableOnePerPrediction [ OK ] CPUBBoxUtilTest.TestMatchBBoxLableOnePerPrediction (0 ms) [ RUN ] CPUBBoxUtilTest.TestMatchBBoxLableAllPerPredictionEx [ OK ] CPUBBoxUtilTest.TestMatchBBoxLableAllPerPredictionEx (0 ms) [ RUN ] CPUBBoxUtilTest.TestMatchBBoxLableOneBipartite [ OK ] CPUBBoxUtilTest.TestMatchBBoxLableOneBipartite (0 ms) [ RUN ] CPUBBoxUtilTest.TestMatchBBoxLableAllPerPrediction [ OK ] CPUBBoxUtilTest.TestMatchBBoxLableAllPerPrediction (0 ms) [ RUN ] CPUBBoxUtilTest.TestCumSum [ OK ] CPUBBoxUtilTest.TestCumSum (0 ms) [ RUN ] CPUBBoxUtilTest.TestGetPriorBBoxes [ OK ] CPUBBoxUtilTest.TestGetPriorBBoxes (0 ms) [ RUN ] CPUBBoxUtilTest.TestOutputBBox F1101 12:03:24.811066 20352 test_bbox_util.cpp:279] Check failed: out_bbox.xmax() == 50. (50 vs. 50) *** Check failure stack trace: *** @ 0x7fa22d6718 google::LogMessage::Fail() @ 0x7fa22d8614 google::LogMessage::SendToLog() @ 0x7fa22d6290 google::LogMessage::Flush() @ 0x7fa22d8eb4 google::LogMessageFatal::~LogMessageFatal() @ 0x9a5560 caffe::CPUBBoxUtilTest_TestOutputBBox_Test::TestBody() @ 0xed0cf4 testing::internal::HandleExceptionsInMethodIfSupported<>() @ 0xec9c4c testing::Test::Run() @ 0xec9d88 testing::TestInfo::Run() @ 0xec9e94 testing::TestCase::Run() @ 0xecb8f8 testing::internal::UnitTestImpl::RunAllTests() @ 0xecbc0c testing::UnitTest::Run() @ 0x92aaf8 main @ 0x7f9e04f8a0 __libc_start_main Aborted (core dumped) src/caffe/test/CMakeFiles/runtest.dir/build.make:57: recipe for target 'src/caffe/test/CMakeFiles/runtest' failed make[3]: *** [src/caffe/test/CMakeFiles/runtest] Error 134 CMakeFiles/Makefile2:328: recipe for target 'src/caffe/test/CMakeFiles/runtest.dir/all' failed make[2]: *** [src/caffe/test/CMakeFiles/runtest.dir/all] Error 2 CMakeFiles/Makefile2:335: recipe for target 'src/caffe/test/CMakeFiles/runtest.dir/rule' failed make[1]: *** [src/caffe/test/CMakeFiles/runtest.dir/rule] Error 2 Makefile:240: recipe for target 'runtest' failed make: *** [runtest] Error 2

[WingsofFAN]

@loulansuiye 请问您遇到的这个问题解决了？ 我也遇到了，同样的问题。。。

[yiran-THU]

I met the same problem

[a-wigand]

Same issue on the Nvidia AGX Xavier. Any updates?