CTranslate
CTranslate copied to clipboard
OpenNMT
Boost and Eigen 3.3 on Ubuntu 16.04 LTS
I have successfully installed both Boost and Eigen on Ubuntu 16.04 LTS:
Selecting previously unselected package libboost1.58-dev:amd64.
(Reading database ... 48598 files and directories currently installed.)
Preparing to unpack .../libboost1.58-dev_1.58.0+dfsg-5ubuntu3.1_amd64.deb ...
Unpacking libboost1.58-dev:amd64 (1.58.0+dfsg-5ubuntu3.1) ...
Selecting previously unselected package libboost-dev:amd64.
Preparing to unpack .../libboost-dev_1.58.0.1ubuntu1_amd64.deb ...
Unpacking libboost-dev:amd64 (1.58.0.1ubuntu1) ...
Selecting previously unselected package pkg-config.
Preparing to unpack .../pkg-config_0.29.1-0ubuntu1_amd64.deb ...
Unpacking pkg-config (0.29.1-0ubuntu1) ...
Selecting previously unselected package libeigen3-dev.
Preparing to unpack .../libeigen3-dev_3.3~beta1-2_all.deb ...
Unpacking libeigen3-dev (3.3~beta1-2) ...
Setting up libboost1.58-dev:amd64 (1.58.0+dfsg-5ubuntu3.1) ...
Setting up libboost-dev:amd64 (1.58.0.1ubuntu1) ...
Setting up pkg-config (0.29.1-0ubuntu1) ...
Setting up libeigen3-dev (3.3~beta1-2) ...
so libeigen3-dev (3.3~beta1-2) and libboost1.58-dev:amd64 (1.58.0+dfsg-5ubuntu3.1).
but when I compile
Step 24/29 : RUN git clone https://github.com/OpenNMT/CTranslate.git && cd CTranslate && git submodule update --init && mkdir build && cd build && cmake -DCMAKE_CXX_FLAGS="-march=native" .. && make
---> Running in bd9def302cf0
Cloning into 'CTranslate'...
Submodule 'lib/tokenizer' (https://github.com/OpenNMT/Tokenizer.git) registered for path 'lib/tokenizer'
Cloning into 'lib/tokenizer'...
Submodule path 'lib/tokenizer': checked out '0d90868e52fce98b1eda968ef23db3668b6f0db9'
-- The C compiler identification is GNU 5.4.0
-- The CXX compiler identification is GNU 5.4.0
-- Check for working C compiler: /usr/bin/cc
-- Check for working C compiler: /usr/bin/cc -- works
-- Detecting C compiler ABI info
-- Detecting C compiler ABI info - done
-- Detecting C compile features
-- Detecting C compile features - done
-- Check for working CXX compiler: /usr/bin/c++
-- Check for working CXX compiler: /usr/bin/c++ -- works
-- Detecting CXX compiler ABI info
-- Detecting CXX compiler ABI info - done
-- Detecting CXX compile features
-- Detecting CXX compile features - done
-- Build type: Release
-- Try OpenMP C flag = [-fopenmp]
-- Performing Test OpenMP_FLAG_DETECTED
-- Performing Test OpenMP_FLAG_DETECTED - Success
-- Try OpenMP CXX flag = [-fopenmp]
-- Performing Test OpenMP_FLAG_DETECTED
-- Performing Test OpenMP_FLAG_DETECTED - Success
-- Found OpenMP: -fopenmp
-- Could NOT find Boost
-- Eigen3 version 3.2.92 found in /usr/include/eigen3, but at least version 3.3 is required
the cmake does not detect boost or eigen3 in the system default folder.
Since I'm building on docker there are not previous packages installed, so it should find the ones I have installed via apt-get:
# OpenNMT dependencies: boot, eigen, intel mkl
RUN apt-get update && apt-get install -y \
libboost-dev \
libeigen3-dev -t xenial
where -t xenial is necessary to install the latest 3.3-beta.
[UPDATE]
I have found a solution for eigen3:
RUN curl http://security.ubuntu.com/ubuntu/pool/universe/e/eigen3/libeigen3-dev_3.3.2-1_all.deb -o libeigen3-dev_3.3.2-1_all.deb && dpkg -i libeigen3-dev_3.3.2-1_all.deb
so using dpkg solves the install dir issue:
-- Could NOT find Boost
-- Found Eigen3: /usr/include/eigen3 (Required is at least version "3.3")
-- Found CUDA: /usr/local/cuda (found suitable version "8.0", minimum required is "6.5")
-- Could NOT find MKL (missing: MKL_INCLUDE_DIR MKL_INTEL_LP64_LIBRARY MKL_GF_LP64_LIBRARY MKL_GNU_THREAD_LIBRARY MKL_CORE_LIBRARY)
The problem with boost is still present, so of course it still exits with error later on.
Reading the apt-get logs I can see that it installs some packages of boost but not all the recommended ones:
Reading state information...
The following additional packages will be installed:
libboost1.58-dev
Suggested packages:
libboost-doc libboost1.58-doc libboost-atomic1.58-dev
libboost-chrono1.58-dev libboost-context1.58-dev libboost-coroutine1.58-dev
libboost-date-time1.58-dev libboost-exception1.58-dev
libboost-filesystem1.58-dev libboost-graph1.58-dev
libboost-graph-parallel1.58-dev libboost-iostreams1.58-dev
libboost-locale1.58-dev libboost-log1.58-dev libboost-math1.58-dev
libboost-mpi1.58-dev libboost-mpi-python1.58-dev
libboost-program-options1.58-dev libboost-python1.58-dev
libboost-random1.58-dev libboost-regex1.58-dev
libboost-serialization1.58-dev libboost-signals1.58-dev
libboost-system1.58-dev libboost-test1.58-dev libboost-thread1.58-dev
libboost-timer1.58-dev libboost-wave1.58-dev libboost1.58-tools-dev
libmpfrc++-dev libntl-dev
The following NEW packages will be installed:
libboost-dev libboost1.58-dev pkg-config
0 upgraded, 3 newly installed, 0 to remove and 9 not upgraded.
Not sure if this can be the issue here.
[UPDATE]
Ok this is my last finding to solve the boost issue:
# OpenNMT dependencies: boot, eigen, intel mkl
RUN apt-get update && apt-get install -y \
libboost-dev \
libboost-all-dev \
pkg-config
It seems that libboost-all-dev will install all needed dependencies not only the boost library alone, so that now I came out with
-- Found Eigen3: /usr/include/eigen3 (Required is at least version "3.3")
...
-- Found CUDA: /usr/local/cuda (found suitable version "8.0", minimum required is "6.5")
-- Could NOT find MKL (missing: MKL_INCLUDE_DIR MKL_INTEL_LP64_LIBRARY MKL_GF_LP64_LIBRARY MKL_GNU_THREAD_LIBRARY MKL_CORE_LIBRARY)
-- Boost version: 1.58.0
I just have some complains about the architecture from nvcc when building
[ 27%] Building C object lib/TH/CMakeFiles/TH.dir/THGeneral.c.o
[ 30%] Building C object lib/TH/CMakeFiles/TH.dir/THFile.c.o
[ 33%] Building C object lib/TH/CMakeFiles/TH.dir/THDiskFile.c.o
[ 36%] Linking C shared library libTH.so
[ 36%] Built target TH
[ 39%] Building NVCC (Device) object CMakeFiles/onmt.dir/src/cuda/onmt_generated_Kernels.cu.o
nvcc warning : The 'compute_20', 'sm_20', and 'sm_21' architectures are deprecated, and may be removed in a future release (Use -Wno-deprecated-gpu-targets to suppress warning).
and the missing Intel MKL, but this is a separate thing.
If you want to make the nvcc warnings quiet, you can pass this flag to cmake:
-DCUDA_NVCC_FLAGS="-gencode=arch=compute_61,code=compute_61"
where 61 is the compute capability of your GPU.
Intel MKL is optional.
@guillaumekln thanks it make sense, by example when I have built the Dockerfile for darkflow I add to change this manually in the Cmake.
Somewhere I have used a command to detected the architecture automatically, I'm searching this right now...
@guillaumekln re-opening since I get
root@18b64b3f8bc0:/mxmlib/CTranslate/build/cli# echo "hello this is me" | ./translate --model /root/nmt/onmt_baseline_wmt15-all.en-de_epoch13_7.19_release.t7 --beam_size 5 --batch_size 1 -
Illegal instruction (core dumped)
where the built file was
root@18b64b3f8bc0:/mxmlib/CTranslate/build/cli# file translate
translate: ELF 64-bit LSB executable, x86-64, version 1 (GNU/Linux), dynamically linked, interpreter /lib64/ld-linux-x86-64.so.2, for GNU/Linux 2.6.32, BuildID[sha1]=9116d7fc8728d04bb935c6781256aa9c486e6358, not stripped
My cmake options were
cmake -DCMAKE_CXX_FLAGS="-march=native" -DEIGEN_ROOT="/usr/include/eigen3" .. &&
I have added the DWITH_OPENMP and it seems to work
cmake -DWITH_OPENMP=OFF -DCMAKE_CXX_FLAGS="-march=native" -DEIGEN_ROOT="/usr/include/eigen3" .. &&
@guillaumekln
0.0.90.190 ubuntu@ai-tools-nmt-i-04fc6623dcd3531f8:~$ g++ -v
Using built-in specs.
COLLECT_GCC=g++
COLLECT_LTO_WRAPPER=/usr/lib/gcc/x86_64-linux-gnu/5/lto-wrapper
Target: x86_64-linux-gnu
Configured with: ../src/configure -v --with-pkgversion='Ubuntu 5.4.0-6ubuntu1~16.04.4' --with-bugurl=file:///usr/share/doc/gcc-5/README.Bugs --enable-languages=c,ada,c++,java,go,d,fortran,objc,obj-c++ --prefix=/usr --program-suffix=-5 --enable-shared --enable-linker-build-id --libexecdir=/usr/lib --without-included-gettext --enable-threads=posix --libdir=/usr/lib --enable-nls --with-sysroot=/ --enable-clocale=gnu --enable-libstdcxx-debug --enable-libstdcxx-time=yes --with-default-libstdcxx-abi=new --enable-gnu-unique-object --disable-vtable-verify --enable-libmpx --enable-plugin --with-system-zlib --disable-browser-plugin --enable-java-awt=gtk --enable-gtk-cairo --with-java-home=/usr/lib/jvm/java-1.5.0-gcj-5-amd64/jre --enable-java-home --with-jvm-root-dir=/usr/lib/jvm/java-1.5.0-gcj-5-amd64 --with-jvm-jar-dir=/usr/lib/jvm-exports/java-1.5.0-gcj-5-amd64 --with-arch-directory=amd64 --with-ecj-jar=/usr/share/java/eclipse-ecj.jar --enable-objc-gc --enable-multiarch --disable-werror --with-arch-32=i686 --with-abi=m64 --with-multilib-list=m32,m64,mx32 --enable-multilib --with-tune=generic --enable-checking=release --build=x86_64-linux-gnu --host=x86_64-linux-gnu --target=x86_64-linux-gnu
Thread model: posix
gcc version 5.4.0 20160609 (Ubuntu 5.4.0-6ubuntu1~16.04.4)
Not sure about this.
- Are you compiling and testing on the same machine?
- Are you compiling with a compiler other than
g++?
@guillaumekln I think the problem were the linked libraries
root@ac0117818001:/mxmlib/CTranslate/build/cli# ldd translate
linux-vdso.so.1 => (0x00007fff2bbda000)
libonmt.so => /mxmlib/CTranslate/build/libonmt.so (0x00007f7cabc49000)
libboost_program_options.so.1.58.0 => /usr/lib/x86_64-linux-gnu/libboost_program_options.so.1.58.0 (0x00007f7cab9b9000)
libstdc++.so.6 => /usr/lib/x86_64-linux-gnu/libstdc++.so.6 (0x00007f7cab637000)
libgcc_s.so.1 => /lib/x86_64-linux-gnu/libgcc_s.so.1 (0x00007f7cab421000)
libc.so.6 => /lib/x86_64-linux-gnu/libc.so.6 (0x00007f7cab057000)
libpthread.so.0 => /lib/x86_64-linux-gnu/libpthread.so.0 (0x00007f7caae3a000)
libdl.so.2 => /lib/x86_64-linux-gnu/libdl.so.2 (0x00007f7caac36000)
librt.so.1 => /lib/x86_64-linux-gnu/librt.so.1 (0x00007f7caaa2d000)
libcublas.so.8.0 => /usr/local/cuda/lib64/libcublas.so.8.0 (0x00007f7ca7a12000)
libOpenNMTTokenizer.so => /mxmlib/CTranslate/build/lib/tokenizer/libOpenNMTTokenizer.so (0x00007f7ca77f9000)
libTH.so => /mxmlib/CTranslate/build/lib/TH/libTH.so (0x00007f7ca75f0000)
libm.so.6 => /lib/x86_64-linux-gnu/libm.so.6 (0x00007f7ca72e7000)
libgomp.so.1 => /usr/lib/x86_64-linux-gnu/libgomp.so.1 (0x00007f7ca70c5000)
/lib64/ld-linux-x86-64.so.2 (0x0000557438c78000)
So I think I was missing
libTH.so => /mxmlib/CTranslate/build/lib/TH/libTH.so (0x00007f7ca75f0000)
libOpenNMTTokenizer.so => /mxmlib/CTranslate/build/lib/tokenizer/libOpenNMTTokenizer.so (0x00007f7ca77f9000)
libonmt.so => /mxmlib/CTranslate/build/libonmt.so (0x00007f7cabc49000)
or libboost compiled on a different machine.
Question: no way to get rid of boost and eigen i.e. not using them?
@guillaumekln I wonder if it would be possibile to make the binaries standalone i.e. like fasttext does. This would be great and make everything more portable. Thank you!
Did you try to compile with static libraries?
cmake -DBUILD_SHARED_LIBS=OFF [...]
@guillaumekln thanks now I have only the boost dependency:
root@e195516272c0:/mxmlib/CTranslate/build/cli# ldd translate
linux-vdso.so.1 => (0x00007ffc8f1cc000)
libboost_program_options.so.1.58.0 => /usr/lib/x86_64-linux-gnu/libboost_program_options.so.1.58.0 (0x00007ffa56042000)
libpthread.so.0 => /lib/x86_64-linux-gnu/libpthread.so.0 (0x00007ffa55e24000)
libdl.so.2 => /lib/x86_64-linux-gnu/libdl.so.2 (0x00007ffa55c20000)
librt.so.1 => /lib/x86_64-linux-gnu/librt.so.1 (0x00007ffa55a18000)
libcublas.so.8.0 => /usr/local/cuda/lib64/libcublas.so.8.0 (0x00007ffa529fc000)
libstdc++.so.6 => /usr/lib/x86_64-linux-gnu/libstdc++.so.6 (0x00007ffa5267a000)
libm.so.6 => /lib/x86_64-linux-gnu/libm.so.6 (0x00007ffa52371000)
libgcc_s.so.1 => /lib/x86_64-linux-gnu/libgcc_s.so.1 (0x00007ffa5215a000)
libc.so.6 => /lib/x86_64-linux-gnu/libc.so.6 (0x00007ffa51d91000)
/lib64/ld-linux-x86-64.so.2 (0x00005637afd00000)
boost is only used for managing command line options. It should be easy to write your own translation client without this dependency.
See cli/translate.cc.