deepdetect icon indicating copy to clipboard operation
deepdetect copied to clipboard
verified publisher icon jolibrain

Building Ubuntu 18.04 Caffe+TF error

Open cchadowitz opened this issue 6 years ago • 4 comments

Configuration

  • Building on Ubuntu 18.04 in docker, CPU only, with Caffe+TF enabled.
  • Deepdetect v0.9.5

Steps to reproduce the problem:

cmake .. -DUSE_TF=ON -DUSE_TF_CPU_ONLY=ON -DUSE_DLIB=ON  -DUSE_DLIB_CPU_ONLY=ON -DUSE_CAFFE_CPU_ONLY=ON
make

Build error

/opt/deepdetect/build/tensorflow_cc/src/tensorflow_cc/tensorflow_cc/build/tensorflow/tensorflow/core/framework/device_base.h: In member function 'virtual tensorflow::Allocator* tensorflow::DeviceBase::GetAllocator(tensorflow::AllocatorAttributes)':
/opt/deepdetect/build/tensorflow_cc/src/tensorflow_cc/tensorflow_cc/build/tensorflow/tensorflow/core/framework/device_base.h:184:14: error: invalid initialization of reference of type 'const string& {aka const std::__cxx11::basic_string<char>&}' from expression of type 'const int'
     LOG(FATAL) << "GetAllocator() is not implemented.";
              ^
In file included from /opt/deepdetect/build/caffe_dd/src/caffe_dd/include/caffe/common.hpp:7:0,
                 from /opt/deepdetect/build/caffe_dd/src/caffe_dd/include/caffe/blob.hpp:8,
                 from /opt/deepdetect/build/caffe_dd/src/caffe_dd/include/caffe/caffe.hpp:7,
                 from /opt/deepdetect/src/backends/caffe/caffeinputconns.h:30,
                 from /opt/deepdetect/src/imginputfileconn.h:735,
                 from /opt/deepdetect/src/backends/tf/tflib.cc:24:
/opt/deepdetect/build/caffe_dd/src/caffe_dd/include/caffe/llogging.h:199:20: note: in passing argument 1 of 'CaffeLogger LOG(const string&)'
 inline CaffeLogger LOG(const std::string &severity)
                    ^~~
In file included from /opt/deepdetect/build/tensorflow_cc/src/tensorflow_cc/tensorflow_cc/build/tensorflow/tensorflow/core/framework/op_kernel.h:28:0,
                 from /opt/deepdetect/build/tensorflow_cc/src/tensorflow_cc/tensorflow_cc/build/tensorflow/tensorflow/core/framework/function.h:26,
                 from /opt/deepdetect/build/tensorflow_cc/src/tensorflow_cc/tensorflow_cc/build/tensorflow/tensorflow/core/graph/graph.h:44,
                 from /opt/deepdetect/build/tensorflow_cc/src/tensorflow_cc/tensorflow_cc/build/tensorflow/tensorflow/cc/framework/ops.h:23,
                 from /opt/deepdetect/build/tensorflow_cc/src/tensorflow_cc/tensorflow_cc/build/tensorflow/tensorflow/cc/ops/const_op.h:19,
                 from /opt/deepdetect/src/backends/tf/tfinputconns.h:29,
                 from /opt/deepdetect/src/imginputfileconn.h:739,
                 from /opt/deepdetect/src/backends/tf/tflib.cc:24:
/opt/deepdetect/build/tensorflow_cc/src/tensorflow_cc/tensorflow_cc/build/tensorflow/tensorflow/core/framework/device_base.h: In member function 'virtual tensorflow::Allocator* tensorflow::DeviceBase::GetScopedAllocator(tensorflow::AllocatorAttributes, tensorflow::int64)':
/opt/deepdetect/build/tensorflow_cc/src/tensorflow_cc/tensorflow_cc/build/tensorflow/tensorflow/core/framework/device_base.h:199:14: error: invalid initialization of reference of type 'const string& {aka const std::__cxx11::basic_string<char>&}' from expression of type 'const int'
     LOG(FATAL) << "Device does not implement GetScopedAllocator()";
              ^
In file included from /opt/deepdetect/build/caffe_dd/src/caffe_dd/include/caffe/common.hpp:7:0,
                 from /opt/deepdetect/build/caffe_dd/src/caffe_dd/include/caffe/blob.hpp:8,
                 from /opt/deepdetect/build/caffe_dd/src/caffe_dd/include/caffe/caffe.hpp:7,
                 from /opt/deepdetect/src/backends/caffe/caffeinputconns.h:30,
                 from /opt/deepdetect/src/imginputfileconn.h:735,
                 from /opt/deepdetect/src/backends/tf/tflib.cc:24:
/opt/deepdetect/build/caffe_dd/src/caffe_dd/include/caffe/llogging.h:199:20: note: in passing argument 1 of 'CaffeLogger LOG(const string&)'
 inline CaffeLogger LOG(const std::string &severity)

This seems to be an issue when building with Caffe+TF enabled, where the TF version is v0.15.0 (and bazel v0.24.0). When following the same steps with TF v0.13.1 and bazel v0.21.0, there is no error.

I'm not sure if any of the changes in #684 are related. I see that the travis builds worked, but I'm not sure if Caffe+TF are built in docker in those builds the same way.

These are the differences in https://github.com/FloopCZ/tensorflow_cc between v0.13.1 and v0.15.0: https://github.com/FloopCZ/tensorflow_cc/compare/v1.13.1...v1.15.0

cchadowitz avatar Jan 08 '20 19:01 cchadowitz

I'll try to reproduce. However, there's no doubt where this comes from: most DL libraries used to rely on glog for logging from C++. glog not being flexible and fast enough, many libs have replaced it with customized versions (e.g. pytorch and mxnet did that) while keeping a seemingly compatible set of macros in order to not have to change all log calls within a large codebase.

In DD, we have a version of this that actually manages to replace the logging lib from most of the underlying backends! Most of the time we have to patch the backend (e.g. pytorch) so that it can build against our 'overlord' log lib and glog-like macros.

I believe that what you are reporting is TF showing incompatibilities with our main logging header (CaffeLogger).

Looking at how TF did modify it's LOG macros from 0.13 to 0.15 would help I think.

beniz avatar Jan 09 '20 06:01 beniz

That makes sense, I wasn't sure if CaffeLogger was original Caffe or from you :)

I tried to dig through and figure out how TF's LOG macros changed between 0.13.1 and 0.15.0, but couldn't find anything concrete.... I'll try to take another pass when I get the chance.

cchadowitz avatar Jan 09 '20 18:01 cchadowitz

The builds I did try did go through fine, on master branch. Though not sure it's using tf 1.15. Could you share a dockerfile to reproduce from ?

beniz avatar Jan 09 '20 20:01 beniz

Interesting.... yeah, here's the relevant chunk of the dockerfile I'm using:

FROM ubuntu:18.04 as intermediate

RUN echo 'building CPU DeepDetect image'

ENV DEBIAN_FRONTEND=noninteractive

COPY libopenblas-base_0.3.5+ds-2_amd64.deb libopenblas-dev_0.3.5+ds-2_amd64.deb /tmp/

RUN apt-get update && apt-get install -y --no-install-recommends \
  git \
  cmake \
  automake \
  build-essential \
  openjdk-8-jdk \
  pkg-config \
  zip \
  g++ \
  zlib1g-dev \
  libarchive-dev \
  libgoogle-glog-dev \
  libgflags-dev \
  libeigen3-dev \
  libopencv-dev \
  libboost-all-dev \
  libboost-iostreams-dev \
  libcurlpp-dev \
  libcurl4-openssl-dev \
  protobuf-compiler \
  libhdf5-dev \
  libprotobuf-dev \
  libleveldb-dev \
  libsnappy-dev \
  libssl-dev \
  openssl \
  liblmdb-dev \
  libutfcpp-dev \
  wget \
  autoconf \
  libtool-bin \
  python-numpy \
  swig \
  python-dev \
  python-wheel \
  unzip \
  libgoogle-perftools-dev \
  screen \
  vim \
  strace \
  curl \
  libspdlog-dev \
  bash-completion \
  /tmp/libopenblas-base_0.3.5+ds-2_amd64.deb \
  /tmp/libopenblas-dev_0.3.5+ds-2_amd64.deb \
  ca-certificates && \
  wget -O /tmp/bazel.deb https://github.com/bazelbuild/bazel/releases/download/0.24.1/bazel_0.24.1-linux-x86_64.deb && \
  dpkg -i /tmp/bazel.deb && \
  apt-get clean && \
  rm -rf /var/lib/apt/lists/* /tmp/* /var/tmp/*

WORKDIR /opt

COPY cpp-netlib-0.11.2-final.tar.gz /opt/

RUN tar xf cpp-netlib-0.11.2-final.tar.gz && \
    cd cpp-netlib-cpp-netlib-0.11.2-final && \
    mkdir build && cd build && cmake .. && \
    make && make install

COPY cmakelists.patch /opt/

RUN git clone https://github.com/jolibrain/deepdetect.git && \
    cd deepdetect && \
    git apply --ignore-whitespace --ignore-space-change /opt/cmakelists.patch && \
    mkdir build && \
    cd build && \
    export PYTHON_BIN_PATH=/usr/bin/python && \
    export PYTHON_LIB_PATH=/usr/local/lib/python2.7/dist-packages && \
    export TF_NEED_CUDA=0 && \
    export TF_NEED_GCP=0 && \
    export TF_NEED_HDFS=0 && \
    export TF_NEED_OPENCL=0 && \
    export TF_NEED_JEMALLOC=1 && \
    export TF_NEED_VERBS=0 && \
    export TF_NEED_MKL=1 && \
    export TF_DOWNLOAD_MKL=1 && \
    export TF_NEED_MPI=0 && \
    export TF_ENABLE_XLA=1 && \
    export TF_NEED_S3=0 && \
    export TF_NEED_GDR=0 && \
    export TF_CUDA_CLANG=0 && \
    export CC_OPT_FLAGS="-march=westmere" && \
    cmake .. -DUSE_TF=ON -DUSE_TF_CPU_ONLY=ON -DUSE_DLIB=ON  -DUSE_DLIB_CPU_ONLY=ON -DUSE_CAFFE_CPU_ONLY=ON && \
    make && \
    mkdir /opt/dede && \
    cp main/dede caffe_dd/src/caffe_dd/build/lib/libcaffe.so.1.0.0-rc3 protobuf/lib/libprotobuf.so.17 tensorflow_cc/src/tensorflow_cc/tensorflow_cc/build/tensorflow/bazel-out/k8-opt/bin/tensorflow/libtensorflow_cc.so /opt/dede && \
    cd /opt && \
    rm -rf deepdetect /opt/cmakelists.patch /opt/caffe_dd.tar.gz /opt/tensorflow_cc_dd_tar.gz


FROM ubuntu:18.04

COPY --from=intermediate /opt/dede /opt/dede

RUN ln -sf /dev/stdout /var/log/deepdetect.log && \
    ln -sf /dev/stderr /var/log/deepdetect.log

COPY libopenblas-base_0.3.5+ds-2_amd64.deb /tmp/libopenblas-base_0.3.5+ds-2_amd64.deb

RUN apt-get update && apt-get install -y --no-install-recommends \
    /tmp/libopenblas-base_0.3.5+ds-2_amd64.deb \
    libarchive13 \
    libgflags2.2 \
    libhdf5-100 \
    libhdf5-cpp-100 \
    libcurlpp0 \
    libcurl4 \
    libboost-iostreams1.65.1 \
    libboost-thread1.65.1 \
    libboost-filesystem1.65.1 \
    libboost-regex1.65.1 \
    libboost-system1.65.1 \
    libopencv-calib3d3.2 \
    libopencv-core3.2 \
    libopencv-highgui3.2 \
    libopencv-imgproc3.2 \
    libopencv-ml3.2 \
    libopencv-objdetect3.2 \
    libgoogle-glog0v5 \
    libleveldb1v5 \
    liblmdb0 \
    wget \
    unzip \
    screen \
    vim \
    strace \
    curl \
    bash-completion \
    ca-certificates && \
    apt-get clean && \
    rm -rf /var/lib/apt/lists/* /tmp/* /var/tmp/*

You will need the two .deb files referenced since it's on Ubuntu 18.04. The patch file simply tweaks the CMakeLists.txt like so (enables certain CPU optimizations in dlib, and pins the TF version):

diff --git a/CMakeLists.txt b/CMakeLists.txt
index 5c7efc8..1afea70 100644
--- a/CMakeLists.txt
+++ b/CMakeLists.txt
@@ -239,7 +239,7 @@ if (USE_DLIB)
           PREFIX dlib
           INSTALL_DIR ${CMAKE_BINARY_DIR}
           URL https://github.com/davisking/dlib/archive/${DLIB_RELEASE_VERSION}.tar.gz
-          CONFIGURE_COMMAND cd dlib && mkdir -p build && cd build && cmake .. -DDLIB_NO_GUI_SUPPORT=ON -DDLIB_USE_CUDA=ON -DCMAKE_INSTALL_PREFIX=${CMAKE_BINARY_DIR}/dlib/build && cmake --build . --config Release --target install
+          CONFIGURE_COMMAND cd dlib && mkdir -p build && cd build && cmake .. -DDLIB_NO_GUI_SUPPORT=ON -DUSE_AVX_INSTRUCTIONS=OFF -DUSE_SSE2_INSTRUCTIONS=ON -DUSE_SSE4_INSTRUCTIONS=ON -DDLIB_USE_CUDA=ON -DCMAKE_INSTALL_PREFIX=${CMAKE_BINARY_DIR}/dlib/build && cmake --build . --config Release --target install
           BUILD_COMMAND ""
           INSTALL_COMMAND ""
           BUILD_IN_SOURCE 1
@@ -250,7 +250,7 @@ if (USE_DLIB)
             PREFIX dlib
             INSTALL_DIR ${CMAKE_BINARY_DIR}
             URL https://github.com/davisking/dlib/archive/${DLIB_RELEASE_VERSION}.tar.gz
-            CONFIGURE_COMMAND cd dlib && mkdir -p build && cd build && cmake .. -DDLIB_NO_GUI_SUPPORT=ON -DDLIB_USE_CUDA=OFF -DCMAKE_INSTALL_PREFIX=${CMAKE_BINARY_DIR}/dlib/build && cmake --build . --config Release --target install
+            CONFIGURE_COMMAND cd dlib && mkdir -p build && cd build && cmake .. -DDLIB_NO_GUI_SUPPORT=ON -DUSE_AVX_INSTRUCTIONS=OFF -DUSE_SSE2_INSTRUCTIONS=ON -DUSE_SSE4_INSTRUCTIONS=ON -DDLIB_USE_CUDA=OFF -DCMAKE_INSTALL_PREFIX=${CMAKE_BINARY_DIR}/dlib/build && cmake --build . --config Release --target install
             BUILD_COMMAND ""
             INSTALL_COMMAND ""
             BUILD_IN_SOURCE 1
@@ -432,7 +432,7 @@ if (USE_TF)
       tensorflow_cc
       PREFIX tensorflow_cc
       INSTALL_DIR ${CMAKE_BINARY_DIR}
-      DOWNLOAD_COMMAND git clone https://github.com/FloopCZ/tensorflow_cc.git
+      URL https://github.com/FloopCZ/tensorflow_cc/archive/v1.15.0.tar.gz
       CONFIGURE_COMMAND cd tensorflow_cc && mkdir -p build && cd build && cmake -DTENSORFLOW_STATIC=OFF -DTENSORFLOW_SHARED=ON .. && make && ln -s ${CMAKE_BINARY_DIR}/tensorflow_cc/src/tensorflow_cc/tensorflow_cc/build/tensorflow/tensorflow/contrib/makefile/gen/protobuf ${CMAKE_BINARY_DIR}/protobuf
       BUILD_COMMAND ""
       INSTALL_COMMAND ""
@@ -444,7 +444,7 @@ if (USE_TF)
       tensorflow_cc
       PREFIX tensorflow_cc
       INSTALL_DIR ${CMAKE_BINARY_DIR}
-      DOWNLOAD_COMMAND git clone https://github.com/FloopCZ/tensorflow_cc.git
+      URL https://github.com/FloopCZ/tensorflow_cc/archive/v1.15.0.tar.gz
       CONFIGURE_COMMAND cd tensorflow_cc && mkdir -p build && cd build && cmake -DTENSORFLOW_STATIC=OFF -DTENSORFLOW_SHARED=ON -DALLOW_CUDA=OFF .. && make && ln -s ${CMAKE_BINARY_DIR}/tensorflow_cc/src/tensorflow_cc/tensorflow_cc/build/tensorflow/tensorflow/contrib/makefile/gen/protobuf ${CMAKE_BINARY_DIR}/protobuf
       BUILD_COMMAND ""
       INSTALL_COMMAND ""

cchadowitz avatar Jan 09 '20 20:01 cchadowitz