serving icon indicating copy to clipboard operation
serving copied to clipboard

Add CentOS 7 Docker instructions

Open mohsenrezaeithe opened this issue 4 years ago • 3 comments

Fixes #1542

This is a starter kit to add support for building CentOS 7 based Docker images that'd be maintained in parallel to the Ubuntu based ones. There are some opportunities for refactoring or consolidation that can also be tackled that's not handled in the first version of this change (could possibly be done through follow up PRs).

Notes:

  • Using the current instructions, CentOS 7 is diverged from master starting at 1a36026, i.e. this is using 4de4c7b to successfully build.
  • The main difference between these CentOS 7 instructions and Ubuntu is Bazel 1.2.1 and Python 3.x.
  • I may have narrowed down the problem to the version of gcc used for libtool out of the box:
    ---> Package libtool.x86_64 0:2.4.2-22.el7_3 will be installed
    --> Processing Dependency: gcc = 4.8.5 for package: libtool-2.4.2-22.el7_3.x86_64
    

A couple of errors that I wasn't able to fully resolve on master, hence the "starter kit" label:

Extracting Bazel installation...
Starting local Bazel server and connecting to it...
WARNING: Download from https://mirror.bazel.build/github.com/tensorflow/tensorflow/archive/2a2c812ab2330c9aac33335f10679a346436acfb.tar.gz failed: class com.google.devtools.build.lib.bazel.repository.downloader.UnrecoverableHttpException GET returned 404 Not Found
DEBUG: Rule 'io_bazel_rules_docker' indicated that a canonical reproducible form can be obtained by modifying arguments shallow_since = "1556410077 -0400"
DEBUG: Call stack for the definition of repository 'io_bazel_rules_docker' which is a git_repository (rule definition at /root/.cache/bazel/_bazel_root/e53bbb0b0da4e26d24b415310219b953/external/bazel_tools/tools/build_defs/repo/git.bzl:195:18):
 - /root/.cache/bazel/_bazel_root/e53bbb0b0da4e26d24b415310219b953/external/bazel_toolchains/repositories/repositories.bzl:37:9
 - /tensorflow-serving/WORKSPACE:39:1
WARNING: Download from http://mirror.tensorflow.org/github.com/abseil/abseil-cpp/archive/36d37ab992038f52276ca66b9da80c1cf0f57dc2.tar.gz failed: class com.google.devtools.build.lib.bazel.repository.downloader.UnrecoverableHttpException GET returned 404 Not Found
WARNING: Download from https://storage.googleapis.com/mirror.tensorflow.org/bitbucket.org/eigen/eigen/get/66be6c76fc01.tar.gz failed: class com.google.devtools.build.lib.bazel.repository.downloader.UnrecoverableHttpException GET returned 404 Not Found
INFO: Analyzed target //tensorflow_serving/model_servers:tensorflow_model_server (208 packages loaded, 15593 targets configured).
INFO: Found 1 target...
ERROR: /root/.cache/bazel/_bazel_root/e53bbb0b0da4e26d24b415310219b953/external/com_google_protobuf/BUILD:406:1: Linking of rule '@com_google_protobuf//:protoc' failed (Exit 1)
bazel-out/host/bin/external/com_google_protobuf/_objs/protoc_lib/code_generator.o:code_generator.cc:function google::protobuf::compiler::ParseGeneratorParameter(std::string const&, std::vector<std::pair<std::string, std::string>, std::allocator<std::pair<std::string, std::string> > >*): error: undefined reference to 'std::__throw_out_of_range_fmt(char const*, ...)'
bazel-out/host/bin/external/com_google_protobuf/_objs/protoc_lib/command_line_interface.o:command_line_interface.cc:function google::protobuf::io::StringOutputStream::~StringOutputStream(): error: undefined reference to 'operator delete(void*, unsigned long)'
bazel-out/host/bin/external/com_google_protobuf/_objs/protoc_lib/command_line_interface.o:command_line_interface.cc:function google::protobuf::internal::ArenaStringPtr::CreateInstanceNoArena(std::string const*) [clone .isra.22] [clone .cold.459]: error: undefined reference to 'operator delete(void*, unsigned long)'
bazel-out/host/bin/external/com_google_protobuf/_objs/protoc_lib/command_line_interface.o:command_line_interface.cc:function non-virtual thunk to google::protobuf::compiler::CommandLineInterface::ErrorPrinter::~ErrorPrinter(): error: undefined reference to 'operator delete(void*, unsigned long)'
bazel-out/host/bin/external/com_google_protobuf/_objs/protoc_lib/command_line_interface.o:command_line_interface.cc:function non-virtual thunk to google::protobuf::compiler::CommandLineInterface::ErrorPrinter::~ErrorPrinter(): error: undefined reference to 'operator delete(void*, unsigned long)'
bazel-out/host/bin/external/com_google_protobuf/_objs/protoc_lib/command_line_interface.o:command_line_interface.cc:function google::protobuf::compiler::(anonymous namespace)::PluginName(std::string const&, std::string const&): error: undefined reference to 'std::__throw_out_of_range_fmt(char const*, ...)'
bazel-out/host/bin/external/com_google_protobuf/_objs/protoc_lib/command_line_interface.o:command_line_interface.cc:function google::protobuf::compiler::CommandLineInterface::InterpretArgument(std::string const&, std::string const&): error: undefined reference to 'std::__throw_out_of_range_fmt(char const*, ...)'
bazel-out/host/bin/external/com_google_protobuf/_objs/protoc_lib/command_line_interface.o:command_line_interface.cc:function google::protobuf::compiler::CommandLineInterface::MemoryOutputStream::~MemoryOutputStream(): error: undefined reference to 'std::__throw_out_of_range_fmt(char const*, ...)'
bazel-out/host/bin/external/com_google_protobuf/_objs/protoc_lib/java_helpers.o:java_helpers.cc:function google::protobuf::compiler::java::SortFieldsByNumber(google::protobuf::Descriptor const*) [clone .cold.236]: error: undefined reference to '__cxa_throw_bad_array_new_length'
bazel-out/host/bin/external/com_google_protobuf/_objs/protoc_lib/objectivec_message.o:objectivec_message.cc:function google::protobuf::compiler::objectivec::(anonymous namespace)::SortFieldsByNumber(google::protobuf::Descriptor const*) [clone .cold.289]: error: undefined reference to '__cxa_throw_bad_array_new_length'
bazel-out/host/bin/external/com_google_protobuf/_objs/protoc_lib/objectivec_message.o:objectivec_message.cc:function google::protobuf::compiler::objectivec::MessageGenerator::GenerateSource(google::protobuf::io::Printer*) [clone .cold.292]: error: undefined reference to '__cxa_throw_bad_array_new_length'
bazel-out/host/bin/external/com_google_protobuf/_objs/protobuf/dynamic_message.o:dynamic_message.cc:function google::protobuf::DynamicMessageFactory::GetPrototypeNoLock(google::protobuf::Descriptor const*) [clone .cold.103]: error: undefined reference to '__cxa_throw_bad_array_new_length'
bazel-out/host/bin/external/com_google_protobuf/_objs/protobuf_lite/extension_set.o:extension_set.cc:function google::protobuf::internal::ExtensionSet::~ExtensionSet(): error: undefined reference to 'operator delete[](void*, unsigned long)'
bazel-out/host/bin/external/com_google_protobuf/_objs/protobuf_lite/extension_set.o:extension_set.cc:function google::protobuf::internal::ExtensionSet::GrowCapacity(unsigned long): error: undefined reference to 'operator delete[](void*, unsigned long)'
collect2: error: ld returned 1 exit status
Target //tensorflow_serving/model_servers:tensorflow_model_server failed to build
INFO: Elapsed time: 1379.471s, Critical Path: 46.90s
INFO: 1571 processes: 1571 local.
FAILED: Build did NOT complete successfully

I've also arrived at #1563 using the Development Tools group or a subset of that, e.g.:

ERROR: /root/.cache/bazel/_bazel_root/e53bbb0b0da4e26d24b415310219b953/external/grpc/BUILD:439:1: Linking of rule '@grpc//:grpc_cpp_plugin' failed (Exit 1)

A couple of follow ups:

  1. CI/checks to build and publish an image for CentOS 7.
  2. Figure out which of the GCC libs is causing the compatibility issue with Bazel 1.2.1, e.g. devtoolset-9, gcc-4.9.x, etc.

mohsenrezaeithe avatar Mar 07 '20 04:03 mohsenrezaeithe

@mohsenrezaeithe I wonder do you have a Dockerfile to build TF Serving 2.3 under Centos7.8? thanks, I have an issue with the old libstdc++.so.6 version when running TF Serving 2.3 on Centos 7.8: $ ./tf23 --version ./tf23: /lib64/libm.so.6: version GLIBC_2.27' not found (required by ./tf23) ./tf23: /lib64/libm.so.6: version GLIBC_2.23' not found (required by ./tf23) ./tf23: /lib64/libstdc++.so.6: version GLIBCXX_3.4.21' not found (required by ./tf23) ./tf23: /lib64/libstdc++.so.6: version GLIBCXX_3.4.22' not found (required by ./tf23) ./tf23: /lib64/libstdc++.so.6: version CXXABI_1.3.9' not found (required by ./tf23) ./tf23: /lib64/libstdc++.so.6: version CXXABI_1.3.11' not found (required by ./tf23) ./tf23: /lib64/libstdc++.so.6: version GLIBCXX_3.4.20' not found (required by ./tf23) ./tf23: /lib64/libstdc++.so.6: version CXXABI_1.3.8' not found (required by ./tf23)

Any tips? thx

weitian avatar Nov 06 '20 01:11 weitian

@weitian my last successful build was on the included Dockerfile, centos:7.7.1908.

If you're using the Dockerfile from this PR, you'd probably have to tweak the gcc version (and then devtoolset, if still necessary) to get Bazel to build successfully on CentOS 7.

There shouldn't be any major differences between CentOS 7.7 → 7.8/7.9.

mohsenrezaeithe avatar Jan 05 '21 22:01 mohsenrezaeithe

Thanks @mohsenrezaeithe any update for 2.6?

joydeb28 avatar Aug 31 '21 14:08 joydeb28