Deploy libtorch model with gRPC (Ubuntu 18.04)

this project shows how to deploy a resnet libtorch model using gRPC.

1. Serving with docker

The easiest and most reliable way to deploy the resnet libtorch model is to use docker, make sure you have already install docker and run the following commands.

# in project root dir
cd docker
# build the image from Dockerfile
[sudo] docker build --pull -t resnet-libtorch-serving -f libtorch_cpu_Dockerfile .
# create docker container and provide service
[sudo] sudo docker run -p 50051:50051 --name=resnet_service -d -it resnet-libtorch-serving /bin/bash -c './resnet_server'

Now you can create a client and send a request to the resnet server, here I use the python client to do the demonstration.

# in project root dir
cd python
# make sure grpcio-tools has installed in your python environment
python -m grpc_tools.protoc -I../protos --python_out=. --grpc_python_out=. ../protos/example.proto
python resnet_client.py  # image category: image_category_num

2. Serving without docker

If you want to get all the things without docker, You need to follow the instructions below.

Since there is no prebuilt gRPC package in C++, it must be compiled and installed from the source code.

Compile and install gRPC

Install prerequisites

• necessary packages

sudo apt-get update && sudo apt-get upgrade
sudo apt-get install build-essential autoconf libtool pkg-config libgflags-dev libgtest-dev clang libc++-dev libssl-dev cmake python3-distutils vim tree git curl

• go language

wget https://storage.googleapis.com/golang/go1.12.9.linux-amd64.tar.gz
sudo tar -C /usr/local -xzf go*linux-amd64.tar.gz
echo 'export PATH=$PATH:/usr/local/go/bin' >> $HOME/.bashrc
echo 'export LD_LIBRARY_PATH=$LD_LIBRARY_PATH:/usr/local/lib' >> $HOME/.bashrc
source $HOME/.bashrc

• download gRPC project

# in $HOME folder
git clone -b v1.23.0 https://github.com/grpc/grpc.git
cd grpc
git submodule update --init

The build process is a bit tricky, If you follow the build instructions given by the grpc's readme, it won't automatically generate *targets.cmake files which is useful when you use cmake to link grpc into your project and you will receive the error:

include could not find load file:

  /usr/local/lib/cmake/grpc/gRPCTargets.cmake

The current workaround is to build zlib, cares, protobuf at first, then tell cmake to use these installed packages when building grpc, then grpc will generate all the files as expected.

Install zlib

GPRC_COMPILE_ROOT_PATH=$(pwd)
cd $GPRC_COMPILE_ROOT_PATH/third_party/zlib
mkdir build && cd build
cmake ..
make -j $(nproc)
sudo make install

Install cares

cd $GPRC_COMPILE_ROOT_PATH/third_party/cares/cares
mkdir build && cd build
cmake ..
make -j $(nproc)
sudo make install

Install protobuf

cd $GPRC_COMPILE_ROOT_PATH/third_party/protobuf/cmake
mkdir build && cd build
cmake -Dprotobuf_BUILD_TESTS=OFF ..
make -j $(nproc)
sudo make install

Install gRPC

cd $GPRC_COMPILE_ROOT_PATH
mkdir build && cd build
cmake -DgRPC_INSTALL=ON -DgRPC_ZLIB_PROVIDER=package -DgRPC_CARES_PROVIDER=package -DgRPC_PROTOBUF_PROVIDER=package -DgRPC_SSL_PROVIDER=package ..
make -j $(nproc)
sudo make install
sudo ldconfig

Install libtorch library

cd $HOME
wget https://download.pytorch.org/libtorch/cpu/libtorch-cxx11-abi-shared-with-deps-1.2.0.zip
unzip libtorch*.zip

Build gRPC server of resnet libtorch model

Generate resnet libtorch saved file from pytorch

curl -O https://bootstrap.pypa.io/get-pip.py
# ubuntu 18.04 python3.6.8
sudo python3 get-pip.py
pip config set global.index-url https://pypi.tuna.tsinghua.edu.cn/simple
# install pytorch package
sudo pip install torch==1.2.0+cpu torchvision==0.4.0+cpu -f https://download.pytorch.org/whl/torch_stable.html

git clone https://github.com/Peter-Chou/libtorch_grpc_serving.git
cd libtorch_grpc_serving
python3 get_resnet_libtorch_save.py

Generate server structure code by protoc

• generate C++ server side code

# in libtorch_grpc_serving root dir
PROTO_SRC_DIR=./protos
## generate C++ server side code
protoc -I $PROTO_SRC_DIR --grpc_out=./protos --plugin=protoc-gen-grpc=`which grpc_cpp_plugin` $PROTO_SRC_DIR/example.proto
## generate C++ message classes needed by server
protoc -I $PROTO_SRC_DIR --cpp_out=./protos $PROTO_SRC_DIR/example.proto 

• generate python client side code

# in libtorch_grpc_serving root dir
sudo pip install grpcio-tools
python3 -m grpc_tools.protoc -I$PROTO_SRC_DIR --python_out=./python --grpc_python_out=./python $PROTO_SRC_DIR/example.proto

Build the project

# in libtorch_grpc_serving root dir
mkdir build && cd build
cmake -DCMAKE_PREFIX_PATH=$HOME/libtorch ..
make -j $(nproc)

Start resnet server

# in libtorch_grpc_serving root dir
./build/bin/resnet_server &  # Server listening on 0.0.0.0:50051

Client request from server

# in libtorch_grpc_serving root dir
python3 python/resnet_client.py  # image category: image_category_num