How to prepare a pre-built library to run the end-to-end model server based on TRTIS

[woskii]

Hi there, I want to deploy a lightseq model on TRT, and the inference readme says that a custom dynamic link library is required. I wonder how can I build such a .so file to deploy a model with LSHFTransformerEncoderLayer? Thanks in advance!

[neopro12]

You can build the dynamic link library from source. A tutorial is available here: https://github.com/bytedance/lightseq/blob/master/docs/inference/build.md

[woskii]

When I build from source, there is an error:

[ 75%] Linking CUDA device code CMakeFiles/lightseq.dir/cmake_device_link.o [ 76%] Linking CXX shared module lightseq/inference.cpython-36m-x86_64-linux-gnu.so /opt/rh/devtoolset-8/root/usr/libexec/gcc/x86_64-redhat-linux/8/ld: cannot open output file lightseq/inference.cpython-36m-x86_64-linux-gnu.so: No such file or directory collect2: error: ld returned 1 exit status make[2]: *** [lightseq/inference/pywrapper/lightseq/inference.cpython-36m-x86_64-linux-gnu.so] Error 1 make[1]: *** [lightseq/inference/pywrapper/CMakeFiles/lightseq.dir/all] Error 2 make[1]: *** Waiting for unfinished jobs.... [ 77%] Building CXX object lightseq/inference/server/CMakeFiles/decoder_generate_server.dir/model_config.pb.cc.o [ 79%] Linking CUDA device code CMakeFiles/transformer_decoder_example.dir/cmake_device_link.o [ 80%] Linking CUDA device code CMakeFiles/transformer_generate_example.dir/cmake_device_link.o [ 82%] Linking CUDA device code CMakeFiles/transformer_example.dir/cmake_device_link.o [ 83%] Linking CUDA device code CMakeFiles/transformer_generate_server.dir/cmake_device_link.o [ 85%] Linking CXX executable transformer_decoder_example [ 86%] Linking CUDA device code CMakeFiles/transformer_server.dir/cmake_device_link.o [ 88%] Linking CXX executable transformer_generate_example [ 89%] Linking CXX executable transformer_example [ 91%] Linking CUDA device code CMakeFiles/decoder_generate_server.dir/cmake_device_link.o [ 94%] Linking CXX shared library libtransformer_generate_server.so [ 94%] Linking CXX shared library libtransformer_server.so [ 94%] Built target transformer_decoder_example [ 94%] Built target transformer_generate_example [ 95%] Linking CXX shared library libdecoder_generate_server.so [ 95%] Built target transformer_example [ 97%] Built target transformer_generate_server [ 98%] Built target transformer_server [100%] Built target decoder_generate_server make: *** [all] Error 2

It seems that inference.cpython-36m-x86_64-linux-gnu.so is not generated.

[Taka152]

inference.cpython-36m-x86_64-linux-gnu.so maybe a path problem and it's needed by python module lightseq.inference. If you just want to use trtis, you can ignore this error and the so files in server/ are all your need.

[ghtwht]

how to choose the fixed version of glibc to compile the libtransformer_server.so?

[Taka152]

@ghtwht Better to update the OS, or check this.