Problem on Building Serving Image with Multiple Models with Config

[gabbygab1233]

I don’t have a problem when creating my own serving image in docker using 1 model but when I try to build a serving image with multiple models it doesn’t work.

Here’s the command I used:

Create Serving Image

docker run -d --name serving_base tensorflow/serving

Copy SavedModel

docker cp /home/model1 serving_base:/models/model1
docker cp /home/model2 serving_base:/models/model2
docker cp /home/model.config serving_base:/models/model.config

Commit

docker commit serving_base acvision

Stop Serving Base

docker kill serving base

Checking Docker image

docker run --rm --name serve_acvision -p8502:8501 -p8503:8501 -p8504:8501 -d acvision

Testing the Server

[pindinagesh]

@gabbygab1233

Could you please refer the similar issues link1 and link2, and let us know if it helps. Thanks

[gabbygab1233]

@gabbygab1233

Could you please refer the similar issues link1 and link2, and let us know if it helps. Thanks

I can run multiple models using tensorflow/serving

docker run -p 8500:8500 \
-p 8501:8501 \
--mount type=bind,source=/tmp/models,target=/models/my_model \
--mount type=bind,source=/tmp/models,target=/models/another_model \
--mount type=bind,source=/tmp/model_config,\
target=/models/model_config \
-e MODEL_NAME=my_model \
-t tensorflow/serving \
--model_config_file=/models/model_config

But I want to create my own serving image I follow this tutorial

So this is my serving image("AcVision") I'll copy all the models and config files

So now instead of using tensorflow/serving image I'm going to use my own image which is "AcVision"

I try to run my image using this command

docker run --rm --name serve_acvision -p8502:8501 -p8503:8501 -p8504:8501 -d acvision

So my question is do I still need to mount my models and config files on docker command even though I copy all my models and config files on my own serving image?

[singhniraj08]

@gabbygab1233,

There is no need to mount the models and config files on docker command if you copy your models and configs files to your own serving image.

I tried the below method to create my own serving image and was able to serve my model using own serving image. I also tried with multiple models but it didn't worked for me. Creating your own image works for single model only.

Thank you!

docker pull tensorflow/serving
docker run -d --name serving_base tensorflow/serving
docker cp "$(pwd)/models/testdata/saved_model_half_plus_two_cpu" serving_base:/models/saved_model_half_plus_two_cpu
docker commit --change "ENV MODEL_NAME saved_model_half_plus_two_cpu" serving_base tensorflow_serving:2.11.0
docker kill serving_base
docker run -p 8501:8501 -t tensorflow_serving:2.11.0

[singhniraj08]

Closing this due to inactivity. Please take a look into the answers provided above, feel free to reopen and post your comments(if you still have queries on this). Thank you!