serve
serve copied to clipboard
Published
20 hours ago •
pytorch
pytorch
Is possible to serve 2 model with 1 handler?
Hi! Thank you for your great works in advance.
I just want to serve 2 model with 1 handler for ensemble.
However, the outputs of models are different when I operate them simultaneously.
For example, there is shared input I, and I have 2 models H and G where the expected outputs are H(I) = A , G(I) = B
When I serve 2 models seperately, they give me the constant output H(I)=A and G(I)=B.
But, the outpus of models are always different even though the seed is fixed when I serve them in 1 handler simultaneously.
Here is my environment details.
> Name Version Build Channel
> _libgcc_mutex 0.1 main
> _openmp_mutex 5.1 1_gnu
> aniso8601 9.0.1 pypi_0 pypi
> ansi2html 1.8.0 pypi_0 pypi
> arrow 1.2.3 pypi_0 pypi
> beautifulsoup4 4.12.2 pypi_0 pypi
> ca-certificates 2023.08.22 h06a4308_0
> certifi 2022.12.7 py37h06a4308_0
> chardet 5.2.0 pypi_0 pypi
> charset-normalizer 3.3.0 pypi_0 pypi
> click 8.1.7 pypi_0 pypi
> cycler 0.11.0 pypi_0 pypi
> enum-compat 0.0.3 pypi_0 pypi
> filelock 3.12.2 pypi_0 pypi
> flask 2.2.5 pypi_0 pypi
> flask-restful 0.3.10 pypi_0 pypi
> fonttools 4.38.0 pypi_0 pypi
> gdown 4.7.1 pypi_0 pypi
> h5py 3.8.0 pypi_0 pypi
> hloc 1.3 dev_0 <develop>
> idna 3.4 pypi_0 pypi
> importlib-metadata 6.7.0 pypi_0 pypi
> itsdangerous 2.1.2 pypi_0 pypi
> jinja2 3.1.2 pypi_0 pypi
> kiwisolver 1.4.5 pypi_0 pypi
> kornia 0.6.12 pypi_0 pypi
> ld_impl_linux-64 2.38 h1181459_1
> libffi 3.4.4 h6a678d5_0
> libgcc-ng 11.2.0 h1234567_1
> libgomp 11.2.0 h1234567_1
> libstdcxx-ng 11.2.0 h1234567_1
> markupsafe 2.1.3 pypi_0 pypi
> matplotlib 3.5.3 pypi_0 pypi
> ncurses 6.4 h6a678d5_0
> numpy 1.21.6 pypi_0 pypi
> nvgpu 0.10.0 pypi_0 pypi
> nvidia-cublas-cu11 11.10.3.66 pypi_0 pypi
> nvidia-cuda-nvrtc-cu11 11.7.99 pypi_0 pypi
> nvidia-cuda-runtime-cu11 11.7.99 pypi_0 pypi
> nvidia-cudnn-cu11 8.5.0.96 pypi_0 pypi
> opencv-python 4.8.1.78 pypi_0 pypi
> openssl 1.1.1w h7f8727e_0
> packaging 23.2 pypi_0 pypi
> pandas 1.3.5 pypi_0 pypi
> pillow 9.5.0 pypi_0 pypi
> pip 22.3.1 py37h06a4308_0
> plotly 5.17.0 pypi_0 pypi
> psutil 5.9.6 pypi_0 pypi
> pycolmap 0.4.0 pypi_0 pypi
> pycryptodomex 3.19.0 pypi_0 pypi
> pynvml 11.5.0 pypi_0 pypi
> pyparsing 3.1.1 pypi_0 pypi
> pyproj 3.2.1 pypi_0 pypi
> pysocks 1.7.1 pypi_0 pypi
> python 3.7.16 h7a1cb2a_0
> python-dateutil 2.8.2 pypi_0 pypi
> pytz 2023.3.post1 pypi_0 pypi
> pyyaml 6.0.1 pypi_0 pypi
> readline 8.2 h5eee18b_0
> requests 2.31.0 pypi_0 pypi
> scipy 1.7.3 pypi_0 pypi
> setuptools 65.6.3 py37h06a4308_0
> six 1.16.0 pypi_0 pypi
> soupsieve 2.4.1 pypi_0 pypi
> sqlite 3.41.2 h5eee18b_0
> tabulate 0.9.0 pypi_0 pypi
> tenacity 8.2.3 pypi_0 pypi
> termcolor 2.3.0 pypi_0 pypi
> tk 8.6.12 h1ccaba5_0
> torch 1.13.1 pypi_0 pypi
> torch-model-archiver 0.9.0 pypi_0 pypi
> torch-workflow-archiver 0.2.11 pypi_0 pypi
> torchaudio 0.13.1 pypi_0 pypi
> torchserve 0.9.0 pypi_0 pypi
> torchvision 0.14.1 pypi_0 pypi
> tqdm 4.66.1 pypi_0 pypi
> typing-extensions 4.7.1 pypi_0 pypi
> urllib3 2.0.6 pypi_0 pypi
> werkzeug 2.2.3 pypi_0 pypi
> wheel 0.38.4 py37h06a4308_0
> xz 5.4.2 h5eee18b_0
> zipp 3.15.0 pypi_0 pypi
> zlib 1.2.13 h5eee18b_0
The inference function in my custom handler.
The input and 2 models are set requires_grad=False
def inference(self, model_input):
"""
Internal inference methods
:param model_input: transformed model input data
:return: list of inference output in NDArray
"""
# Do some inference call to engine here and return output
model_input = model_input.to("cuda:0")
with torch.set_grad_enabled(False):
model_output = self.model(model_input)
emodel_output = self.emodel(model_input)
return model_output
How are you fixing the seed?
presumably you're doing
torch.manual_seed(0)
np.random.seed(0)
random.seed(0)
torch.backends.cudnn.deterministic = True
torch.backends.cudnn.benchmark = False
