How to use inference of multiple models

[polarisunny]

How to use inference of multiple models, such as PaddleOCR, which has three models, namely det, cls, and rec.

[frankfliu]

There are two ways to solve your problem, see the following demo:

1. User DJLServing workflow: https://github.com/deepjavalibrary/djl-demo/tree/master/djl-serving/workflows/multi-model
2. Use a custom Translator and load smaller model in the Translator

[polarisunny]

Which of these two methods performs better? I want to try it.

[frankfliu]

@faquir-sun It's really depends on how heavy your smaller models are. For option 1, each model can scale up to X number of workers independently, I can leverage more CPU power concurrently. However, the communication between each workflow may cause more overhead.

For option 2, everything runs in the same thread, and there is not serialization/deserialization between models. But only one worker is used for small model. This may become bottleneck.

[polarisunny]

For option 2, is there any example for reference?

[frankfliu]

@faquir-sun Here is an example that involves two models: https://github.com/deepjavalibrary/djl/blob/master/examples/src/main/java/ai/djl/examples/inference/clip/ImageTextComparison.java

Above example treat two model equally, and use an utility function to run the inference. You can wrap one model into a custom translator if you want to.