marqo icon indicating copy to clipboard operation
marqo copied to clipboard

[ENHANCEMENT] Better internal batching for images inference

Open jn2clark opened this issue 1 year ago • 1 comments

Is your feature request related to a problem? Please describe. Currently batching is effectively performed over text based fields (due to the internal splitting creating batches) but for images this not the case. This means there is probably up to a 2x improvement for throughput that could be gained by better batching of images for inference.

Describe the solution you'd like Batching of images for inference.

Describe alternatives you've considered Batching at the model level via an inference server. However the performance of earlier tests for this was that although batching at the model worked reasonably well, the overall throughput was worse.

jn2clark avatar Feb 22 '23 06:02 jn2clark