text-embeddings-inference icon indicating copy to clipboard operation
text-embeddings-inference copied to clipboard

Published 20 hours ago • verified publisher icon huggingface

`bge-reranker-v2-m3` model throughput benchmark

Open rere950303 opened this issue 1 year ago • 0 comments

Hello. I'm trying to estimate how much GPU equipment is needed to serve a bge-reranker-v2-m3 fp16 reranker model. We plan to input about 23 documents with a chunk size of 512, and the required TPS (transactions per second) is around 37.5. Could you help me determine how many T4 or A100 GPU units would be needed to meet these conditions? When we ran a load test after allocating 2 SM slices on an A100 device, we observed about 5.4 TPS. However, we're unable to check the GPU utilization, so I was wondering if you could provide some assistance with this.

rere950303 avatar Aug 29 '24 07:08 rere950303