Perf_analyzer reported metrics for decoupled model

[ZhanqiuHu]

I am trying to profile our decoupled models (python backend) with perf_analyzer, and I'm curious how the following latency metrics are calculated?

Client Send, Network+Server Send/Recv,Server Queue,Server Compute Input,Server Compute Infer,Server Compute Output, and Client Recv

Also, when using grpc or http endpoints, is it possible to measure the latencies spend on network overhead and (un)marshalling protobuf?

Thanks!

[nv-hwoo]

Hi @ZhanqiuHu,

I am trying to profile our decoupled models (python backend) with perf_analyzer, and I'm curious how the following latency metrics are calculated?

Please see here for the details of the metrics being calculated.

Also, when using grpc or http endpoints, is it possible to measure the latencies spend on network overhead and (un)marshalling protobuf?

I believe if you run perf analyzer with -i grpc, you should see output like this

*** Measurement Settings ***
  Batch size: 1
  Service Kind: Triton
  Using "time_windows" mode for stabilization
  Measurement window: 5000 msec
  Using synchronous calls for inference
  Stabilizing using average latency

Request concurrency: 1
  Client:
    Request count: 30375
    Throughput: 1685.54 infer/sec
    Avg latency: 591 usec (standard deviation 144 usec)
    p50 latency: 569 usec
    p90 latency: 710 usec
    p95 latency: 891 usec
    p99 latency: 1105 usec
    Avg gRPC time: 578 usec ((un)marshal request/response 6 usec + response wait 572 usec)
  Server:
    Inference count: 30376
    Execution count: 30376
    Successful request count: 30376
    Avg request latency: 319 usec (overhead 107 usec + queue 26 usec + compute input 46 usec + compute infer 85 usec + compute output 53 usec)

Inferences/Second vs. Client Average Batch Latency
Concurrency: 1, throughput: 1685.54 infer/sec, latency 591 usec

[ZhanqiuHu]

Thanks a lot for providing the details! I was more interested in what "Compute Input", "Compute Output", and "Network+Server Send/Recv" specifically are. When I use -i grpc the flag, it doesn't seem to report the gRPC time, and I was wondering if it is because I'm using a custom decoupled python model.

Thank you very much!

[nv-hwoo]

For compute input, compute infer, and compute output metrics, you could read the Triton doc here for more details.

When I use -i grpc the flag, it doesn't seem to report the gRPC time, and I was wondering if it is because I'm using a custom decoupled python model.

Yes you are correct. The gRPC time reports are not supported in decoupled model.

[ZhanqiuHu]

Thanks for the answer! However, it seems like the description on the doc is a little bit vague. What specific steps are involved in preprocessing of inputs and outputs? For example, for inputs, copying/moving the data to the device is probably part of it? And I guess for decoupled python model, (de)serailization will be part of comptue infer time rather than compute input or compute output time?

Thanks!