Streaming video inference with client-side batching

[agunapal]

Description

For some near real-time streaming video applications, there might be a need for batching on the application side. This example shows how this can be achieved using TorchServe.

Client:

This example is using OpenCV on the client side application to read frames from a video On the client side, we have one thread for reading frames from a video source and another thread which batches(size n) the read frames and sends the request to TorchServe for image classification inference. To send the batched data, we create a json payload of n frames.

To choose batch size as 10, use the arg --batch_size 10 with the given client side script

TorchServe Handler

On the TorchServe side, we read the json payload and preprocess the n frames. The postprocess function in the handler returns the output as a list of length 1.

Output

To measure the performance, we print the Frames per second (FPS) as follows

With Batch Size 10, FPS at frame number 20 is 25.9
With Batch Size 10, FPS at frame number 40 is 26.1
With Batch Size 10, FPS at frame number 60 is 25.6
With Batch Size 10, FPS at frame number 80 is 25.8
With Batch Size 10, FPS at frame number 100 is 24.9
With Batch Size 10, FPS at frame number 120 is 27.4
With Batch Size 10, FPS at frame number 140 is 24.9
With Batch Size 10, FPS at frame number 160 is 26.5
With Batch Size 10, FPS at frame number 180 is 25.0
With Batch Size 10, FPS at frame number 200 is 26.9
With Batch Size 10, FPS at frame number 220 is 28.0
With Batch Size 10, FPS at frame number 240 is 25.1
With Batch Size 10, FPS at frame number 260 is 26.6
With Batch Size 10, FPS at frame number 280 is 25.4
With Batch Size 10, FPS at frame number 300 is 24.7
With Batch Size 10, FPS at frame number 320 is 25.9
With Batch Size 10, FPS at frame number 340 is 25.5
With Batch Size 10, FPS at frame number 360 is 26.2
Done reading 375 frames
Length of queue is 0 , snd_cnt is 370
With Batch Size 10, FPS at frame number 375 is 26.0

Printing the output response for batch size 1 will give the following output

[
  {
    "tabby": 0.5186409950256348,
    "tiger_cat": 0.29040342569351196,
    "Egyptian_cat": 0.10797449946403503,
    "lynx": 0.01395314373075962,
    "bucket": 0.006002399604767561
  }
]

Fixes #(issue)

Type of change

Please delete options that are not relevant.

• [ ] New feature (non-breaking change which adds functionality)

Feature/Issue validation/testing

Please describe the Unit or Integration tests that you ran to verify your changes and relevant result summary. Provide instructions so it can be reproduced. Please also list any relevant details for your test configuration.

• Output Logs

Checklist:

• [ ] Did you have fun?
• [ ] Have you added tests that prove your fix is effective or that this feature works?
• [ ] Has code been commented, particularly in hard-to-understand areas?
• [ ] Have you made corresponding changes to the documentation?

[codecov[bot]]

Codecov Report

Merging #1867 (fd0b993) into master (13f2e78) will not change coverage. The diff coverage is n/a.

@@           Coverage Diff           @@
##           master    #1867   +/-   ##
=======================================
  Coverage   44.95%   44.95%           
=======================================
  Files          63       63           
  Lines        2609     2609           
  Branches       56       56           
=======================================
  Hits         1173     1173           
  Misses       1436     1436           

:mega: We’re building smart automated test selection to slash your CI/CD build times. Learn more

[lxning]

@agunapal could you please attach regression test log to check pytest?

[agunapal]

@lxning Updated with regression logs. The test is being skipped from the regression suite, to avoid installing other client libraries ( so that binary size doesn't increase)