tator icon indicating copy to clipboard operation
tator copied to clipboard

Optimize latencies for server-side media util

Open bctcvai opened this issue 6 months ago • 0 comments

Tator provides an innovative server-side media manipulation service with GetFrame and GetLocalizationGraphic -- A common use of these utilities is to fetch multiple frames or multiple related RoIs.

GetFrame supports sending back a tiled image of multiple frames. This can be used to lower latency of fetching nearby related frames in some UI applications. GetLocalizationGraphic could benefit from a similar trick. Because the extraction occurs in a child ffmpeg process, there is limited need for additional middleware for a scatter/gather to be implemented. If the Schema was extended to support multiple localizations in GetLocalizationGraphic; it could fetch N localization via asynchronous ffmpeg processes, join on them, and then create a tile image to return in one REST call. Some profiling should be done to set a limit on the number of N in this concept.

bctcvai avatar Aug 15 '24 12:08 bctcvai