Viz DSL: Apply bounding boxes to Videos

[da-liii]

UDT: TimeSeriesBoundingBox

array<tuple<ts, box2d>>

DSL to append the bounding box to the video (one video)

video = Video("xyz.flv")
video | TimeSeriesOfBoundingBox

Assuming that we are processing videos in a frame-by-frame way, we can apply the time series of bounding boxes to the video via the same pipe operator (|).

How it works with the Spark SQL and Rikai format? (multiple videos)

create or replace temporary view frames as
from (
  from (
    from video.`xyz.fly`
    select video_uri, frame_id, ts, image_data
  )
  select video_uri, frame_id, ts, to_image(image_data) as image, ML_PREDICT(model, image_data) as pred
)
select video_uri, frame_id, ts, image, to_box(pred) as box;

-- cache it

create or replace temporary view video_and_ts_boxes as
select video_uri, agg__time_series_box(box, ts) as ts_boxes from frames group by video_uri;


create or replace temporary view boxed_video as
select boxed_video(to_video(video_uri), ts_boxes) from frames

df = spark.table("video_and_ts_boxes")
row = df.collect().head
video_uri = row.video_uri
ts_boxes = row.ts_boxes
video_to_check = Video(video_uri) | ts_boxes
video_to_check # we might need a server to play the video

Issues

Blocked by #689

[changhiskhan]

One thing here is that you almost never label all frames in a video, so alignment will be tricky.