How do you extract RGB and optical frames ?

[zhujingsong]

Can you show me the codes of your extracting RGB and optical frames?

[Rhythmblue]

A. I have used two ways to extract flow

this is a repo which can extract flow and warped flow(which is modified by RANSAC to remove motion of background) And I use the branch of opencv3.1. https://github.com/yjxiong/dense_flow/tree/opencv-3.1 (CPU or GPU)

api of python-opencv you need to install by:

pip install opencv-python pip install opencv-contrib-python

Then, you can use the TVL1 api to extract optical flow (Note: opencv-python cannot use GPU to compute flow. You can use multi-processing) I add a py file in this mail.

B. There are some ways to extract rgb. Sometimes I use ffmpeg. Sometimes use VideoCapture api of opencv-python.

[Rhythmblue]

import os
import numpy as np
import cv2
from glob import glob
from multiprocessing import Pool


_IMAGE_SIZE = 256


def cal_for_frames(video_path):
    frames = glob(os.path.join(video_path, '*.jpg'))
    frames.sort()

    flow = []
    prev = cv2.imread(frames[0])
    prev = cv2.cvtColor(prev, cv2.COLOR_BGR2GRAY)
    for i, frame_curr in enumerate(frames):
        curr = cv2.imread(frame_curr)
        curr = cv2.cvtColor(curr, cv2.COLOR_BGR2GRAY)
        tmp_flow = compute_TVL1(prev, curr)
        flow.append(tmp_flow)
        prev = curr

    return flow


def compute_TVL1(prev, curr, bound=15):
    """Compute the TV-L1 optical flow."""
    TVL1 = cv2.DualTVL1OpticalFlow_create()
    flow = TVL1.calc(prev, curr, None)
    assert flow.dtype == np.float32

    flow = (flow + bound) * (255.0 / (2*bound))
    flow = np.round(flow).astype(int)
    flow[flow >= 255] = 255
    flow[flow <= 0] = 0

    return flow


def save_flow(video_flows, flow_path):
    for i, flow in enumerate(video_flows):
        cv2.imwrite(os.path.join(flow_path.format('u'), "{:06d}.jpg".format(i)),
                    flow[:, :, 0])
        cv2.imwrite(os.path.join(flow_path.format('v'), "{:06d}.jpg".format(i)),
                    flow[:, :, 1])


def gen_video_path():
    path = []
    flow_path = []
    length = []
    base = ''
    flow_base = ''
    for task in ['train', 'dev', 'test']:
        videos = os.listdir(os.path.join(base, task))
        for video in videos:
            tmp_path = os.path.join(base, task, video, '1')
            tmp_flow = os.path.join(flow_base, task, '{:s}', video)
            tmp_len = len(glob(os.path.join(tmp_path, '*.png')))
            u = False
            v = False
            if os.path.exists(tmp_flow.format('u')):
                if len(glob(os.path.join(tmp_flow.format('u'), '*.jpg'))) == tmp_len:
                    u = True
            else:
                os.makedirs(tmp_flow.format('u'))
            if os.path.exists(tmp_flow.format('v')):
                if len(glob(os.path.join(tmp_flow.format('v'), '*.jpg'))) == tmp_len:
                    v = True
            else:
                os.makedirs(tmp_flow.format('v'))
            if u and v:
                print('skip:' + tmp_flow)
                continue

            path.append(tmp_path)
            flow_path.append(tmp_flow)
            length.append(tmp_len)
    return path, flow_path, length


def extract_flow(args):
    video_path, flow_path = args
    flow = cal_for_frames(video_path)
    save_flow(flow, flow_path)
    print('complete:' + flow_path)
    return


if __name__ =='__main__':
    pool = Pool(2)   # multi-processing

    video_paths, flow_paths, video_lengths = gen_video_path()

    pool.map(extract_flow, zip(video_paths, flow_paths))

[XudongLinthu]

Hi, I wonder which one of the flow extraction code you used for producing your results reported 96.3%. Or their results are similar? Thanks a lot!

[vra]

@XudongLinthu , We use optical flow from here.

[XudongLinthu]

@XudongLinthu , We use optical flow from here.

Hi, Thanks a lot for your reply! So did you use warping? And you used TV-L1 and truncated the flow into [-20, 20], right? Have you used other pre-processing techniques? Thanks, Xudong

[Gavin666Github]

@zhujingsong hi,jing song,"We use list files in data/ucf101/ subdir to make the code find RGB images and flow data saved on disk" ,have you solved the problem? if YES,please share me the way you used.thank you!

[Gavin666Github]

@XudongLinthu hello,xu,I'am in China but i can't download the link shared.would you please share the datasets with baidu cloud disk to me?many thanks！

[Rhythmblue]

@Gavin666Github I recommend you to download the whole dataset at http://crcv.ucf.edu/data/UCF101.php. It can be downloaded in China about 6.5G. Then, you can use code above to extract optical flow. The whole process will no more than 12 hours. The whole decoded rgbs and flows are about 130G. It's way more time-comsuming to upload and download on Baidu Cloud with turtle-like speed.

[Gavin666Github]

@Rhythmblue aha~~ It's so big. It's going to be very slow to download. Thank you for your advice.I could try.

[chunduri11]

@Rhythmblue what should be the input format to the model. images of rgb/flow from the videos (or) .npy file of the rgb/flow of the videos.

[rishabh2301]

Hi @Rhythmblue thank you very much for sharing your code to extract flow. My question is have you used this code for generating flow for UCF101 ? if yes can you please tell how much time it took for you to generate the flow for all the videos. If you may have used any other code which offers GPU support can you please share it. I am working on our own dataset of 4000 videos and I want to extract flow for them. Many thanks!

[YuLengChuanJiang]

Hi, @vra @Rhythmblue @chunduri11 @rishabh2301 @XudongLinthu I wonder to know how to solve the video frames less than 64 in the code. Because I get o problem "No such file or directory: '/data/ange/UCF101/images/v_CricketShot_g20_c05/img_00062.jpg'". Finally, I find that the video frames less than 64.

thanks!