Hello，is there any way I can get the frame from camera real-time？

[zhangcheng096]

Hello Mr，Liuyuan。

Thanks for the source code,I can get the frame from a video. I want to konw if there is anyway I can use a camera to get the frame and play it in real-time？ Any help would be appreciated

[liuyuan-pal]

Hi, you may refer to https://docs.opencv.org/4.x/dd/d43/tutorial_py_video_display.html

[apirrone]

Hi, if this can help you, here is the script I use to compute poses with Gen6D in real time (I just modified predict.py):

import argparse
import numpy as np
from dataset.database import parse_database_name, get_ref_point_cloud
from estimator import name2estimator
import sys
sys.path.insert(0, "./utils/")
from base_utils import load_cfg, project_points
from draw_utils import pts_range_to_bbox_pts, draw_bbox_3d
from pose_utils import pnp
import cv2

parser = argparse.ArgumentParser()
parser.add_argument('--cfg', type=str, default='configs/gen6d_pretrain.yaml')
parser.add_argument('--database', type=str, default="custom/mouse")
parser.add_argument('--resolution', type=int, default=960)
parser.add_argument('--transpose', action='store_true', dest='transpose', default=False)
# smooth poses
parser.add_argument('--num', type=int, default=5)
parser.add_argument('--std', type=float, default=2.5)
args = parser.parse_args()


def weighted_pts(pts_list, weight_num=10, std_inv=10):
    weights=np.exp(-(np.arange(weight_num)/std_inv)**2)[::-1] # wn
    pose_num=len(pts_list)
    if pose_num<weight_num:
        weights = weights[-pose_num:]
    else:
        pts_list = pts_list[-weight_num:]
    pts = np.sum(np.asarray(pts_list) * weights[:,None,None],0)/np.sum(weights)
    return pts

cfg = load_cfg(args.cfg)
ref_database = parse_database_name(args.database)
estimator = name2estimator[cfg['type']](cfg)
estimator.build(ref_database, split_type='all')

object_pts = get_ref_point_cloud(ref_database)
object_bbox_3d = pts_range_to_bbox_pts(np.max(object_pts,0), np.min(object_pts,0))

def get_pose_img(im, pose_init, hist_pts):
    h, w = im.shape[:2]
    f=np.sqrt(h**2+w**2)
    K = np.asarray([[f,0,w/2],[0,f,h/2],[0,0,1]],np.float32)
    if pose_init is not None:
        estimator.cfg['refine_iter'] = 1 # we only refine one time after initialization
    pose_pr, inter_results = estimator.predict(im, K, pose_init=pose_init)
    pose_init = pose_pr

    pts, _ = project_points(object_bbox_3d, pose_pr, K)
    bbox_img = draw_bbox_3d(im, pts, (0,0,255))

    hist_pts.append(pts)
    pts_ = weighted_pts(hist_pts, weight_num=args.num, std_inv=args.std)
    pose_ = pnp(object_bbox_3d, pts_, K)
    pts__, _ = project_points(object_bbox_3d, pose_, K)
    bbox_img_ = draw_bbox_3d(im, pts__, (0,0,255))

    return bbox_img_, pose_init, inter_results

cap = cv2.VideoCapture(0)
pose_init = None
hist_pts = []
i = 0
while True:
    # When pose_init is none, the 4 steps are computed (detection, selection, pose, refine)
    # when it is not, juste the refine step is computed, initialized with the previous pose_init
    if i%20==0: # I recompute all every 20 frames
        pose_init = None

    ret, im = cap.read()

    pose_im, pose_init, inter_results = get_pose_img(im, pose_init, hist_pts)

    cv2.imshow("pose_im", pose_im)
    cv2.waitKey(1)

    i += 1

[AmokraneIlhem]

@apirrone Please, how can we use this code in colab especially the videocapture part ? Thank u in advance.

[apirrone]

@AmokraneIlhem I have never used collab, so I have no idea sorry :)

[EvdoTheo]

@apirrone Please, how can we use this code in colab especially the videocapture part ? Thank u in advance.

You're not able to access your local camera with "VideoCapture" command because google colab does not run in your own machine

[EvdoTheo]

Hello @apirrone did you experience any freezing of the video during the recomputation of the 4 steps( detection, selection, pose, refine)?