MINI-Net: Multiple Instance Ranking Network for Video Highlight Detection

This repo contains source code for our ECCV 2020 work MINI-Net: Multiple Instance Ranking Network for Video Highlight Detection. Our model is implemented under Pytorch.

Prerequisites

1. Pytorch 1.4 +
2. numpy
3. tqdm
4. moviepy
5. cv2

Getting Started

Change directory to src and run following code:

CUDA_VISIBLE_DEVICES=1 python trainMIL.py --dataset youtube --domain gymnastics \
	--train_path /home/share/Highlight/proDataset/TrainingSet/ \
	--test_path /home/share/Highlight/proDataset/DomainSpecific \
	--topk_mAP 1 --FNet MILModel10 --AM AttentionModule \
	--DS MILDataset --AHLoss AdaptiveHingerLoss \
	--short_lower 10 --short_upper 40 --long_lower 60 --long_upper 60000 --bagsize 60 

Parameters:

• CUDA_VISIBLE_DEVICES: specify GPU Id for training
• dataset: choose dataset, alternatives: youtube, tvsum, cosum
• domain: choose target domain in given dataset, e.g., gymnastics for youtube dataset
• train_path: extracted feature file for training, mentioned above
• test_path: extracted feature file for testing
• topk_mAP: specify test metric, 1 or 5 in our paper
• FNet: which model to use to predict highlight score for each segment in video, in our paper: MILModel10
• AM: which model to fuse visual feature and audio feature, in our paper: AttentionModule_1
• DS: dataset model: in out paper: MILDataset
• AHLoss: hinger loss used in our paper

See visual-audio fusion/opts.py for details of data selection hyper-parameters.

The extracted features of three test datasets are available at here.

Main Results

• Youtube:

• TVsum:

• CoSum:

Reference

If you find our work helpful in your research, please cite our paper via:

Bib:
@inproceedings{hong2020mini,
title={MINI-Net: Multiple Instance Ranking Network for Video Highlight Detec- tion},
author={Hong, Fa-Ting and Huang, Xuanteng and Li, Wei-Hong and Zheng, Wei-Shi},
booktitle={European Conference on Computer Vision},
year={2020}
}

More information about our work can be viewed in https://harlanhong.github.io.