C3D
C3D copied to clipboard
facebookarchive
Python wrapper
Made pycaffe available for C3D classification with a sample python wrapper (run_c3d_classification.py)
Thank you for your pull request and welcome to our community. We require contributors to sign our Contributor License Agreement, and we don't seem to have you on file. In order for us to review and merge your code, please sign up at https://code.facebook.com/cla - and if you have received this in error or have any questions, please drop us a line at [email protected]. Thanks!
Thank you for signing our Contributor License Agreement. We can now accept your code for this (and any) Facebook open source project. Thanks!
I believe the way you read in the mean is wrong (mixed dimension). I had the problem that the mean didn't match in my Pull request to Lasagne Recipes: https://github.com/Lasagne/Recipes/pull/41
It appears the values are read in a different order from the one they are written.
The elements in C3D are shaped as (channel,frameNr,y,x), but you read in the mean as (frameNr,channel,y,x). Fix: image_mean.reshape(3,16,128,171) did the trick for me :)
Here is the mean difference between the frame means visualized: Difference between the first channel of image 5 and image 0: https://data.vision.ee.ethz.ch/gyglim/C3D/mean_diff/5_0.png
Difference between the first channel of image 6 and image 5: https://data.vision.ee.ethz.ch/gyglim/C3D/mean_diff/6_5.png
i.e. there is a jump in the values, which comes from reading the input wrong.
I believe this commit completes all there is to support C3D in python scripts. Please let me know if anyone has a feedback or a question. Thanks.
@chuckcho Thank you very much for your contribution. I did not have time to fully review this for now (so it may take sometime to merge as soon as I fully try/test it).
@chuckcho I wonder whether I could directly load the pre-trained Sport1m model in run_c3d_classification.py?
I edited the prototxt file for model conv3d_deepnetA_sport1m_iter_1900000 just like conv3d_ucf101_deploy.prototxt. But I still have trouble when using net = caffe.Net(model_def_file, model_file) to load the sport1m model. The python console goes wrong like that It seems the kernel died unexpectedly. Use 'Restart kernel' to continue using this console.
@fengwhu. conv3d_deepnetA_sport1m_iter_1900000 was created with c3d_finetuning/c3d_ucf101_finetuning_train.prototxt, so the deploy proto file should be modified as such. For example, the following deploy file works fine with conv3d_deepnetA_sport1m_iter_1900000:
name: "DeepConv3DNet_ucf101_finetuning"
input: "data"
input_dim: 30
input_dim: 3
input_dim: 16
input_dim: 112
input_dim: 112
# ----------- 1st layer group ---------------
layers {
name: "conv1a"
type: CONVOLUTION3D
bottom: "data"
top: "conv1a"
blobs_lr: 1
blobs_lr: 2
weight_decay: 1
weight_decay: 0
convolution_param {
num_output: 64
kernel_size: 3
kernel_depth: 3
pad: 1
temporal_pad: 1
weight_filler {
type: "gaussian"
std: 0.01
}
bias_filler {
type: "constant"
value: 0
}
}
}
layers {
name: "relu1a"
type: RELU
bottom: "conv1a"
top: "conv1a"
}
layers {
name: "pool1"
type: POOLING3D
bottom: "conv1a"
top: "pool1"
pooling_param {
pool: MAX
kernel_size: 2
kernel_depth: 1
stride: 2
temporal_stride: 1
}
}
# ------------- 2nd layer group --------------
layers {
name: "conv2a"
type: CONVOLUTION3D
bottom: "pool1"
top: "conv2a"
blobs_lr: 1
blobs_lr: 2
weight_decay: 1
weight_decay: 0
convolution_param {
num_output: 128
kernel_size: 3
kernel_depth: 3
pad: 1
temporal_pad: 1
weight_filler {
type: "gaussian"
std: 0.01
}
bias_filler {
type: "constant"
value: 1
}
}
}
layers {
name: "relu2a"
type: RELU
bottom: "conv2a"
top: "conv2a"
}
layers {
name: "pool2"
type: POOLING3D
bottom: "conv2a"
top: "pool2"
pooling_param {
pool: MAX
kernel_size: 2
kernel_depth: 2
stride: 2
temporal_stride: 2
}
}
# ----------------- 3rd layer group --------------
layers {
name: "conv3a"
type: CONVOLUTION3D
bottom: "pool2"
top: "conv3a"
blobs_lr: 1
blobs_lr: 2
weight_decay: 1
weight_decay: 0
convolution_param {
num_output: 256
kernel_size: 3
kernel_depth: 3
pad: 1
temporal_pad: 1
weight_filler {
type: "gaussian"
std: 0.01
}
bias_filler {
type: "constant"
value: 1
}
}
}
layers {
name: "relu3a"
type: RELU
bottom: "conv3a"
top: "conv3a"
}
layers {
name: "conv3b"
type: CONVOLUTION3D
bottom: "conv3a"
top: "conv3b"
blobs_lr: 1
blobs_lr: 2
weight_decay: 1
weight_decay: 0
convolution_param {
num_output: 256
kernel_size: 3
kernel_depth: 3
pad: 1
temporal_pad: 1
weight_filler {
type: "gaussian"
std: 0.01
}
bias_filler {
type: "constant"
value: 1
}
}
}
layers {
name: "relu3b"
type: RELU
bottom: "conv3b"
top: "conv3b"
}
layers {
name: "pool3"
type: POOLING3D
bottom: "conv3b"
top: "pool3"
pooling_param {
pool: MAX
kernel_size: 2
kernel_depth: 2
stride: 2
temporal_stride: 2
}
}
# --------- 4th layer group
layers {
name: "conv4a"
type: CONVOLUTION3D
bottom: "pool3"
top: "conv4a"
blobs_lr: 1
blobs_lr: 2
weight_decay: 1
weight_decay: 0
convolution_param {
num_output: 512
kernel_size: 3
kernel_depth: 3
pad: 1
temporal_pad: 1
weight_filler {
type: "gaussian"
std: 0.01
}
bias_filler {
type: "constant"
value: 1
}
}
}
layers {
name: "relu4a"
type: RELU
bottom: "conv4a"
top: "conv4a"
}
layers {
name: "conv4b"
type: CONVOLUTION3D
bottom: "conv4a"
top: "conv4b"
blobs_lr: 1
blobs_lr: 2
weight_decay: 1
weight_decay: 0
convolution_param {
num_output: 512
kernel_size: 3
kernel_depth: 3
pad: 1
temporal_pad: 1
weight_filler {
type: "gaussian"
std: 0.01
}
bias_filler {
type: "constant"
value: 1
}
}
}
layers {
name: "relu4b"
type: RELU
bottom: "conv4b"
top: "conv4b"
}
layers {
name: "pool4"
type: POOLING3D
bottom: "conv4b"
top: "pool4"
pooling_param {
pool: MAX
kernel_size: 2
kernel_depth: 2
stride: 2
temporal_stride: 2
}
}
# --------------- 5th layer group --------
layers {
name: "conv5a"
type: CONVOLUTION3D
bottom: "pool4"
top: "conv5a"
blobs_lr: 1
blobs_lr: 2
weight_decay: 1
weight_decay: 0
convolution_param {
num_output: 512
kernel_size: 3
kernel_depth: 3
pad: 1
temporal_pad: 1
weight_filler {
type: "gaussian"
std: 0.01
}
bias_filler {
type: "constant"
value: 1
}
}
}
layers {
name: "relu5a"
type: RELU
bottom: "conv5a"
top: "conv5a"
}
layers {
name: "conv5b"
type: CONVOLUTION3D
bottom: "conv5a"
top: "conv5b"
blobs_lr: 1
blobs_lr: 2
weight_decay: 1
weight_decay: 0
convolution_param {
num_output: 512
kernel_size: 3
kernel_depth: 3
pad: 1
temporal_pad: 1
weight_filler {
type: "gaussian"
std: 0.01
}
bias_filler {
type: "constant"
value: 1
}
}
}
layers {
name: "relu5b"
type: RELU
bottom: "conv5b"
top: "conv5b"
}
layers {
name: "pool5"
type: POOLING3D
bottom: "conv5b"
top: "pool5"
pooling_param {
pool: MAX
kernel_size: 2
kernel_depth: 2
stride: 2
temporal_stride: 2
}
}
# ---------------- fc layers -------------
layers {
name: "fc6"
type: INNER_PRODUCT
bottom: "pool5"
top: "fc6"
blobs_lr: 1
blobs_lr: 2
weight_decay: 1
weight_decay: 0
inner_product_param {
num_output: 4096
weight_filler {
type: "gaussian"
std: 0.005
}
bias_filler {
type: "constant"
value: 1
}
}
}
layers {
name: "relu6"
type: RELU
bottom: "fc6"
top: "fc6"
}
layers {
name: "drop6"
type: DROPOUT
bottom: "fc6"
top: "fc6"
dropout_param {
dropout_ratio: 0.5
}
}
layers {
name: "fc7"
type: INNER_PRODUCT
bottom: "fc6"
top: "fc7"
blobs_lr: 1
blobs_lr: 2
weight_decay: 1
weight_decay: 0
inner_product_param {
num_output: 4096
weight_filler {
type: "gaussian"
std: 0.005
}
bias_filler {
type: "constant"
value: 1
}
}
}
layers {
name: "relu7"
type: RELU
bottom: "fc7"
top: "fc7"
}
layers {
name: "drop7"
type: DROPOUT
bottom: "fc7"
top: "fc7"
dropout_param {
dropout_ratio: 0.5
}
}
layers {
name: "fc8"
type: INNER_PRODUCT
bottom: "fc7"
top: "fc8"
blobs_lr: 1
blobs_lr: 2
weight_decay: 1
weight_decay: 0
inner_product_param {
num_output: 101
weight_filler {
type: "gaussian"
std: 0.01
}
bias_filler {
type: "constant"
value: 0
}
}
}
layers {
name: "prob"
type: SOFTMAX
bottom: "fc8"
top: "prob"
}