HOPE
HOPE copied to clipboard
bardiadoosti
Test error is bigger than reported
Hi,
I know this issue has been raised in issue 13 and issue 17, and it doesn't seem that they are addressed. I was hoping if you could provide with some feedback.
I followed your github instruction to evaluate on the pre-trained model. The test error I got is:
Test files loaded
HopeNet is loaded
Begin testing the network...
test error: 1042.98572
which is different from the test error 253 from issue 17.
When I do python make_data.py, there are a few sequences without video files (I am not sure if it is expected). See the log:
...
pour_milk
Error in ====Subject_6, pour_milk, 8====
Error in ====Subject_6, pour_milk, 7====
Error in ====Subject_6, pour_milk, 6====
Error in ====Subject_6, pour_milk, 9====
Error in ====Subject_6, pour_milk, 10====
open_juice_bottle
pour_juice_bottle
open_milk
Error in ====Subject_6, open_milk, 8====
Error in ====Subject_6, open_milk, 7====
Error in ====Subject_6, open_milk, 6====
Error in ====Subject_6, open_milk, 9====
Error in ====Subject_6, open_milk, 10====
put_salt
close_liquid_soap
Error in ====Subject_6, close_liquid_soap, 8====
Error in ====Subject_6, close_liquid_soap, 7====
Error in ====Subject_6, close_liquid_soap, 6====
Error in ====Subject_6, close_liquid_soap, 9====
Error in ====Subject_6, close_liquid_soap, 10====
open_liquid_soap
Error in ====Subject_6, open_liquid_soap, 8====
Error in ====Subject_6, open_liquid_soap, 7====
Error in ====Subject_6, open_liquid_soap, 6====
Error in ====Subject_6, open_liquid_soap, 5====
pour_liquid_soap
Error in ====Subject_6, pour_liquid_soap, 8====
Error in ====Subject_6, pour_liquid_soap, 7====
Error in ====Subject_6, pour_liquid_soap, 6====
Error in ====Subject_6, pour_liquid_soap, 5====
close_juice_bottle
close_milk
Error in ====Subject_6, close_milk, 8====
Error in ====Subject_6, close_milk, 7====
Error in ====Subject_6, close_milk, 6====
Error in ====Subject_6, close_milk, 9====
Error in ====Subject_6, close_milk, 10====
Subject_2
pour_milk
open_juice_bottle
pour_juice_bottle
open_milk
...
If I let model.eval() in HOPE.py, the error drops a bit but I couldn't get it around 200:
Test files loaded
HopeNet is loaded
Begin testing the network...
test error: 969.59106
The test script that I used is:
#!/bin/bash
export CUDA_VISIBLE_DEVICES=0
python HOPE.py \
--input_file ./datasets/fhad/ \
--test \
--batch_size 16 \
--model_def HopeNet \
--gpu \
--gpu_number 0 \
--pretrained_model ./checkpoints/fhad/model-0.pkl
I use pytorch 1.4, torchvision 0.5, and python 3.7.7
I'm interested in the same question. Using the same library versions as recommended in the issues posts, I've been unable to replicate the author's numbers in the paper
@pgrady3 Me too, I could not even get closer, something is wrong either on the BatchNorm during inference or something with the test split maybe.
Hi, today I checked the weights and found out unfortunately something is wrong with the provided weights specially regarding to the GraphUNet's performance. I try to solve this problem by providing the right weights for the GraphUNet in the next couple of days. I apologize for any inconvenience.
Hi, today I checked the weights and found out unfortunately something is wrong with the provided weights specially regarding to the GraphUNet's performance. I try to solve this problem by providing the right weights for the GraphUNet in the next couple of days. I apologize for any inconvenience.
Hi bardiadoosti,
Thanks for updating the trained weight and sharing it again. However, there are some missing weights in the shared model. I run the updated testing code and get the following error:
RuntimeError: Error(s) in loading state_dict for DataParallel: Missing key(s) in state_dict: "module.pool1.fc.weight", "module.pool1.fc.bias", "module.pool2.fc.weight", "module.pool2.fc.bias", "module.pool3.fc.weight", "module.pool3.fc.bias", "module.pool4.fc.weight", "module.pool4.fc.bias", "module.pool5.fc.weight", "module.pool5.fc.bias".
can you please advise.
@Almadani92 Hi, you need to remove the string "module" from the keys in the dictionary and replace them accordingly with the model. Would you please tell me where is the updated version? Thanks.
@Almadani92 I created the code for you:
"""# Load Snapshot"""
if args.pretrained_model != '':
loaded_model = torch.load(args.pretrained_model)
resnet = {}
graphnet = {}
graphunet = {}
for key, val in loaded_model.items():
if 'resnet' in key:
resnet[key.replace('module.resnet.', '')] = val
if 'graphnet' in key:
graphnet[key.replace('module.graphnet.', '')] = val
if 'graphunet' in key:
graphunet[key.replace('module.graphunet.', '')] = val
model.resnet.load_state_dict(resnet)
model.graphnet.load_state_dict(graphnet)
model.graphunet.load_state_dict(graphunet)
losses = np.load(args.pretrained_model[:-4] + '-losses.npy').tolist()
start = len(losses)
else:
losses = []
start = 0
@Almadani92 I created the code for you:
"""# Load Snapshot"""
if args.pretrained_model != '': loaded_model = torch.load(args.pretrained_model) resnet = {} graphnet = {} graphunet = {} for key, val in loaded_model.items(): if 'resnet' in key: resnet[key.replace('module.resnet.', '')] = val if 'graphnet' in key: graphnet[key.replace('module.graphnet.', '')] = val if 'graphunet' in key: graphunet[key.replace('module.graphunet.', '')] = val model.resnet.load_state_dict(resnet) model.graphnet.load_state_dict(graphnet) model.graphunet.load_state_dict(graphunet) losses = np.load(args.pretrained_model[:-4] + '-losses.npy').tolist() start = len(losses) else: losses = [] start = 0
Many thanks @hedjm,
Actually, it seems to me that really there are missing weights and biases for all pooling layers in the new pre-trained graphunet model. The new model has been uploaded from 3 days ago. you can find its link in the README.md file.
Thanks.
@Almadani92 After downloading the new pretrained model, I agree that the weights for the GraphUNet pooling layers are missing. Here are the layers present in the weights file, but the FC layer from the down-pooling layers aren't present. Has anyone found a workaround, or could the author comment?
module.A_0
module.A_1
module.A_2
module.A_3
module.A_4
module.A_5
module.gconv1.fc.weight
module.gconv1.fc.bias
module.gconv2.fc.weight
module.gconv2.fc.bias
module.gconv3.fc.weight
module.gconv3.fc.bias
module.gconv4.fc.weight
module.gconv4.fc.bias
module.gconv5.fc.weight
module.gconv5.fc.bias
module.fc1.weight
module.fc1.bias
module.fc2.weight
module.fc2.bias
module.unpool6.fc.weight
module.unpool6.fc.bias
module.gconv6.fc.weight
module.gconv6.fc.bias
module.unpool7.fc.weight
module.unpool7.fc.bias
module.gconv7.fc.weight
module.gconv7.fc.bias
module.unpool8.fc.weight
module.unpool8.fc.bias
module.gconv8.fc.weight
module.gconv8.fc.bias
module.unpool9.fc.weight
module.unpool9.fc.bias
module.gconv9.fc.weight
module.gconv9.fc.bias
module.unpool10.fc.weight
module.unpool10.fc.bias
module.gconv10.fc.weight
module.gconv10.fc.bias
Hi @bardiadoosti , just want to check if there is any update on this. Thanks.
Cleaning up the make_data.py on my fork got the test error rate down to ~700. Still way over what the paper outlines
python Graph.py \
--input_file ./datasets/fhad/ \
--test \
--batch_size 64 \
--model_def GraphUNet \
--pretrained_model ./checkpoints/graphunet/model-0.pkl
Test files loaded
NOT using CUDA
GraphUNet is loaded
Begin testing the network...
test error: 700.60317
Python 3.9.10 numpy==1.22.2 Pillow==9.0.1 scipy==1.8.0 torch==1.9.0 torchvision==0.10.0 trimesh==3.10.0
