How to reproduce the performence on your paper?

[ionicbond-lzj]

Thanks for sharing your code! And the YAML file I used is as follows.

name: xib
comment: default config
tags:
  - ${dataset.name}

model:
  name: xib.models.relational_network.build_relational_network
  input_node_model:  # f_n
    in_conv_shape: [256, 7, 7]  
    conv_channels: 256
    conv_layers: 2
    lin_features: 1024
    lin_layers: 1
    fc_features: 1024
    fc_layers: 1
  input_edge_model:  # f_e
    fc_features: 1024
    fc_layers: 1
  edge_model:  # f_r
    fc_features: 1024
    fc_layers: 1
  output_global_model:  # f_p
    fc_features: 1024
    fc_layers: 1
    mode: FC_AGG # AGG_FC
    pooling: max # mean
    last_bias: yes

dataset:
  name: hico  # unvrd
  eager: yes
  folder: ./data
  graph_type: fully_connected # fully_connected, human-object
  trainval:
    split: .85
    name: ${dataset.name}_relationship_detection_train
  test:
    name: ${dataset.name}_relationship_detection_test

dataloader:
  batch_size: 128  # 64, 128
  num_workers: 1

optimizer:
  name: torch.optim.Adam
  weight_decay: 1.0e-05

scheduler: {}

losses:
  rank:
    weight: 0.
  bce:
    weight: 1.

checkpoint:
  keep: 2
  folder: ./runs  

session:
  seed: ${random_seed:}  
  device: cuda  
  max_epochs: 18
  early_stopping:
    patience: 5

visual_relations:
  top_k_predicates: 10
  top_x_relations: 100
  activations: no
  channel_mean: no
  relevance_fn: l1 # l1 l2 sum_positives
  object_scores: yes
  frequencies: uniform # only uniform

hparams:
  data--: ${dataset.name}
  data/graph--dgt: ${dataset.graph_type}
  loss/bce--lbce: ${losses.bce.weight}
  loss/rank--lrank: ${losses.rank.weight}
  opt/weight_decay--wd: ${optimizer.weight_decay}
  model/conv_layers--cvl: ${model.input_node_model.conv_layers}
  model/fc_layers--fcl: ${model.input_node_model.fc_layers}
  model/fc_features--fcf: ${model.input_node_model.fc_features}
  model/readout_mode--ro: ${model.output_global_model.mode}
  model/readout_pool--rp: ${model.output_global_model.pooling}
  model/readout_bias--rb: ${model.output_global_model.last_bias}
  vr/topk--topk: ${visual_relations.top_k_predicates}
  vr/act--vra: ${visual_relations.activations}
  vr/chan_mean--vrcm: ${visual_relations.channel_mean}
  vr/rel_fn--vrrf: ${visual_relations.relevance_fn}
  vr/obj_score--vros: ${visual_relations.object_scores}
  vr/freq--vrf: ${visual_relations.frequencies}

However, the result I got is much lower than that on your paper(around 6%). Could you give me more details about configuration? Or could you explain this problem?

[baldassarreFe]

Hi, thanks for reporting this.

I am not sure what the problem could be. It could be due to differences in data preprocessing or in the training configuration. Could you detail the steps you took for downloading and preprocessing HICO? Also, can you attach the effective configuration (the one above is the input config that gets interpolated and filled up)?

[ionicbond-lzj]

Thanks for your reply. According to the README.md on the data folder, I download the HICO data and quickly test detectron detections. Here is the result, which is similar to that on your paper. AP | AP50 | AP75 | APs | APm | APl
20.22|34.090|20.757|2.305|11.489|29.699

Then, I preprocessed graphs for training.

python -m xib.preprocessing.hico_det \
  --skip-existing \
  --confidence-threshold=.3 \
  --dataset="hico" \
  --nms-threshold=.7 \
  --data-dir="./data"

Finally, I run the train.py. And the train_hico.yaml is just what I gave before(Yes, I don't use several yaml files to run the train.py but it works for me).

# bash scripts/train_hico.sh 0
export CUDA_VISIBLE_DEVICES=$1
export PYTHONPATH=~/ws-vrd-master/src

python src/xib/train.py ~/ws-vrd-master/config/train_hico.yaml 

[ionicbond-lzj]

Thanks for your reply. According to the README.md on the data folder, I download the HICO data and quickly tested detectron detections. Here is the result, which is similar to that on your paper. AP | AP50 | AP75 | APs | APm | APl 20.22|34.090|20.757|2.305|11.489|29.699

Then, I preprocess graphs for training. python -m xib.preprocessing.hico_det \ --skip-existing \ --confidence-threshold=.3 \ --dataset="hico" \ --nms-threshold=.7 \ --data-dir="./data"

Finally, I run the train.py. And the train_hico.yaml is just what I gave before(Yes, I don't use several yaml files to run the train.py but it works for me).

bash scripts/train_hico.sh 1 export CUDA_VISIBLE_DEVICES=$1 export PYTHONPATH=~/ws-vrd-master/src python src/xib/train.py ~/ws-vrd-master/config/train_hico.yaml

------------------ 原始邮件 ------------------ 发件人: "Federico Baldassarre"<[email protected]>; 发送时间: 2020年12月10日(星期四) 下午5:09 收件人: "baldassarreFe/ws-vrd"<[email protected]>; 抄送: "16级信3黎子建"<[email protected]>; "Author"<[email protected]>; 主题: Re: [baldassarreFe/ws-vrd] How to reproduce the performence on your paper? (#5)

Hi, thanks for reporting this.

I am not sure what the problem could be. It could be due to differences in data preprocessing or in the training configuration. Could you detail the steps you took for downloading and preprocessing HICO? Also, can you attach the effective configuration (the one above is the input config that gets interpolated and filled up)?

— You are receiving this because you authored the thread. Reply to this email directly, view it on GitHub, or unsubscribe.

[baldassarreFe]

The following is your email, formatted in markdown for my own sake:

Detecron output:

AP | AP50 | AP75 | APs | APm | APl
20.22|34.090|20.757|2.305|11.489|29.699

Preprocessing

python -m xib.preprocessing.hico_det \
   --skip-existing \
   --confidence-threshold=.3 \
   --dataset="hico" \
   --nms-threshold=.7 \
   --data-dir="./data"

Content of scripts/train_hico.sh:

export CUDA_VISIBLE_DEVICES=$1 
export PYTHONPATH=~/ws-vrd-master/src 
python src/xib/train.py ~/ws-vrd-master/config/train_hico.yaml

Launched as:

scripts/train_hico.sh 1 

I will have a look and check what could be going wrong :wink:

[ionicbond-lzj]

Thanks! I am looking forward to your reply!

------------------ 原始邮件 ------------------ 发件人: "Federico Baldassarre"<[email protected]>; 发送时间: 2020年12月11日(星期五) 凌晨0:40 收件人: "baldassarreFe/ws-vrd"<[email protected]>; 抄送: "16级信3黎子建"<[email protected]>; "Author"<[email protected]>; 主题: Re: [baldassarreFe/ws-vrd] How to reproduce the performence on your paper? (#5)

The following is your email, formatted in markdown for my own sake:

Detecron output: AP | AP50 | AP75 | APs | APm | APl 20.22|34.090|20.757|2.305|11.489|29.699
Preprocessing python -m xib.preprocessing.hico_det \ --skip-existing \ --confidence-threshold=.3 \ --dataset="hico" \ --nms-threshold=.7 \ --data-dir="./data"
Content of scripts/train_hico.sh: export CUDA_VISIBLE_DEVICES=$1 export PYTHONPATH=~/ws-vrd-master/src python src/xib/train.py ~/ws-vrd-master/config/train_hico.yaml

Launched as: scripts/train_hico.sh 1

I will have a look and check what could be going wrong 😉

— You are receiving this because you authored the thread. Reply to this email directly, view it on GitHub, or unsubscribe.