A-Boundary-Based-Out-of-Distribution-Classifier-for-Generalized-Zero-Shot-Learning icon indicating copy to clipboard operation
A-Boundary-Based-Out-of-Distribution-Classifier-for-Generalized-Zero-Shot-Learning copied to clipboard

Published 20 hours ago • verified publisher icon Chenxingyu1990

Reproducing the OOD results

Open alexander-g opened this issue 4 years ago • 35 comments

Hi, I'm interested in how you have achieved the 2.5% OOD false positive rate on CUB from your paper. I've run your code but getting different results, here is the log output of the first 50 epochs:

Epoch: [10/1000], Step: [56/56], Reconstruction Loss: 0.0102 KL_Loss: 1.8555, attr_Recon Loss: 0.0193, cls_Loss: 4.8152, k1: 100.8736, k2: 101.0579, u: 0.1654 epoch 10, unseen 0.32142857142857145, seen 0.8509915014164305, ts 0.004034093115478754 tr 0.44816842675209045, H 0.007996209884418749 Epoch: [20/1000], Step: [56/56], Reconstruction Loss: 0.0085 KL_Loss: 1.9026, attr_Recon Loss: 0.0147, cls_Loss: 4.6595, k1: 101.0545, k2: 101.5119, u: 0.4802 epoch 20, unseen 0.6431940700808625, seen 0.6521246458923513, ts 0.009085135534405708 tr 0.4941219091415405, H 0.017842217083875896 Epoch: [30/1000], Step: [56/56], Reconstruction Loss: 0.0083 KL_Loss: 1.9305, attr_Recon Loss: 0.0125, cls_Loss: 4.5038, k1: 101.2280, k2: 102.0253, u: 0.5839 epoch 30, unseen 0.8507412398921833, seen 0.47705382436260624, ts 0.01209078449755907 tr 0.4203478693962097, H 0.023505464182759138 Epoch: [40/1000], Step: [56/56], Reconstruction Loss: 0.0080 KL_Loss: 1.8965, attr_Recon Loss: 0.0106, cls_Loss: 4.3186, k1: 101.4022, k2: 102.6113, u: 0.3912 epoch 40, unseen 0.941711590296496, seen 0.32577903682719545, ts 0.012429768219590187 tr 0.3073452115058899, H 0.02389323691892209 Epoch: [50/1000], Step: [56/56], Reconstruction Loss: 0.0079 KL_Loss: 1.9184, attr_Recon Loss: 0.0098, cls_Loss: 4.1350, k1: 101.6243, k2: 103.3307, u: 0.5312 epoch 50, unseen 0.9649595687331537, seen 0.24759206798866856, ts 0.012429768219590187 tr 0.24070467054843903, H 0.02363884889727425

The unseen OOD accuracy becomes larger during the training but the seen OOD accuracy goes down all the way to 0% and it stays that way for the rest of the training. The only differences are that I use pytorch1.0 and changed model_train.py slightly to add logging and removed the ZSL classifier checkpoints because I am only interested in OOD. No other changes. Log and diff files attached. Could you take a look?

Could you maybe provide pretrained checkpoints to reproduce the results?

log.txt model_train.py.diff.txt

alexander-g avatar Aug 25 '20 07:08 alexander-g

Hi, On CUB, we train the model by about 300 epochs. On arbitrary datasets, the cls_loss should be small enough. Otherwise, the OOD accuracy would be poor.

2020-08-25

chenxingyu_1990

发件人：alexander-g [email protected] 发送时间：2020-08-25 15:26 主题：[Chenxingyu1990/gzsl_svae] Reproducing the OOD results (#1) 收件人："Chenxingyu1990/gzsl_svae"[email protected] 抄送："Subscribed"[email protected]

Hi, I'm interested in how you have achieved the 2.5% OOD false positive rate on CUB from your paper. I've run your code but getting different results, here is the log output of the first 50 epochs: Epoch: [10/1000], Step: [56/56], Reconstruction Loss: 0.0102 KL_Loss: 1.8555, attr_Recon Loss: 0.0193, cls_Loss: 4.8152, k1: 100.8736, k2: 101.0579, u: 0.1654 epoch 10, unseen 0.32142857142857145, seen 0.8509915014164305, ts 0.004034093115478754 tr 0.44816842675209045, H 0.007996209884418749 Epoch: [20/1000], Step: [56/56], Reconstruction Loss: 0.0085 KL_Loss: 1.9026, attr_Recon Loss: 0.0147, cls_Loss: 4.6595, k1: 101.0545, k2: 101.5119, u: 0.4802 epoch 20, unseen 0.6431940700808625, seen 0.6521246458923513, ts 0.009085135534405708 tr 0.4941219091415405, H 0.017842217083875896 Epoch: [30/1000], Step: [56/56], Reconstruction Loss: 0.0083 KL_Loss: 1.9305, attr_Recon Loss: 0.0125, cls_Loss: 4.5038, k1: 101.2280, k2: 102.0253, u: 0.5839 epoch 30, unseen 0.8507412398921833, seen 0.47705382436260624, ts 0.01209078449755907 tr 0.4203478693962097, H 0.023505464182759138 Epoch: [40/1000], Step: [56/56], Reconstruction Loss: 0.0080 KL_Loss: 1.8965, attr_Recon Loss: 0.0106, cls_Loss: 4.3186, k1: 101.4022, k2: 102.6113, u: 0.3912 epoch 40, unseen 0.941711590296496, seen 0.32577903682719545, ts 0.012429768219590187 tr 0.3073452115058899, H 0.02389323691892209 Epoch: [50/1000], Step: [56/56], Reconstruction Loss: 0.0079 KL_Loss: 1.9184, attr_Recon Loss: 0.0098, cls_Loss: 4.1350, k1: 101.6243, k2: 103.3307, u: 0.5312 epoch 50, unseen 0.9649595687331537, seen 0.24759206798866856, ts 0.012429768219590187 tr 0.24070467054843903, H 0.02363884889727425 The unseen OOD accuracy becomes larger during the training but the seen OOD accuracy goes down all the way to 0% and it stays that way for the rest of the training. The only differences are that I use pytorch1.0 and changed model_train.py slightly to add logging and removed the ZSL classifier checkpoints because I am only interested in OOD. No other changes. Log and diff files attached. Could you take a look? Could you maybe provide pretrained checkpoints to reproduce the results? log.txt model_train.py.diff.txt — You are receiving this because you are subscribed to this thread. Reply to this email directly, view it on GitHub, or unsubscribe. 从网易chenxingyu_1990邮箱发来的超大附件 Checkpoint_300_classifier.pth.tar (153 KB, 2020-09-09 15:34 到期) 下载Checkpoint_300_attr_Dec.pth.tar (2.2 MB, 2020-09-09 15:34 到期) 下载Checkpoint_300_attr_Enc.pth.tar (2.2 MB, 2020-09-09 15:34 到期) 下载Checkpoint_300_Dec.pth.tar (12.4 MB, 2020-09-09 15:34 到期) 下载Checkpoint_300_Enc.pth.tar (12.4 MB, 2020-09-09 15:34 到期) 下载从网易chenxingyu_1990邮箱发来的超大附件 Checkpoint_300_classifier.pth.tar(153 KB,2020-09-09 15:34到期) 下载:http://fs.163.com/fs/display/?p=X-NETEASE-HUGE-ATTACHMENT&file=a6HXzWzAz6PqfL-JcMMfOjRG4Pi3uG6j-ellUMFlAIhozd-3FRe7uA0cBVO7t4UHXgVu6f_Z_lLxVFOy8R8n3w Checkpoint_300_attr_Dec.pth.tar(2.2 MB,2020-09-09 15:34到期) 下载:http://fs.163.com/fs/display/?p=X-NETEASE-HUGE-ATTACHMENT&file=a6HXzWzAz6PqfL-JcMMfOjRG4Pi3uG6j-ellUMFlAIimW0Z4UDUrZ7FTKETiS3f8rutDD_08qKYDBrlznrwRfA Checkpoint_300_attr_Enc.pth.tar(2.2 MB,2020-09-09 15:34到期) 下载:http://fs.163.com/fs/display/?p=X-NETEASE-HUGE-ATTACHMENT&file=a6HXzWzAz6PqfL-JcMMfOjRG4Pi3uG6j-ellUMFlAIgT7xLKKJPk1ziqnU7WorHAphuVultnv0aguNXfysSTlw Checkpoint_300_Dec.pth.tar(12.4 MB,2020-09-09 15:34到期) 下载:http://fs.163.com/fs/display/?p=X-NETEASE-HUGE-ATTACHMENT&file=a6HXzWzAz6PqfL-JcMMfOjRG4Pi3uG6j-ellUMFlAIjXAiUkv4_hOgkChMe3epHox6EmBqDzdMOxihkP6B3QsA Checkpoint_300_Enc.pth.tar(12.4 MB,2020-09-09 15:34到期) 下载:http://fs.163.com/fs/display/?p=X-NETEASE-HUGE-ATTACHMENT&file=a6HXzWzAz6PqfL-JcMMfOjRG4Pi3uG6j-ellUMFlAIidmZA_rod0D8531Y3n8FBdo6W-5LqNmMNpav9cC-lwIQ

Chenxingyu1990 avatar Aug 25 '20 07:08 Chenxingyu1990

@alexander-g @Chenxingyu1990 Sorry to disturb you. I meet the problem "ImportError: cannot import name 'general'" in "from common import general". Can you tell me the version of the "common" package? Thank you!

NanAlbert avatar Sep 13 '20 14:09 NanAlbert

@NanAlbert I had removed this import to get it running. It seems to be only for logging. I also removed all references to this module in model_train.py: general.logger_setup(...)

alexander-g avatar Sep 13 '20 14:09 alexander-g

@alexander-g Thank you so much! It solves the problem.

NanAlbert avatar Sep 14 '20 01:09 NanAlbert

@Chenxingyu1990 @alexander-g Can you please also share the pretrained classifier you used?

I mean the following weights:

ood_pretrained

Thanks a lot!

WilliamYi96 avatar Mar 15 '21 08:03 WilliamYi96

Hi, We have uploaded the pre-trained zsl models to github. Best,

-Xingyu

2021-03-15

chenxingyu_1990

发件人：Kai Yi @.> 发送时间：2021-03-15 16:49 主题：Re: [Chenxingyu1990/A-Boundary-Based-Out-of-Distribution-Classifier-for-Generalized-Zero-Shot-Learning] Reproducing the OOD results (#1) @.> 抄送："Xingyu @.@.>

@Chenxingyu1990 @alexander-g Can you please also share the pretrained classifier you used? I mean the following weights:

Thanks a lot! — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub, or unsubscribe.

Chenxingyu1990 avatar Mar 15 '21 09:03 Chenxingyu1990

I will have a try on it. Thanks a lot.

WilliamYi96 avatar Mar 15 '21 10:03 WilliamYi96

Here is the reproduced result on sun dataset:

In general, I have the following concern: @Chenxingyu1990 The reproduced results by using the current GitHub code are far lower than the reported one. Perhaps please double-check if you've submitted the wrong version, after the ICCV21 deadline. I guess the problem is mainly on the processing of seen acc, pretty low.

src/test.py:23: YAMLLoadWarning: calling yaml.load() without Loader=... is deprecated, as the default Loader is unsafe. Please read https://msg.pyyaml.org/load for full details.
  template = yaml.load(f)
[DIR INFO] save_path: /ibex/scratch/yik/OOD-ZSL/data/exp_out/SUN_cls1.0_512_64, 
[DIR INFO] config_copy: /ibex/scratch/yik/OOD-ZSL/data/exp_out/SUN_cls1.0_512_64/./config/sun.yaml
[DIR INFO] template_path: ./config/sun.yaml
norm features done!
Attr_Encoder(
  (fc1): Linear(in_features=102, out_features=512, bias=True)
  (relu1): ReLU()
  (fc2): Linear(in_features=512, out_features=64, bias=True)
  (fc3): Linear(in_features=512, out_features=1, bias=False)
)
Attr_Decoder(
  (fc1): Linear(in_features=64, out_features=512, bias=True)
  (fc2): Linear(in_features=512, out_features=102, bias=True)
  (relu): ReLU()
)
Encoder(
  (fc1): Linear(in_features=2048, out_features=512, bias=True)
  (relu2): ReLU()
  (fc2): Linear(in_features=512, out_features=64, bias=True)
  (fc3): Linear(in_features=512, out_features=1, bias=False)
)
Decoder(
  (fc1): Linear(in_features=64, out_features=512, bias=True)
  (fc2): Linear(in_features=512, out_features=2048, bias=True)
  (relu1): ReLU()
)
LINEAR_LOGSOFTMAX(
  (fc): Linear(in_features=64, out_features=717, bias=True)
  (logic): LogSoftmax()
)
############### Checkpoint I: Passed!
############### Checkpoint II: Passed!
epoch 50, unseen 0.9361554476058292, seen 0.15187911662146455, ts 0.5291666388511658  tr 0.09186046570539474, H 0.15654548067481977
epoch 100, unseen 0.9937543372657877, seen 0.04726850058117009, ts 0.5750000476837158  tr 0.03488372266292572, H 0.06577693219509995
epoch 150, unseen 0.9958362248438585, seen 0.02053467648198373, ts 0.5763888955116272  tr 0.014728682115674019, H 0.028723385597253976
epoch 200, unseen 0.9965301873698821, seen 0.01666020922123208, ts 0.5770833492279053  tr 0.012015503831207752, H 0.02354086058490095
epoch 250, unseen 0.9972241498959056, seen 0.01084850833010461, ts 0.5777777433395386  tr 0.00891472864896059, H 0.01755854120822857
epoch 300, unseen 0.9972241498959056, seen 0.006974041069352964, ts 0.5777777433395386  tr 0.0058139534667134285, H 0.011512065425810271
epoch 350, unseen 0.9979181124219292, seen 0.0065865943432778, ts 0.5784722566604614  tr 0.0058139534667134285, H 0.011512203121118456
epoch 400, unseen 0.9979181124219292, seen 0.006199147617202635, ts 0.5784722566604614  tr 0.005038759671151638, H 0.009990497876340518
epoch 450, unseen 0.9986120749479528, seen 0.005424254165052305, ts 0.5791666507720947  tr 0.0042635658755898476, H 0.008464817347218895
epoch 500, unseen 0.9986120749479528, seen 0.005036807438977141, ts 0.5791666507720947  tr 0.0038759689778089523, H 0.0077004043552271065
epoch 550, unseen 0.9986120749479528, seen 0.00581170089112747, ts 0.5791666507720947  tr 0.004651162773370743, H 0.009228215319322264
epoch 600, unseen 0.9986120749479528, seen 0.005036807438977141, ts 0.5791666507720947  tr 0.0038759689778089523, H 0.0077004043552271065
epoch 650, unseen 0.9986120749479528, seen 0.004649360712901976, ts 0.5791666507720947  tr 0.0038759689778089523, H 0.0077004043552271065
epoch 700, unseen 0.9986120749479528, seen 0.004649360712901976, ts 0.5791666507720947  tr 0.0038759689778089523, H 0.0077004043552271065
epoch 750, unseen 0.9986120749479528, seen 0.004261913986826811, ts 0.5791666507720947  tr 0.003488372080028057, H 0.0069349743176326745
epoch 800, unseen 0.9986120749479528, seen 0.004261913986826811, ts 0.5791666507720947  tr 0.003488372080028057, H 0.0069349743176326745
epoch 850, unseen 0.9986120749479528, seen 0.0030995738086013174, ts 0.5791666507720947  tr 0.0023255813866853714, H 0.004632561118222641
epoch 900, unseen 0.9986120749479528, seen 0.003487020534676482, ts 0.5791666507720947  tr 0.0027131782844662666, H 0.00540105442253606
epoch 950, unseen 0.9986120749479528, seen 0.0038744672607516468, ts 0.5791666507720947  tr 0.003100775182247162, H 0.006168525203327391
epoch 1000, unseen 0.9986120749479528, seen 0.0038744672607516468, ts 0.5791666507720947  tr 0.003100775182247162, H 0.006168525203327391
epoch 1050, unseen 0.9986120749479528, seen 0.0038744672607516468, ts 0.5791666507720947  tr 0.003100775182247162, H 0.006168525203327391
epoch 1100, unseen 0.9986120749479528, seen 0.0030995738086013174, ts 0.5791666507720947  tr 0.0023255813866853714, H 0.004632561118222641
epoch 1150, unseen 0.9986120749479528, seen 0.003487020534676482, ts 0.5791666507720947  tr 0.0027131782844662666, H 0.00540105442253606
epoch 1200, unseen 0.9986120749479528, seen 0.0038744672607516468, ts 0.5791666507720947  tr 0.003100775182247162, H 0.006168525203327391
epoch 1250, unseen 0.9986120749479528, seen 0.0038744672607516468, ts 0.5791666507720947  tr 0.003100775182247162, H 0.006168525203327391
epoch 1300, unseen 0.9986120749479528, seen 0.003487020534676482, ts 0.5791666507720947  tr 0.0027131782844662666, H 0.00540105442253606
epoch 1350, unseen 0.9986120749479528, seen 0.0030995738086013174, ts 0.5791666507720947  tr 0.0023255813866853714, H 0.004632561118222641
epoch 1400, unseen 0.9986120749479528, seen 0.003487020534676482, ts 0.5791666507720947  tr 0.0027131782844662666, H 0.00540105442253606
epoch 1450, unseen 0.9986120749479528, seen 0.003487020534676482, ts 0.5791666507720947  tr 0.0027131782844662666, H 0.00540105442253606
epoch 1500, unseen 0.9986120749479528, seen 0.0030995738086013174, ts 0.5791666507720947  tr 0.0023255813866853714, H 0.004632561118222641
epoch 1550, unseen 0.9986120749479528, seen 0.003487020534676482, ts 0.5791666507720947  tr 0.0027131782844662666, H 0.00540105442253606
epoch 1600, unseen 0.9986120749479528, seen 0.003487020534676482, ts 0.5791666507720947  tr 0.0027131782844662666, H 0.00540105442253606
epoch 1650, unseen 0.9986120749479528, seen 0.003487020534676482, ts 0.5791666507720947  tr 0.0027131782844662666, H 0.00540105442253606
epoch 1700, unseen 0.9986120749479528, seen 0.003487020534676482, ts 0.5791666507720947  tr 0.0027131782844662666, H 0.00540105442253606
epoch 1750, unseen 0.9986120749479528, seen 0.0038744672607516468, ts 0.5791666507720947  tr 0.0027131782844662666, H 0.00540105442253606
epoch 1800, unseen 0.9986120749479528, seen 0.003487020534676482, ts 0.5791666507720947  tr 0.0027131782844662666, H 0.00540105442253606
epoch 1850, unseen 0.9986120749479528, seen 0.003487020534676482, ts 0.5791666507720947  tr 0.0027131782844662666, H 0.00540105442253606
epoch 1900, unseen 0.9986120749479528, seen 0.003487020534676482, ts 0.5791666507720947  tr 0.0027131782844662666, H 0.00540105442253606
epoch 1950, unseen 0.9986120749479528, seen 0.0030995738086013174, ts 0.5791666507720947  tr 0.0023255813866853714, H 0.004632561118222641

WilliamYi96 avatar Mar 15 '21 13:03 WilliamYi96

Could you show the training logs?

Chenxingyu1990 avatar Mar 15 '21 13:03 Chenxingyu1990

If the cls_loss is good enough, the seen acc shouldn't be that poor.

Chenxingyu1990 avatar Mar 15 '21 13:03 Chenxingyu1990

Here is the training log for cub and sun:

ood_rep_2.14705478.txt

WilliamYi96 avatar Mar 15 '21 13:03 WilliamYi96

It seems that the cls_loss is quite low in Epoch 1996. If you set n = 0.9, what the unseen and seen results will be?

threshold = model_train_obj.search_thres_by_traindata(test_epoch, dataset = data, n = 0.95)

Chenxingyu1990 avatar Mar 15 '21 13:03 Chenxingyu1990

I re-run the codes on SUN.

epoch 50, unseen 0.9056210964607911, seen 0.9434327779930259, ts 0.5111110806465149 tr 0.9054263830184937, H 0.6533867962809392 epoch 100, unseen 0.9798750867453158, seen 0.9438202247191011, ts 0.5659722089767456 tr 0.9364340901374817, H 0.7055290567435776 epoch 150, unseen 0.9882026370575989, seen 0.9469197985277025, ts 0.5708333253860474 tr 0.9441860318183899, H 0.7115061976712177

Epoch: [143/2000], Step: [49/101], Reconstruction Loss: 0.0083 KL_Loss: 0.7925, attr_Recon Loss: 0.0163, cls_Loss: 3.9392, k1: 132.8209, k2: 148.6010, u: 0.5033 Epoch: [143/2000], Step: [99/101], Reconstruction Loss: 0.0082 KL_Loss: 0.8000, attr_Recon Loss: 0.0162, cls_Loss: 3.9327, k1: 132.9910, k2: 148.2477, u: 0.6954 Epoch: [144/2000], Step: [49/101], Reconstruction Loss: 0.0081 KL_Loss: 0.8031, attr_Recon Loss: 0.0161, cls_Loss: 3.8994, k1: 133.2533, k2: 149.5416, u: 0.4318 Epoch: [144/2000], Step: [99/101], Reconstruction Loss: 0.0082 KL_Loss: 0.8020, attr_Recon Loss: 0.0159, cls_Loss: 3.9127, k1: 133.5501, k2: 148.9448, u: 0.5117 Epoch: [145/2000], Step: [49/101], Reconstruction Loss: 0.0081 KL_Loss: 0.7877, attr_Recon Loss: 0.0160, cls_Loss: 3.8742, k1: 134.1663, k2: 149.5072, u: 0.4976 Epoch: [145/2000], Step: [99/101], Reconstruction Loss: 0.0082 KL_Loss: 0.7910, attr_Recon Loss: 0.0164, cls_Loss: 3.8728, k1: 134.4247, k2: 150.1674, u: 0.4874 Epoch: [146/2000], Step: [49/101], Reconstruction Loss: 0.0081 KL_Loss: 0.7892, attr_Recon Loss: 0.0155, cls_Loss: 3.8738, k1: 134.6434, k2: 150.4792, u: 0.6007 Epoch: [146/2000], Step: [99/101], Reconstruction Loss: 0.0082 KL_Loss: 0.7889, attr_Recon Loss: 0.0158, cls_Loss: 3.8880, k1: 135.1180, k2: 150.5984, u: 0.6251 Epoch: [147/2000], Step: [49/101], Reconstruction Loss: 0.0082 KL_Loss: 0.7883, attr_Recon Loss: 0.0161, cls_Loss: 3.8712, k1: 135.7278, k2: 150.5138, u: 0.7333 Epoch: [147/2000], Step: [99/101], Reconstruction Loss: 0.0082 KL_Loss: 0.7892, attr_Recon Loss: 0.0154, cls_Loss: 3.8529, k1: 135.8707, k2: 150.8152, u: 0.6345 Epoch: [148/2000], Step: [49/101], Reconstruction Loss: 0.0081 KL_Loss: 0.7910, attr_Recon Loss: 0.0159, cls_Loss: 3.8430, k1: 136.3832, k2: 151.4635, u: 0.6583 Epoch: [148/2000], Step: [99/101], Reconstruction Loss: 0.0082 KL_Loss: 0.7888, attr_Recon Loss: 0.0154, cls_Loss: 3.8348, k1: 136.6835, k2: 151.9745, u: 0.4627 Epoch: [149/2000], Step: [49/101], Reconstruction Loss: 0.0081 KL_Loss: 0.7823, attr_Recon Loss: 0.0151, cls_Loss: 3.8611, k1: 136.8273, k2: 152.2602, u: 0.6280 Epoch: [149/2000], Step: [99/101], Reconstruction Loss: 0.0083 KL_Loss: 0.7782, attr_Recon Loss: 0.0159, cls_Loss: 3.7808, k1: 137.5218, k2: 152.3071, u: 0.6406 Epoch: [150/2000], Step: [49/101], Reconstruction Loss: 0.0082 KL_Loss: 0.7780, attr_Recon Loss: 0.0156, cls_Loss: 3.7797, k1: 137.5836, k2: 152.6961, u: 0.5039 Epoch: [150/2000], Step: [99/101], Reconstruction Loss: 0.0082 KL_Loss: 0.7841, attr_Recon Loss: 0.0152, cls_Loss: 3.8180, k1: 137.9278, k2: 152.9695, u: 0.4442 Epoch: [151/2000], Step: [49/101], Reconstruction Loss: 0.0082 KL_Loss: 0.7865, attr_Recon Loss: 0.0161, cls_Loss: 3.7696, k1: 138.5943, k2: 153.5411, u: 0.5678 Epoch: [151/2000], Step: [99/101], Reconstruction Loss: 0.0081 KL_Loss: 0.7818, attr_Recon Loss: 0.0156, cls_Loss: 3.7535, k1: 138.4066, k2: 154.3065, u: 0.6349 Epoch: [152/2000], Step: [49/101], Reconstruction Loss: 0.0083 KL_Loss: 0.7695, attr_Recon Loss: 0.0156, cls_Loss: 3.7289, k1: 139.6448, k2: 154.1946, u: 0.6280 Epoch: [152/2000], Step: [99/101], Reconstruction Loss: 0.0082 KL_Loss: 0.7775, attr_Recon Loss: 0.0150, cls_Loss: 3.7970, k1: 139.9114, k2: 153.7278, u: 0.4561 Epoch: [153/2000], Step: [49/101], Reconstruction Loss: 0.0084 KL_Loss: 0.7686, attr_Recon Loss: 0.0154, cls_Loss: 3.7515, k1: 140.6187, k2: 154.8667, u: 0.6511 Epoch: [153/2000], Step: [99/101], Reconstruction Loss: 0.0082 KL_Loss: 0.7712, attr_Recon Loss: 0.0154, cls_Loss: 3.7436, k1: 140.2935, k2: 154.6846, u: 0.5469 Epoch: [154/2000], Step: [49/101], Reconstruction Loss: 0.0082 KL_Loss: 0.7694, attr_Recon Loss: 0.0150, cls_Loss: 3.7061, k1: 140.9117, k2: 155.6408, u: 0.2820 Epoch: [154/2000], Step: [99/101], Reconstruction Loss: 0.0082 KL_Loss: 0.7715, attr_Recon Loss: 0.0158, cls_Loss: 3.6934, k1: 141.3717, k2: 155.7104, u: 0.5040 Epoch: [155/2000], Step: [49/101], Reconstruction Loss: 0.0082 KL_Loss: 0.7711, attr_Recon Loss: 0.0152, cls_Loss: 3.7070, k1: 141.5082, k2: 156.3288, u: 0.6587

Chenxingyu1990 avatar Mar 15 '21 13:03 Chenxingyu1990

Maybe a deeper thought on what exactly is going on is required. Because I honestly follow the steps/code in this Github repo. Or you can provide a way to get the results like yours.

WilliamYi96 avatar Mar 15 '21 14:03 WilliamYi96

I meet the same problem, on all of these five datasets, when unseen grows up seen keeps low level @Chenxingyu1990

time-iter avatar Mar 16 '21 01:03 time-iter

That's weird. I forked the repo and run the codes again. The results seems all right. I use n=0.95 when searching the threshold, which means the seen accuracy should be around 95%. Could provide more details?

threshold = model_train_obj.search_thres_by_traindata(test_epoch, dataset = data, n = 0.95)

Chenxingyu1990 avatar Mar 16 '21 02:03 Chenxingyu1990

awa1.log sun.log flo.log awa2.log cub.log

these are training logs, I didn't modify your code except the data path and save every

time-iter avatar Mar 16 '21 02:03 time-iter

Could you show the visualization results on awa1 or flo? First uncomment line 199 to 205 in datasets.py. Then use function model_train_obj.testing(epoch, sample_rate = 5). Line 143 in test.py

Chenxingyu1990 avatar Mar 16 '21 02:03 Chenxingyu1990

awa1 this is the visualization results on awa1

time-iter avatar Mar 16 '21 03:03 time-iter

We train on awa1 for 60 epochs. It is not the best result, but still reasonable.

epoch 60, unseen 0.8461132606401688, seen 0.951401492236338, ts 0.5317897796630859 tr 0.948604941368103, H 0.6815187944010915

[58/500], Step: [149/194], Reconstruction Loss: 0.0081 KL_Loss: 0.5129, attr_Recon Loss: 0.0020, cls_Loss: 0.2026, k1: 186.6495, k2: 229.4482, u: 0.7248 Epoch: [59/500], Step: [49/194], Reconstruction Loss: 0.0083 KL_Loss: 0.5084, attr_Recon Loss: 0.0019, cls_Loss: 0.2053, k1: 188.4473, k2: 230.9966, u: 0.7978 Epoch: [59/500], Step: [99/194], Reconstruction Loss: 0.0081 KL_Loss: 0.5116, attr_Recon Loss: 0.0021, cls_Loss: 0.2033, k1: 187.5344, k2: 230.9939, u: 0.6919 Epoch: [59/500], Step: [149/194], Reconstruction Loss: 0.0083 KL_Loss: 0.5009, attr_Recon Loss: 0.0019, cls_Loss: 0.1952, k1: 190.8037, k2: 232.9591, u: 0.7332 Epoch: [60/500], Step: [49/194], Reconstruction Loss: 0.0084 KL_Loss: 0.4953, attr_Recon Loss: 0.0019, cls_Loss: 0.1875, k1: 192.4006, k2: 234.4215, u: 0.7072 Epoch: [60/500], Step: [99/194], Reconstruction Loss: 0.0083 KL_Loss: 0.4961, attr_Recon Loss: 0.0018, cls_Loss: 0.1724, k1: 192.5414, k2: 234.4863, u: 0.7636 Epoch: [60/500], Step: [149/194], Reconstruction Loss: 0.0083 KL_Loss: 0.5010, attr_Recon Loss: 0.0020, cls_Loss: 0.1958, k1: 194.0911, k2: 235.7670, u: 0.7953 Epoch: [61/500], Step: [49/194], Reconstruction Loss: 0.0084 KL_Loss: 0.4878, attr_Recon Loss: 0.0019, cls_Loss: 0.1602, k1: 196.4712, k2: 237.4183, u: 0.7710 Epoch: [61/500], Step: [99/194], Reconstruction Loss: 0.0083 KL_Loss: 0.5024, attr_Recon Loss: 0.0021, cls_Loss: 0.1982, k1: 195.3680, k2: 238.2173, u: 0.6118 Epoch: [61/500], Step: [149/194], Reconstruction Loss: 0.0083 KL_Loss: 0.4938, attr_Recon Loss: 0.0019, cls_Loss: 0.1616, k1: 197.5031, k2: 239.8863, u: 0.7386

image

Chenxingyu1990 avatar Mar 16 '21 03:03 Chenxingyu1990

could you share the python env or conda env of this project

time-iter avatar Mar 16 '21 08:03 time-iter

Sorry for disturbing you. I downloaded your repo and do the same work on AWA1, but I did not get a good result as your's. My best performance for H is around 61 and during my test, my best performance is seen around 80 and unseen around 80, if the unseen goes up, then the seen goes down. And I get the visualization as you did(60 epoch), but it's obvious that I did not get a good result as you. Hoping to hear more instruction from you, thank you very much! VIS

I also noticed while training, my cls_loss is much higher and my KL_loss have a considerable amount higher than yours.

Epoch: [58/500], Step: [49/155], Reconstruction Loss: 0.0082 KL_Loss: 0.5622, attr_Recon Loss: 0.0027, cls_Loss: 0.4665, k1: 176.9186, k2: 192.8341, u: 0.6714 Epoch: [58/500], Step: [99/155], Reconstruction Loss: 0.0084 KL_Loss: 0.5558, attr_Recon Loss: 0.0025, cls_Loss: 0.4515, k1: 177.5039, k2: 193.9994, u: 0.6445 Epoch: [58/500], Step: [149/155], Reconstruction Loss: 0.0082 KL_Loss: 0.5551, attr_Recon Loss: 0.0026, cls_Loss: 0.4557, k1: 177.5148, k2: 194.7354, u: 0.6674 Epoch: [59/500], Step: [49/155], Reconstruction Loss: 0.0082 KL_Loss: 0.5536, attr_Recon Loss: 0.0026, cls_Loss: 0.4468, k1: 178.7918, k2: 195.4286, u: 0.6548 Epoch: [59/500], Step: [99/155], Reconstruction Loss: 0.0082 KL_Loss: 0.5536, attr_Recon Loss: 0.0026, cls_Loss: 0.4253, k1: 179.6972, k2: 196.5926, u: 0.8012 Epoch: [59/500], Step: [149/155], Reconstruction Loss: 0.0082 KL_Loss: 0.5563, attr_Recon Loss: 0.0026, cls_Loss: 0.4440, k1: 180.2421, k2: 197.5156, u: 0.7522 Epoch: [60/500], Step: [49/155], Reconstruction Loss: 0.0083 KL_Loss: 0.5438, attr_Recon Loss: 0.0025, cls_Loss: 0.4153, k1: 181.1044, k2: 197.5611, u: 0.6848 Epoch: [60/500], Step: [99/155], Reconstruction Loss: 0.0084 KL_Loss: 0.5456, attr_Recon Loss: 0.0024, cls_Loss: 0.4022, k1: 182.9650, k2: 199.2755, u: 0.7074 Epoch: [60/500], Step: [149/155], Reconstruction Loss: 0.0080 KL_Loss: 0.5443, attr_Recon Loss: 0.0022, cls_Loss: 0.3773, k1: 182.4378, k2: 200.2141, u: 0.8126 Epoch: [61/500], Step: [49/155], Reconstruction Loss: 0.0082 KL_Loss: 0.5401, attr_Recon Loss: 0.0023, cls_Loss: 0.4041, k1: 182.9166, k2: 201.1226, u: 0.6560 Epoch: [61/500], Step: [99/155], Reconstruction Loss: 0.0083 KL_Loss: 0.5396, attr_Recon Loss: 0.0023, cls_Loss: 0.3788, k1: 185.2990, k2: 201.9901, u: 0.7349 Epoch: [61/500], Step: [149/155], Reconstruction Loss: 0.0082 KL_Loss: 0.5405, attr_Recon Loss: 0.0022, cls_Loss: 0.3608, k1: 186.4567, k2: 202.5472, u: 0.6965 Epoch: [62/500], Step: [49/155], Reconstruction Loss: 0.0082 KL_Loss: 0.5349, attr_Recon Loss: 0.0024, cls_Loss: 0.3798, k1: 186.9906, k2: 202.8341, u: 0.7660

ZiyangW2000 avatar Mar 16 '21 14:03 ZiyangW2000

Thank you for your analysis. Would you please export your environments like these:
conda env export > environment.yml pip freeze > requirements.txt (both conda and pip) That would help a lot, really sorry for disturbing!

ZiyangW2000 avatar Mar 17 '21 02:03 ZiyangW2000

environments.txt

Chenxingyu1990 avatar Mar 17 '21 03:03 Chenxingyu1990

I guess the problems maybe in S-VAE or Sinkhorn iteration for calculating EMD. As these codes are provided officially, I doubt the problems are caused by python env.

@Chenxingyu1990 what do you mean exactly by saying this? For the previous experiments, I believe using the same envs, e.g, python, torch, torchvision, will not lead to any other problems. I do find the results are also wield. But could you please ellabrate what exactly is going on for the reproduce? Almostly we all can not reproduce. I don't mean the reported results can not be reproduced, but it will be a good start to look further into this paper/code. Otherwise, it will mislead the direction of ZSL.

Looking forward to your reply.

Kind regards, Kai

WilliamYi96 avatar Mar 19 '21 02:03 WilliamYi96

Actually, I would like to help you to reproduce the results. I explained that I guess the problems maybe in S-VAE or EMD. I'm not pretty sure since I do not meet the problems. I give some suggestions all based on your experimental details. By using the same codes, you and @ZiyangW2000, and I have totally different results. So I think the problems may be caused by different env settings, which leads to vMF sampling does not work very well. I also want to find the problems you met, maybe you can provide more details. For example, does the vMF sampling work in your env? You can try to generate some 3D points for visualization. Also, you can visualize the latent spaces on train set ant test seen set to see if the manifold of each class seems like a circle. Both EMD loss and cls_loss encourage the latent variables of each seen class to be concentrated.

Chenxingyu1990 avatar Mar 19 '21 02:03 Chenxingyu1990

I think I've found the true reason. The reproducing issue is caused by the Datasets!!! Link 1 has two kinds of dataset splits. The first one called split-V2 is different from the previous one. Currently, most previous GZSL algorithms use the previous split (Link 2), including f-CLSGWAN, CADA-VAE, COSMO, et al., and ours.

Link 1: https://www.mpi-inf.mpg.de/departments/computer-vision-and-machine-learning/research/zero-shot-learning/zero-shot-learning-the-good-the-bad-and-the-ugly

Link 2: The previous dataset splits link: wget http://chechiklab.biu.ac.il/~yuvval/COSMO/data.zip

@ZiyangW2000 's results have no problem on split-V2. I also get similar results.

The following link describes the details. https://drive.google.com/file/d/1p9gtkuHCCCyjkyezSarCw-1siCSXUykH/view

Interestingly, I visit some previous git repos. I find the reproducing issues also may be caused by the dataset problem.

https://github.com/akku1506/Feature-Generating-Networks-for-ZSL/issues https://github.com/stevehuanghe/GDAN/issues/3

Chenxingyu1990 avatar Mar 19 '21 12:03 Chenxingyu1990

Thanks for your sincerely help，the problem has been solved. I have a suggestion that you put your environment and datasets links in the readme document. By doing this some misunderstanding and trouble can be avoided.

time-iter avatar Mar 20 '21 06:03 time-iter

By the way, I'm curious why the author put much more inferior testing results of CUB and SUN in the paper. The re-run result of SUN outperforms that in the paper with 30% on H.

I re-run the codes on SUN.

epoch 50, unseen 0.9056210964607911, seen 0.9434327779930259, ts 0.5111110806465149 tr 0.9054263830184937, H 0.6533867962809392 epoch 100, unseen 0.9798750867453158, seen 0.9438202247191011, ts 0.5659722089767456 tr 0.9364340901374817, H 0.7055290567435776 epoch 150, unseen 0.9882026370575989, seen 0.9469197985277025, ts 0.5708333253860474 tr 0.9441860318183899, H 0.7115061976712177

Epoch: [143/2000], Step: [49/101], Reconstruction Loss: 0.0083 KL_Loss: 0.7925, attr_Recon Loss: 0.0163, cls_Loss: 3.9392, k1: 132.8209, k2: 148.6010, u: 0.5033 Epoch: [143/2000], Step: [99/101], Reconstruction Loss: 0.0082 KL_Loss: 0.8000, attr_Recon Loss: 0.0162, cls_Loss: 3.9327, k1: 132.9910, k2: 148.2477, u: 0.6954 Epoch: [144/2000], Step: [49/101], Reconstruction Loss: 0.0081 KL_Loss: 0.8031, attr_Recon Loss: 0.0161, cls_Loss: 3.8994, k1: 133.2533, k2: 149.5416, u: 0.4318 Epoch: [144/2000], Step: [99/101], Reconstruction Loss: 0.0082 KL_Loss: 0.8020, attr_Recon Loss: 0.0159, cls_Loss: 3.9127, k1: 133.5501, k2: 148.9448, u: 0.5117 Epoch: [145/2000], Step: [49/101], Reconstruction Loss: 0.0081 KL_Loss: 0.7877, attr_Recon Loss: 0.0160, cls_Loss: 3.8742, k1: 134.1663, k2: 149.5072, u: 0.4976 Epoch: [145/2000], Step: [99/101], Reconstruction Loss: 0.0082 KL_Loss: 0.7910, attr_Recon Loss: 0.0164, cls_Loss: 3.8728, k1: 134.4247, k2: 150.1674, u: 0.4874 Epoch: [146/2000], Step: [49/101], Reconstruction Loss: 0.0081 KL_Loss: 0.7892, attr_Recon Loss: 0.0155, cls_Loss: 3.8738, k1: 134.6434, k2: 150.4792, u: 0.6007 Epoch: [146/2000], Step: [99/101], Reconstruction Loss: 0.0082 KL_Loss: 0.7889, attr_Recon Loss: 0.0158, cls_Loss: 3.8880, k1: 135.1180, k2: 150.5984, u: 0.6251 Epoch: [147/2000], Step: [49/101], Reconstruction Loss: 0.0082 KL_Loss: 0.7883, attr_Recon Loss: 0.0161, cls_Loss: 3.8712, k1: 135.7278, k2: 150.5138, u: 0.7333 Epoch: [147/2000], Step: [99/101], Reconstruction Loss: 0.0082 KL_Loss: 0.7892, attr_Recon Loss: 0.0154, cls_Loss: 3.8529, k1: 135.8707, k2: 150.8152, u: 0.6345 Epoch: [148/2000], Step: [49/101], Reconstruction Loss: 0.0081 KL_Loss: 0.7910, attr_Recon Loss: 0.0159, cls_Loss: 3.8430, k1: 136.3832, k2: 151.4635, u: 0.6583 Epoch: [148/2000], Step: [99/101], Reconstruction Loss: 0.0082 KL_Loss: 0.7888, attr_Recon Loss: 0.0154, cls_Loss: 3.8348, k1: 136.6835, k2: 151.9745, u: 0.4627 Epoch: [149/2000], Step: [49/101], Reconstruction Loss: 0.0081 KL_Loss: 0.7823, attr_Recon Loss: 0.0151, cls_Loss: 3.8611, k1: 136.8273, k2: 152.2602, u: 0.6280 Epoch: [149/2000], Step: [99/101], Reconstruction Loss: 0.0083 KL_Loss: 0.7782, attr_Recon Loss: 0.0159, cls_Loss: 3.7808, k1: 137.5218, k2: 152.3071, u: 0.6406 Epoch: [150/2000], Step: [49/101], Reconstruction Loss: 0.0082 KL_Loss: 0.7780, attr_Recon Loss: 0.0156, cls_Loss: 3.7797, k1: 137.5836, k2: 152.6961, u: 0.5039 Epoch: [150/2000], Step: [99/101], Reconstruction Loss: 0.0082 KL_Loss: 0.7841, attr_Recon Loss: 0.0152, cls_Loss: 3.8180, k1: 137.9278, k2: 152.9695, u: 0.4442 Epoch: [151/2000], Step: [49/101], Reconstruction Loss: 0.0082 KL_Loss: 0.7865, attr_Recon Loss: 0.0161, cls_Loss: 3.7696, k1: 138.5943, k2: 153.5411, u: 0.5678 Epoch: [151/2000], Step: [99/101], Reconstruction Loss: 0.0081 KL_Loss: 0.7818, attr_Recon Loss: 0.0156, cls_Loss: 3.7535, k1: 138.4066, k2: 154.3065, u: 0.6349 Epoch: [152/2000], Step: [49/101], Reconstruction Loss: 0.0083 KL_Loss: 0.7695, attr_Recon Loss: 0.0156, cls_Loss: 3.7289, k1: 139.6448, k2: 154.1946, u: 0.6280 Epoch: [152/2000], Step: [99/101], Reconstruction Loss: 0.0082 KL_Loss: 0.7775, attr_Recon Loss: 0.0150, cls_Loss: 3.7970, k1: 139.9114, k2: 153.7278, u: 0.4561 Epoch: [153/2000], Step: [49/101], Reconstruction Loss: 0.0084 KL_Loss: 0.7686, attr_Recon Loss: 0.0154, cls_Loss: 3.7515, k1: 140.6187, k2: 154.8667, u: 0.6511 Epoch: [153/2000], Step: [99/101], Reconstruction Loss: 0.0082 KL_Loss: 0.7712, attr_Recon Loss: 0.0154, cls_Loss: 3.7436, k1: 140.2935, k2: 154.6846, u: 0.5469 Epoch: [154/2000], Step: [49/101], Reconstruction Loss: 0.0082 KL_Loss: 0.7694, attr_Recon Loss: 0.0150, cls_Loss: 3.7061, k1: 140.9117, k2: 155.6408, u: 0.2820 Epoch: [154/2000], Step: [99/101], Reconstruction Loss: 0.0082 KL_Loss: 0.7715, attr_Recon Loss: 0.0158, cls_Loss: 3.6934, k1: 141.3717, k2: 155.7104, u: 0.5040 Epoch: [155/2000], Step: [49/101], Reconstruction Loss: 0.0082 KL_Loss: 0.7711, attr_Recon Loss: 0.0152, cls_Loss: 3.7070, k1: 141.5082, k2: 156.3288, u: 0.6587

Bang2hen1iu avatar Sep 19 '22 06:09 Bang2hen1iu

I think I've found the true reason. The reproducing issue is caused by the Datasets!!! Link 1 has two kinds of dataset splits. The first one called split-V2 is different from the previous one. Currently, most previous GZSL algorithms use the previous split (Link 2), including f-CLSGWAN, CADA-VAE, COSMO, et al., and ours.

Link 1: https://www.mpi-inf.mpg.de/departments/computer-vision-and-machine-learning/research/zero-shot-learning/zero-shot-learning-the-good-the-bad-and-the-ugly

Link 2: The previous dataset splits link: wget http://chechiklab.biu.ac.il/~yuvval/COSMO/data.zip

@ZiyangW2000 's results have no problem on split-V2. I also get similar results.

The following link describes the details. https://drive.google.com/file/d/1p9gtkuHCCCyjkyezSarCw-1siCSXUykH/view

Interestingly, I visit some previous git repos. I find the reproducing issues also may be caused by the dataset problem.

https://github.com/akku1506/Feature-Generating-Networks-for-ZSL/issues stevehuanghe/GDAN#3

Could you show your results on split-V2? I run your code on split-V2 and I had a really bad accuracy on the ood classification just as @WilliamYi96. The dataset split will have such a big influence on the classification and is it correct? I've retrained the model on the split-V2 but the ood accuracy is not satisfying.

LixDemon avatar Dec 21 '22 07:12 LixDemon