For drmm model, dense_output should be flipped at dim=-1?

[wangcongcong123]

Hi,

In drmm model,

Should it be x = torch.einsum('bl,bl->b', torch.flip(dense_output,(-1,)), attention_probs)

Instead of x = torch.einsum('bl,bl->b', dense_output, attention_probs)

After I revise this, I got the training loss reduction much faster than this as in https://github.com/NTMC-Community/MatchZoo-py/blob/master/tutorials/ranking/drmm.ipynb

Below are my training logs:

Epoch 1/10: 100%|██████████| 319/319 [02:36<00:00, 6.02it/s, loss=2.167][Iter-319 Loss-2.134]: Validation: normalized_discounted_cumulative_gain@3(0.0): 0.5825 - normalized_discounted_cumulative_gain@5(0.0): 0.6421 - mean_average_precision(0.0): 0.6019 Epoch 1/10: 100%|██████████| 319/319 [02:44<00:00, 1.93it/s, loss=2.167] Epoch 2/10: 100%|█████████▉| 318/319 [01:11<00:00, 6.27it/s, loss=1.729][Iter-638 Loss-1.776]: Epoch 2/10: 100%|██████████| 319/319 [01:19<00:00, 4.02it/s, loss=0.877] 0%| | 0/319 [00:00<?, ?it/s] Validation: normalized_discounted_cumulative_gain@3(0.0): 0.5726 - normalized_discounted_cumulative_gain@5(0.0): 0.6363 - mean_average_precision(0.0): 0.589

[wangcongcong123]

My understand is that query is padded to the left while match_hist is to the right. When calculating the einsum between them, should this be consistent ?

[faneshion]

Thanks for your advices, we will check it right now!

[Chriskuei]

Hi @wangcongcong123, the pad_mode of text_left and match_histogram is consistent. https://github.com/NTMC-Community/MatchZoo-py/blob/49548ad4dd7da4c890ac786a09d9df9172c3af47/matchzoo/dataloader/callbacks/padding.py#L226-L239