Failing to converge when set "hm encoder" as None

[wisematch]

When I remove the HM Encoder by set hm_proto = None, I found that the losses wouldn't decrease. Have you conduct the ablation study about the HM Encoder? Is the HM Encoder necessary in training? Thanks!

[wisematch]