loss_contrast

[Summer77723]

Hello. Thanks for your great work. I have some questions about the code,what is the meaning of with_embed,Why is loss_contrast not used? 1.if with_embed is True: return loss + self.loss_weight * loss_contrast

    return loss + 0 * loss_contrast  # just a trick to avoid errors in distributed training

2.        if is_distributed():
            import torch.distributed as dist
            def reduce_tensor(inp):
                """
                Reduce the loss from all processes so that 
                process with rank 0 has the averaged results.
                """
                world_size = get_world_size()
                if world_size < 2:
                    return inp
                with torch.no_grad():
                    reduced_inp = inp
                    dist.reduce(reduced_inp, dst=0)
                return reduced_inp
   
            loss = self.pixel_loss(outputs, targets, with_embed=with_embed)  
   
            backward_loss = loss
            display_loss = reduce_tensor(backward_loss) / get_world_size()
        else:
            backward_loss = display_loss = self.pixel_loss(outputs, targets)
   

Looking forword to your reply!

[tfzhou]

Hi, @Summer77723, our code has a warmup stage, in which the cotrastive loss is not applied, i.e., the weight of contrastive loss is zero.

[Summer77723]

Thank you for your reply.