How to generate visualizations of PAN

[Fritskee]

In the repository you show how PAN works for different actions (e.g. Yoyo). This is a very useful insight and I was wondering how I can generate similar visuals from data that is given to the model for inference.

[incognite-lab]

You have to modify the forward method in PAN module.

def forward(self, input, no_reshape=False): if not no_reshape: sample_len = (3 if self.modality in ['RGB', 'PA', 'Lite'] else 2) * self.new_length

        if self.modality == 'RGBDiff':
            sample_len = 3 * self.new_length
            input = self._get_diff(input)

        if self.modality == 'PA':
            base_out = self.PA(input.view((-1, sample_len) + input.size()[-2:]))
            base_out = self.base_model(base_out)
        elif self.modality == 'Lite':
            input = input.view((-1, sample_len) + input.size()[-2:])
            PA = self.PA(input)
            RGB = input.view((-1, self.data_length, sample_len) + input.size()[-2:])[:,0,:,:,:]
            base_out = torch.cat((RGB, PA), 1)
            base_out = self.base_model(base_out)
        else:
            base_out = self.base_model(input.view((-1, sample_len) + input.size()[-2:]))
    else:
        base_out = self.base_model(input)

    if self.has_VIP:
        return base_out

    if self.dropout > 0:
        base_out = self.new_fc(base_out)

    if not self.before_softmax:
        base_out = self.softmax(base_out)

    if self.reshape:
        if self.is_shift and self.temporal_pool:
            base_out = base_out.view((-1, self.num_segments // 2) + base_out.size()[1:])
        else:
            base_out = base_out.view((-1, self.num_segments) + base_out.size()[1:])
        output = self.consensus(base_out)
        return output.squeeze(1)

Rewrite it to return both base_out and PA. Then visualize it in main loop