Where2comm icon indicating copy to clipboard operation
Where2comm copied to clipboard

Some questions about the implementation of spatial confidence-aware message fusion.

Open Little-Podi opened this issue 1 year ago • 2 comments

Thanks for your excellent work and wonderful open-source code! When I read the code, I found a possible issue about the implementation of MHA message fusion part. In this line (https://github.com/MediaBrain-SJTU/Where2comm/blob/efeee4a33b78c362b277d7f698ad4b3b2da0392c/opencood/models/fuse_modules/where2comm_attn.py#L100), you feed the spatial confidence map as an input of parameter quality_map. However, in this line (https://github.com/MediaBrain-SJTU/Where2comm/blob/efeee4a33b78c362b277d7f698ad4b3b2da0392c/opencood/models/fuse_modules/where2comm_attn.py#L76), the self.attn is implemented by the official class nn.MultiheadAttention, which doesn't have a parameter that corresponds to quality_map. Thus, it seems that the code cannot run smoothly. Maybe I miss something. Could you please give me some guides? Another minor question is, for the equation (3) in your paper image I think the three terms of input parameters correspond to the query, key and value of the multi-head attention mechanism, respectively. My confusion is, since the output of this formulation is the attetion weight, why we need the value to serve as an input? Maybe you express in this form for the sake of generality, so my main concern is about the code implementation. Thanks a lot for your reading. Looking forward to your reply!

Little-Podi avatar Nov 20 '22 01:11 Little-Podi