How to use Relative Position Bias with xformers

[CFOP-xyn]

❓ Questions and Help

Can I call this: xformers.ops.memory_efficient_attention(q, k, v, attn_bias=rp_bias, scale=self.scale, op=None) where the rp_bais is a Relative Position Bias

############## the original code:

qkv = self.qkv(x).reshape(B, N, 3, self.num_heads, C // self.num_heads).permute(2, 0, 3, 1, 4) q, k, v = qkv[0], qkv[1], qkv[2] # B, num_heads, num_patches, dims_head

attn = (q @ k.transpose(-2, -1)) * self.scale # B, num_heads, num_patches(q), num_patches(k) attn += rp_bias # B, num_heads, num_patches(q), num_patches(k) attn = attn.softmax(dim=-1) # B, num_heads, num_patches(q), num_patches(k) attn = self.attn_drop(attn)

x = (attn @ v).transpose(1, 2).reshape(B, N, C) # B, num_patches(q), C(v)