mamba icon indicating copy to clipboard operation
mamba copied to clipboard

Published 20 hours ago • verified publisher icon state-spaces

Does mamba support data packing?

Open ScottHoang opened this issue 7 months ago • 0 comments

To accelerate training, it is common practice to pack many text sequences into the same sentence. Self-attention avoids sample contamination with causal mask/sequence IDs. I do not see a similar mechanism built for mamba. Does mamba support data packing?

ScottHoang avatar Jul 18 '24 17:07 ScottHoang