mamba
mamba copied to clipboard
Using input if size torch.Size([200, 100])
My input is a batch of 200 and values of 100. (torch.Size([200, 100])) So my feature is just a 1-D vector.
In the example code, it says you set batch, length, and dim. But I have no length. How do I set Mamba in this case?
batch, length, dim = 2, 64, 16
x = torch.randn(batch, length, dim).to("cuda")
model = Mamba(
# This module uses roughly 3 * expand * d_model^2 parameters
d_model=dim, # Model dimension d_model
d_state=16, # SSM state expansion factor
d_conv=4, # Local convolution width
expand=2, # Block expansion factor
).to("cuda")
Mamba is a sequence-to-sequence layer, just like attention. If you don't have the length dimension (i.e. it's not a sequence) then Mamba is likely not a good choice.