diffusers-rs icon indicating copy to clipboard operation
diffusers-rs copied to clipboard

Implement Attention-Based Building Blocks

Open sanowl opened this issue 1 year ago • 0 comments

This pull request implements attention-based building blocks for neural networks using the tch-rs library. The implemented components include:

GeGlu: Gated Linear Unit activation function.
FeedForward: A feed-forward layer with GeGlu activation.
CrossAttention: Cross-attention layer for query-key-value attention.
BasicTransformerBlock: A basic Transformer block composed of cross-attention and feed-forward layers.
SpatialTransformer: A spatial transformer model (also known as Transformer2DModel) that applies a series of BasicTransformerBlock layers.
AttentionBlock: An attention block that performs self-attention on the input tensor

sanowl avatar Apr 04 '24 07:04 sanowl