Driss Guessous

Nested tensor empty like

1

## Summary To better enable users to write triton kernels on nested nested tensors having empty_like support is helpful. This will be used to create a nested_tensor output buffer to...

Update shallow_copy_and_detach for nested tensor impls

1

# Summary This change fixes a bug that was encountered when trying to add more backward formulas for nested tensor ops. If a derivative is defined that stores the "result"...

Exposing native _scaled_dot_product_attention to torch.nn

15

# Summary This exposes the _scaled_dot_product_attention function to python in the nn namespace. It is still underscored because the api for args, and kwargs is still in flux for the...

Custom SDP implementation runtime dispatch

2

# Summary This shows the run mechanism for dispatching to a custom backend kernel or defaulting to the composite version for SDP. Currently we short circuit for composite but in...

Remove windows check for cmake to build Fused kernels

6

# Summary At the start of integration of the fused code, Windows was not supported. This PR is to remove that limitation.

[SDPA] Update SDPA API and make function Public

1

# Summary In preparation for pt2.0 launch this PR adjusts the private functions api and makes it public.. TODO fill out the rest of the description

[SDPA] Wire up FlashAttention's backward

3

# Summary This PR creates _flash_attention_backward and _scaled_dot_product_flash_attention_backward native functions and registers them to the respective derivatives.yaml. The goal is to replicate the torch.autograd.Function defined in the FlashAttention repo [here](https://github.com/HazyResearch/flash-attention/blob/33e0860c9c5667fded5af674882e731909096a7f/flash_attn/flash_attn_interface.py#L126)...

Add sliding window attention bias

1

# Summary WIP still

Update cutlass from 3.3.0 to 3.4.1

2

# Summary ### Current Status - [PR](https://github.com/pytorch/pytorch/pull/118935) aims to update the flash attention kernels to a more recent version. - The update is **partially blocked** due to a dependency on...

Update flash_attention kernel from 2.3.6 to 2.5.5

6

# Summary Updates FlashAttention kernel code from tag [2.3.6](https://github.com/Dao-AILab/flash-attention/releases/tag/v2.3.6) to [2.5.3](https://github.com/Dao-AILab/flash-attention/releases/tag/v2.5.5). The usual changes were then re-rellod on top of the modified kernel, changing how dropout saved for backward, removing...