Megatron-DeepSpeed icon indicating copy to clipboard operation
Megatron-DeepSpeed copied to clipboard

Published 20 hours ago • verified publisher icon bigscience-workshop

Add FlashAttention

Open NouamaneTazi opened this issue 1 year ago • 3 comments

This PR aims to add an option to use FlashAttention. Inspired by https://github.com/NVIDIA/Megatron-LM/pull/267

cc @thomasw21

NouamaneTazi avatar Dec 12 '22 19:12 NouamaneTazi