NeMo icon indicating copy to clipboard operation
NeMo copied to clipboard

Context Parallel SFT Support for dataset in THD format

Open tomlifu opened this issue 1 year ago • 0 comments

What does this PR do ?

This PR adds CP support for THD format and is compatible with cu_seqlen_padded in the latest CUDNN fused attention.

PR Type:

  • [x] New Feature
  • [ ] Bugfix
  • [ ] Documentation

If you haven't finished some of the above items you can still open "Draft" PR.

tomlifu avatar Sep 30 '24 23:09 tomlifu