TransformerEngine icon indicating copy to clipboard operation
TransformerEngine copied to clipboard

[WIP] [PyTorch] Proof-of-concept for using operation-based API in modules

Open timmoon10 opened this issue 1 year ago • 1 comments

Description

This is an experiment in replacing the internal implementation in te.Linear and te.LayerNormLinear with a te.Sequential.

It depends on several other PRs:

  • https://github.com/NVIDIA/TransformerEngine/pull/1033
  • https://github.com/NVIDIA/TransformerEngine/pull/1063
  • https://github.com/NVIDIA/TransformerEngine/pull/1117
  • https://github.com/NVIDIA/TransformerEngine/pull/1127

Type of change

  • [ ] Documentation change (change only to the documentation, either a fix or a new content)
  • [ ] Bug fix (non-breaking change which fixes an issue)
  • [ ] New feature (non-breaking change which adds functionality)
  • [ ] Breaking change (fix or feature that would cause existing functionality to not work as expected)
  • [ ] Infra/Build change
  • [x] Code refractor

Changes

  • Replace implementation of te.Linear with te.Sequential
  • Replace implementation of te.LayerNormLinear with te.Sequential`

Checklist:

  • [x] I have read and followed the contributing guidelines
  • [ ] The functionality is complete
  • [ ] I have commented my code, particularly in hard-to-understand areas
  • [ ] I have made corresponding changes to the documentation
  • [ ] My changes generate no new warnings
  • [ ] I have added tests that prove my fix is effective or that my feature works
  • [ ] New and existing unit tests pass locally with my changes

timmoon10 avatar Sep 10 '24 00:09 timmoon10

/te-ci pytorch

timmoon10 avatar Sep 10 '24 20:09 timmoon10