nntrainer icon indicating copy to clipboard operation
nntrainer copied to clipboard

Support transformer

Open lhs8928 opened this issue 1 year ago • 1 comments

  • [ ] Implement multi head attention
    • [x] implement forwarding - #1961
    • [x] implement backwarding - #1963
    • [ ] implement attention mask
      • [x] support float type mask - #1963
      • [ ] support bool type mask
    • [x] verify with unittest
  • [x] Implement layer normalization
    • [x] integrate core algorithm with bn layer - #1993
    • [x] implement forwarding - #1993
    • [x] verify with unittest - #1993
  • [ ] Make transformer as an application
    • [x] implement positional encoding layer - #1989
    • [x] verify encoder/decoder layer with unittest - #1996
    • [ ] train with dataset

lhs8928 avatar Aug 03 '22 07:08 lhs8928

:octocat: cibot: Thank you for posting issue #1972. The person in charge will reply soon.

taos-ci avatar Aug 03 '22 07:08 taos-ci