nntrainer icon indicating copy to clipboard operation
nntrainer copied to clipboard

[activation] Overwrite derivative in softmax

Open lhs8928 opened this issue 1 year ago • 1 comments

Commit 1: [mol attention] adjust tensor lifespan to save memory

 - Modify tensor lifespan of fc_out to FORWARD_FUNC_LIFESPAN
 - remove unused enum updated_state
 - Change param enum name from AttentionParams to MoLAttentionParams

Signed-off-by: hyeonseok lee <[email protected]>

Commit 2: [activation] revise softmaxPrime to overwrite incoming derivative

 - Revise softmaxPrime to overwrite incoming derivative by using additional temp memory
 - Remove attention_score tensor in attention/multi_head_attention layer
 - Reuse temporary tensor for softmax in mol attention layer

close #1986

Signed-off-by: hyeonseok lee <[email protected]>

lhs8928 avatar Sep 08 '22 04:09 lhs8928

:memo: TAOS-CI Version: 1.5.20200925. Thank you for submitting PR #1999. Please a submit 1commit/1PR (one commit per one PR) policy to get comments quickly from reviewers. Your PR must pass all verificiation processes of cibot before starting a review process from reviewers. If you are new member to join this project, please read manuals in documentation folder and wiki page. In order to monitor a progress status of your PR in more detail, visit http://ci.nnstreamer.ai/.

taos-ci avatar Sep 08 '22 04:09 taos-ci