nntrainer
nntrainer copied to clipboard
[activation] Overwrite derivative in softmax
Commit 1: [mol attention] adjust tensor lifespan to save memory
- Modify tensor lifespan of fc_out to FORWARD_FUNC_LIFESPAN
- remove unused enum updated_state
- Change param enum name from AttentionParams to MoLAttentionParams
Signed-off-by: hyeonseok lee <[email protected]>
Commit 2: [activation] revise softmaxPrime to overwrite incoming derivative
- Revise softmaxPrime to overwrite incoming derivative by using additional temp memory
- Remove attention_score tensor in attention/multi_head_attention layer
- Reuse temporary tensor for softmax in mol attention layer
close #1986
Signed-off-by: hyeonseok lee <[email protected]>
:memo: TAOS-CI Version: 1.5.20200925. Thank you for submitting PR #1999. Please a submit 1commit/1PR (one commit per one PR) policy to get comments quickly from reviewers. Your PR must pass all verificiation processes of cibot before starting a review process from reviewers. If you are new member to join this project, please read manuals in documentation folder and wiki page. In order to monitor a progress status of your PR in more detail, visit http://ci.nnstreamer.ai/.