marian-dev icon indicating copy to clipboard operation
marian-dev copied to clipboard

Published 20 hours ago • verified publisher icon marian-nmt

Check size on transformer cache

Open graemenail opened this issue 2 years ago • 0 comments

Description

This PR reinstates a shape check on the entries of transformer cache. This PR is the minimal change required to address the issue highlighted in #881.

#881 was reported to have increased memory usage in normal operation. In that PR, the tensor shape was added to the cache key for better retrieval. However, this change led to many more cache entries and because this cache is a member of the Transformer, the lifetime of cache entries were tied to that of the Transformer and ultimately persisted too long.

Related: #881

List of changes:

  • Check shape instead of elements in transformer cache retrieval

Added dependencies: none

How to test

  • Passes CI tests
  • @emjotde has a benchmark that should be checked

Checklist

  • [ ] I have tested the code manually
  • [ ] I have run regression tests
  • [ ] I have read and followed CONTRIBUTING.md
  • [ ] I have updated CHANGELOG.md

graemenail avatar Jun 08 '22 18:06 graemenail