matrix-appservice-discord
matrix-appservice-discord copied to clipboard
fix: precaching of all transactions causes errors
This PR changes the storage implementation to only load users and transaction ids when they are needed. Recent queries for txIds and users are cached. This fix also reduces memory consumption.
Fixes #765
Kinda sad to see that this is still marked as draft and has no responses so far...
Tested it, works fine
Nov-16 21:37:58.622 [DiscordStore] verbose: User registration status added to registeredUsersCache.
userId: '@_discord_000000000000000000:matrix.redacted.invalid' ; registered: true
[...]
Nov-16 21:38:27.140 [DiscordStore] verbose: User registration status retrieved from registeredUsersCache.
userId: '@_discord_000000000000000000:matrix.redacted.invalid' ; registered: true
Nov-16 21:38:27.640 [DiscordStore] verbose: User registration status retrieved from registeredUsersCache.
userId: '@_discord_000000000000000000:matrix.redacted.invalid' ; registered: true
Nov-16 21:38:27.698 [DiscordBot] verbose: i:redacted c:redacted g:redacted
Nov-16 21:38:27.699 [UserSync] verbose: State update requested for 000000000000000000
Nov-16 21:38:27.701 [DiscordStore] verbose: User registration status retrieved from registeredUsersCache.
userId: '@_discord_000000000000000000:matrix.redacted.invalid' ; registered: true
Nov-16 21:38:27.706 [DiscordStore] verbose: User registration status retrieved from registeredUsersCache.
userId: '@_discord_000000000000000000:matrix.redacted.invalid' ; registered: true
Nov-16 21:38:28.140 [DiscordStore] verbose: User registration status retrieved from registeredUsersCache.
userId: '@_discord_000000000000000000:matrix.redacted.invalid' ; registered: true
Nov-16 21:38:28.640 [DiscordStore] verbose: User registration status retrieved from registeredUsersCache.
userId: '@_discord_000000000000000000:matrix.redacted.invalid' ; registered: true
Nov-16 21:38:28.764 [DiscordStore] verbose: Txn completion status added to txnsCompletionStatusCache.
txnId: '51' ; completed: false
Nov-16 21:38:28.765 [bot-sdkAppservice] info: [ 'Processing transaction 51' ]
Nov-16 21:38:28.766 [bot-sdkAppservice] info: [ 'Processing event of type m.room.message' ]
Nov-16 21:38:28.768 [DiscordStore] verbose: Txn completion status added to txnsCompletionStatusCache.
txnId: '51' ; completed: true
Lets see what the CI says, otherwise should be ready to review
Added log.silly for cache pruning, works too
Nov-16 22:07:10.713 [Timedcache] silly: removed entry with key='54' from cache 'txnsCompletionStatusCache'
Nov-16 22:07:10.713 [Timedcache] silly: removed entry with key='55' from cache 'txnsCompletionStatusCache'
Nov-16 22:07:10.713 [Timedcache] silly: removed entry with key='56' from cache 'txnsCompletionStatusCache'
Nov-16 22:07:10.713 [Timedcache] silly: removed entry with key='57' from cache 'txnsCompletionStatusCache'
Nov-16 22:07:10.713 [Timedcache] silly: removed entry with key='58' from cache 'txnsCompletionStatusCache'
Nov-16 22:07:10.713 [Timedcache] silly: removed entry with key='59' from cache 'txnsCompletionStatusCache'
Nov-16 22:07:10.713 [Timedcache] silly: removed entry with key='60' from cache 'txnsCompletionStatusCache'
Nov-16 22:07:10.713 [Timedcache] silly: removed entry with key='61' from cache 'txnsCompletionStatusCache'
Nov-16 22:07:10.713 [Timedcache] silly: removed entry with key='62' from cache 'txnsCompletionStatusCache'
Nov-16 22:07:10.713 [Timedcache] silly: removed entry with key='63' from cache 'txnsCompletionStatusCache'
Nov-16 22:07:10.713 [Timedcache] silly: removed entry with key='64' from cache 'txnsCompletionStatusCache'
Nov-16 22:07:10.713 [Timedcache] silly: removed entry with key='65' from cache 'txnsCompletionStatusCache'
Nov-16 22:07:10.714 [Timedcache] silly: removed entry with key='66' from cache 'txnsCompletionStatusCache'
Nov-16 22:07:10.714 [Timedcache] silly: removed entry with key='67' from cache 'txnsCompletionStatusCache'
Nov-16 22:07:10.714 [Timedcache] silly: removed entry with key='68' from cache 'txnsCompletionStatusCache'
Nov-16 22:07:10.714 [Timedcache] silly: removed entry with key='69' from cache 'txnsCompletionStatusCache'
Nov-16 22:07:10.714 [Timedcache] silly: removed entry with key='70' from cache 'txnsCompletionStatusCache'
Nov-16 22:07:10.714 [Timedcache] silly: removed entry with key='71' from cache 'txnsCompletionStatusCache'
Nov-16 22:07:10.714 [Timedcache] silly: removed entry with key='72' from cache 'txnsCompletionStatusCache'
Nov-16 22:07:10.714 [Timedcache] silly: removed entry with key='73' from cache 'txnsCompletionStatusCache'
Nov-16 22:07:10.714 [Timedcache] silly: removed entry with key='74' from cache 'txnsCompletionStatusCache'
Nov-16 22:07:10.714 [Timedcache] silly: removed entry with key='75' from cache 'txnsCompletionStatusCache'
Nov-16 22:07:10.714 [Timedcache] silly: removed entry with key='@_discord_bot:matrix.redacted.invalid' from cache 'registeredUsersCache'