matrix-appservice-discord icon indicating copy to clipboard operation
matrix-appservice-discord copied to clipboard

fix: precaching of all transactions causes errors

Open NickUfer opened this issue 3 years ago • 3 comments

This PR changes the storage implementation to only load users and transaction ids when they are needed. Recent queries for txIds and users are cached. This fix also reduces memory consumption.

Fixes #765

NickUfer avatar Jan 03 '22 18:01 NickUfer

Kinda sad to see that this is still marked as draft and has no responses so far...

verymilan avatar Feb 17 '22 09:02 verymilan

Tested it, works fine

Nov-16 21:37:58.622 [DiscordStore] verbose: User registration status added to registeredUsersCache. 
                    userId: '@_discord_000000000000000000:matrix.redacted.invalid' ; registered: true
                    
[...]

Nov-16 21:38:27.140 [DiscordStore] verbose: User registration status retrieved from registeredUsersCache. 
                        userId: '@_discord_000000000000000000:matrix.redacted.invalid' ; registered: true
Nov-16 21:38:27.640 [DiscordStore] verbose: User registration status retrieved from registeredUsersCache. 
                        userId: '@_discord_000000000000000000:matrix.redacted.invalid' ; registered: true
Nov-16 21:38:27.698 [DiscordBot] verbose: i:redacted c:redacted g:redacted
Nov-16 21:38:27.699 [UserSync] verbose: State update requested for 000000000000000000
Nov-16 21:38:27.701 [DiscordStore] verbose: User registration status retrieved from registeredUsersCache. 
                        userId: '@_discord_000000000000000000:matrix.redacted.invalid' ; registered: true
Nov-16 21:38:27.706 [DiscordStore] verbose: User registration status retrieved from registeredUsersCache. 
                        userId: '@_discord_000000000000000000:matrix.redacted.invalid' ; registered: true
Nov-16 21:38:28.140 [DiscordStore] verbose: User registration status retrieved from registeredUsersCache. 
                        userId: '@_discord_000000000000000000:matrix.redacted.invalid' ; registered: true
Nov-16 21:38:28.640 [DiscordStore] verbose: User registration status retrieved from registeredUsersCache. 
                        userId: '@_discord_000000000000000000:matrix.redacted.invalid' ; registered: true
Nov-16 21:38:28.764 [DiscordStore] verbose: Txn completion status added to txnsCompletionStatusCache. 
                    txnId: '51' ; completed: false
Nov-16 21:38:28.765 [bot-sdkAppservice] info: [ 'Processing transaction 51' ]
Nov-16 21:38:28.766 [bot-sdkAppservice] info: [ 'Processing event of type m.room.message' ]
Nov-16 21:38:28.768 [DiscordStore] verbose: Txn completion status added to txnsCompletionStatusCache. 
                            txnId: '51' ; completed: true

Lets see what the CI says, otherwise should be ready to review

NickUfer avatar Nov 16 '22 21:11 NickUfer

Added log.silly for cache pruning, works too

Nov-16 22:07:10.713 [Timedcache] silly: removed entry with key='54' from cache 'txnsCompletionStatusCache'
Nov-16 22:07:10.713 [Timedcache] silly: removed entry with key='55' from cache 'txnsCompletionStatusCache'
Nov-16 22:07:10.713 [Timedcache] silly: removed entry with key='56' from cache 'txnsCompletionStatusCache'
Nov-16 22:07:10.713 [Timedcache] silly: removed entry with key='57' from cache 'txnsCompletionStatusCache'
Nov-16 22:07:10.713 [Timedcache] silly: removed entry with key='58' from cache 'txnsCompletionStatusCache'
Nov-16 22:07:10.713 [Timedcache] silly: removed entry with key='59' from cache 'txnsCompletionStatusCache'
Nov-16 22:07:10.713 [Timedcache] silly: removed entry with key='60' from cache 'txnsCompletionStatusCache'
Nov-16 22:07:10.713 [Timedcache] silly: removed entry with key='61' from cache 'txnsCompletionStatusCache'
Nov-16 22:07:10.713 [Timedcache] silly: removed entry with key='62' from cache 'txnsCompletionStatusCache'
Nov-16 22:07:10.713 [Timedcache] silly: removed entry with key='63' from cache 'txnsCompletionStatusCache'
Nov-16 22:07:10.713 [Timedcache] silly: removed entry with key='64' from cache 'txnsCompletionStatusCache'
Nov-16 22:07:10.713 [Timedcache] silly: removed entry with key='65' from cache 'txnsCompletionStatusCache'
Nov-16 22:07:10.714 [Timedcache] silly: removed entry with key='66' from cache 'txnsCompletionStatusCache'
Nov-16 22:07:10.714 [Timedcache] silly: removed entry with key='67' from cache 'txnsCompletionStatusCache'
Nov-16 22:07:10.714 [Timedcache] silly: removed entry with key='68' from cache 'txnsCompletionStatusCache'
Nov-16 22:07:10.714 [Timedcache] silly: removed entry with key='69' from cache 'txnsCompletionStatusCache'
Nov-16 22:07:10.714 [Timedcache] silly: removed entry with key='70' from cache 'txnsCompletionStatusCache'
Nov-16 22:07:10.714 [Timedcache] silly: removed entry with key='71' from cache 'txnsCompletionStatusCache'
Nov-16 22:07:10.714 [Timedcache] silly: removed entry with key='72' from cache 'txnsCompletionStatusCache'
Nov-16 22:07:10.714 [Timedcache] silly: removed entry with key='73' from cache 'txnsCompletionStatusCache'
Nov-16 22:07:10.714 [Timedcache] silly: removed entry with key='74' from cache 'txnsCompletionStatusCache'
Nov-16 22:07:10.714 [Timedcache] silly: removed entry with key='75' from cache 'txnsCompletionStatusCache'
Nov-16 22:07:10.714 [Timedcache] silly: removed entry with key='@_discord_bot:matrix.redacted.invalid' from cache 'registeredUsersCache'

NickUfer avatar Nov 16 '22 22:11 NickUfer