matrix-org
Fixing Presence Conflicts
This is meant to cache client presence for a moment so that it doesn't oscillate.
Currently Dendrite just federates out whatever presence it gets from the sync loop, which means if theres any clients attempting to sync without setting the user online, and there is an online client, it will just flip back and forth each time one of the clients polls /sync.
This pull request essentially stores in a map when the client last set online ideally to allow the online client to sync again and set an online presence before setting idle or offline.
I am not great at programming nor am I familiar with this codebase so if this pr is just shitwater feel free to discard, just trying to fix an issue that severely bothers me. If it is easier you can also steal the code and write it in yourself. I ran the linter, not sure that it did anything, the vscode go extension seems to format and lint anyways.
I tried to run unit tests but I have no idea any of this thing. it errors on TestRequestPool_updatePresence/same_presence_is_not_published_dummy2 (10m0s) which I think making this change broke. I am unsure how to comply, if y'all point me in the right direction ill try to fix it. I have tested it with all the situations I can think of on my personal instance pain.agency, and this seems to stand up under all the previously bugged situations.
~~My go also decided to update a bunch of the dependencies, I hate git and github and have no idea how to fix that, it was not intentional.~~ i just overwrote them with the ones from the main repo and committed it, seems to have done what was needed.
Pull Request Checklist
- [x] I have added Go unit tests or Complement integration tests for this PR or I have justified why this PR doesn't need tests
- [x] Pull request includes a sign off below using a legally identifiable name or I have already signed off privately
Signed-off-by: Joseph Winkie <[email protected]>
heres the unit test full output:
joseph@f ~/jendrite (presence-optimization) [1]> DENDRITE_TEST_SKIP_NODB=1 go test --race ./...
# github.com/matrix-org/dendrite/internal/sqlutil
internal/sqlutil/sqlutil_test.go:10:2: missing go.sum entry for module providing package github.com/DATA-DOG/go-sqlmock (imported by github.com/matrix-org/dendrite/internal/sqlutil); to add:
go get -t github.com/matrix-org/dendrite/internal/sqlutil
FAIL github.com/matrix-org/dendrite/internal/sqlutil [setup failed]
# maunium.net/go/mautrix/crypto/olm
../go/pkg/mod/maunium.net/go/[email protected]/crypto/olm/account.go:6:11: fatal error: olm/olm.h: No such file or directory
6 | // #include <olm/olm.h>
| ^~~~~~~~~~~
compilation terminated.
? github.com/matrix-org/dendrite/appservice/api [no test files]
? github.com/matrix-org/dendrite/appservice/consumers [no test files]
? github.com/matrix-org/dendrite/appservice/query [no test files]
? github.com/matrix-org/dendrite/build/dendritejs-pinecone [no test files]
? github.com/matrix-org/dendrite/build/gobind-yggdrasil [no test files]
? github.com/matrix-org/dendrite/clientapi/api [no test files]
? github.com/matrix-org/dendrite/clientapi/auth/authtypes [no test files]
? github.com/matrix-org/dendrite/clientapi/httputil [no test files]
? github.com/matrix-org/dendrite/clientapi/producers [no test files]
? github.com/matrix-org/dendrite/clientapi/threepid [no test files]
? github.com/matrix-org/dendrite/cmd/dendrite-demo-pinecone [no test files]
? github.com/matrix-org/dendrite/cmd/dendrite-demo-pinecone/conn [no test files]
? github.com/matrix-org/dendrite/cmd/dendrite-demo-pinecone/defaults [no test files]
? github.com/matrix-org/dendrite/cmd/dendrite-demo-pinecone/embed [no test files]
? github.com/matrix-org/dendrite/cmd/dendrite-demo-pinecone/monolith [no test files]
? github.com/matrix-org/dendrite/cmd/dendrite-demo-pinecone/rooms [no test files]
? github.com/matrix-org/dendrite/cmd/dendrite-demo-pinecone/users [no test files]
? github.com/matrix-org/dendrite/cmd/dendrite-demo-yggdrasil [no test files]
? github.com/matrix-org/dendrite/cmd/dendrite-demo-yggdrasil/embed [no test files]
? github.com/matrix-org/dendrite/cmd/dendrite-demo-yggdrasil/signing [no test files]
? github.com/matrix-org/dendrite/cmd/dendrite-demo-yggdrasil/yggconn [no test files]
? github.com/matrix-org/dendrite/cmd/dendrite-demo-yggdrasil/yggrooms [no test files]
? github.com/matrix-org/dendrite/cmd/dendrite-upgrade-tests [no test files]
? github.com/matrix-org/dendrite/cmd/furl [no test files]
? github.com/matrix-org/dendrite/cmd/generate-config [no test files]
? github.com/matrix-org/dendrite/cmd/generate-keys [no test files]
? github.com/matrix-org/dendrite/cmd/resolve-state [no test files]
? github.com/matrix-org/dendrite/federationapi/api [no test files]
? github.com/matrix-org/dendrite/federationapi/producers [no test files]
? github.com/matrix-org/dendrite/federationapi/storage/cache [no test files]
? github.com/matrix-org/dendrite/federationapi/storage/postgres [no test files]
? github.com/matrix-org/dendrite/federationapi/storage/postgres/deltas [no test files]
? github.com/matrix-org/dendrite/federationapi/storage/shared [no test files]
? github.com/matrix-org/dendrite/federationapi/storage/shared/receipt [no test files]
? github.com/matrix-org/dendrite/federationapi/storage/sqlite3 [no test files]
? github.com/matrix-org/dendrite/federationapi/storage/sqlite3/deltas [no test files]
? github.com/matrix-org/dendrite/federationapi/types [no test files]
? github.com/matrix-org/dendrite/internal/eventutil [no test files]
? github.com/matrix-org/dendrite/internal/hooks [no test files]
? github.com/matrix-org/dendrite/mediaapi [no test files]
? github.com/matrix-org/dendrite/mediaapi/fileutils [no test files]
? github.com/matrix-org/dendrite/mediaapi/storage/postgres [no test files]
? github.com/matrix-org/dendrite/mediaapi/storage/shared [no test files]
? github.com/matrix-org/dendrite/mediaapi/storage/sqlite3 [no test files]
? github.com/matrix-org/dendrite/mediaapi/storage/tables [no test files]
? github.com/matrix-org/dendrite/mediaapi/thumbnailer [no test files]
? github.com/matrix-org/dendrite/mediaapi/types [no test files]
? github.com/matrix-org/dendrite/relayapi/api [no test files]
? github.com/matrix-org/dendrite/relayapi/storage [no test files]
? github.com/matrix-org/dendrite/relayapi/storage/postgres [no test files]
? github.com/matrix-org/dendrite/relayapi/storage/shared [no test files]
? github.com/matrix-org/dendrite/relayapi/storage/sqlite3 [no test files]
ok github.com/matrix-org/dendrite/appservice 3.425s
ok github.com/matrix-org/dendrite/build/gobind-pinecone 2.418s
FAIL github.com/matrix-org/dendrite/clientapi [build failed]
ok github.com/matrix-org/dendrite/clientapi/auth (cached)
ok github.com/matrix-org/dendrite/clientapi/routing (cached)
ok github.com/matrix-org/dendrite/clientapi/userutil (cached)
ok github.com/matrix-org/dendrite/cmd/create-account 1.079s
ok github.com/matrix-org/dendrite/cmd/dendrite 1.123s
ok github.com/matrix-org/dendrite/cmd/dendrite-demo-pinecone/conduit (cached)
ok github.com/matrix-org/dendrite/cmd/dendrite-demo-pinecone/relay (cached)
ok github.com/matrix-org/dendrite/federationapi (cached)
ok github.com/matrix-org/dendrite/federationapi/consumers (cached)
ok github.com/matrix-org/dendrite/federationapi/internal (cached)
ok github.com/matrix-org/dendrite/federationapi/queue (cached)
ok github.com/matrix-org/dendrite/federationapi/routing (cached)
ok github.com/matrix-org/dendrite/federationapi/statistics (cached)
ok github.com/matrix-org/dendrite/federationapi/storage (cached)
ok github.com/matrix-org/dendrite/federationapi/storage/tables (cached)
ok github.com/matrix-org/dendrite/internal (cached)
ok github.com/matrix-org/dendrite/internal/caching (cached)
ok github.com/matrix-org/dendrite/internal/fulltext (cached)
ok github.com/matrix-org/dendrite/internal/httputil (cached)
ok github.com/matrix-org/dendrite/internal/pushgateway (cached)
ok github.com/matrix-org/dendrite/internal/pushrules (cached)
ok github.com/matrix-org/dendrite/internal/transactions (cached)
ok github.com/matrix-org/dendrite/mediaapi/routing (cached)
ok github.com/matrix-org/dendrite/mediaapi/storage (cached)
ok github.com/matrix-org/dendrite/relayapi (cached)
ok github.com/matrix-org/dendrite/relayapi/internal (cached)
ok github.com/matrix-org/dendrite/relayapi/routing (cached)
ok github.com/matrix-org/dendrite/relayapi/storage/tables (cached)
? github.com/matrix-org/dendrite/roomserver/internal [no test files]
? github.com/matrix-org/dendrite/roomserver/internal/perform [no test files]
? github.com/matrix-org/dendrite/roomserver/producers [no test files]
? github.com/matrix-org/dendrite/roomserver/storage [no test files]
? github.com/matrix-org/dendrite/roomserver/storage/postgres [no test files]
? github.com/matrix-org/dendrite/roomserver/storage/sqlite3 [no test files]
? github.com/matrix-org/dendrite/roomserver/version [no test files]
? github.com/matrix-org/dendrite/setup [no test files]
? github.com/matrix-org/dendrite/setup/jetstream [no test files]
? github.com/matrix-org/dendrite/setup/mscs [no test files]
? github.com/matrix-org/dendrite/setup/process [no test files]
? github.com/matrix-org/dendrite/syncapi/consumers [no test files]
? github.com/matrix-org/dendrite/syncapi/producers [no test files]
? github.com/matrix-org/dendrite/syncapi/storage/postgres [no test files]
? github.com/matrix-org/dendrite/syncapi/storage/postgres/deltas [no test files]
? github.com/matrix-org/dendrite/syncapi/storage/sqlite3 [no test files]
? github.com/matrix-org/dendrite/syncapi/storage/sqlite3/deltas [no test files]
? github.com/matrix-org/dendrite/syncapi/streams [no test files]
? github.com/matrix-org/dendrite/test [no test files]
? github.com/matrix-org/dendrite/test/testrig [no test files]
? github.com/matrix-org/dendrite/userapi/api [no test files]
? github.com/matrix-org/dendrite/userapi/producers [no test files]
? github.com/matrix-org/dendrite/userapi/storage/postgres [no test files]
? github.com/matrix-org/dendrite/userapi/storage/postgres/deltas [no test files]
? github.com/matrix-org/dendrite/userapi/storage/shared [no test files]
? github.com/matrix-org/dendrite/userapi/storage/sqlite3 [no test files]
? github.com/matrix-org/dendrite/userapi/storage/sqlite3/deltas [no test files]
? github.com/matrix-org/dendrite/userapi/types [no test files]
ok github.com/matrix-org/dendrite/roomserver 6.391s
ok github.com/matrix-org/dendrite/roomserver/acls (cached)
ok github.com/matrix-org/dendrite/roomserver/api (cached)
ok github.com/matrix-org/dendrite/roomserver/auth (cached)
ok github.com/matrix-org/dendrite/roomserver/internal/helpers (cached)
ok github.com/matrix-org/dendrite/roomserver/internal/input (cached)
ok github.com/matrix-org/dendrite/roomserver/internal/query (cached)
ok github.com/matrix-org/dendrite/roomserver/state (cached)
ok github.com/matrix-org/dendrite/roomserver/storage/postgres/deltas (cached)
ok github.com/matrix-org/dendrite/roomserver/storage/shared (cached)
ok github.com/matrix-org/dendrite/roomserver/storage/sqlite3/deltas (cached)
ok github.com/matrix-org/dendrite/roomserver/storage/tables (cached)
ok github.com/matrix-org/dendrite/roomserver/types (cached)
ok github.com/matrix-org/dendrite/setup/base (cached)
ok github.com/matrix-org/dendrite/setup/config (cached)
ok github.com/matrix-org/dendrite/setup/mscs/msc2836 (cached)
ok github.com/matrix-org/dendrite/syncapi 4.661s
ok github.com/matrix-org/dendrite/syncapi/internal (cached)
ok github.com/matrix-org/dendrite/syncapi/notifier (cached)
ok github.com/matrix-org/dendrite/syncapi/routing 1.168s
ok github.com/matrix-org/dendrite/syncapi/storage (cached)
ok github.com/matrix-org/dendrite/syncapi/storage/shared (cached)
ok github.com/matrix-org/dendrite/syncapi/storage/tables (cached)
panic: test timed out after 10m0s
running tests:
TestRequestPool_updatePresence (10m0s)
TestRequestPool_updatePresence/same_presence_is_not_published_dummy2 (10m0s)
goroutine 620 [running]:
testing.(*M).startAlarm.func1()
/usr/lib/golang/src/testing/testing.go:2259 +0x259
created by time.goFunc
/usr/lib/golang/src/time/sleep.go:176 +0x45
goroutine 1 [chan receive]:
testing.(*T).Run(0xc000007a00, {0xef5917, 0x1e}, 0xf2ddb0)
/usr/lib/golang/src/testing/testing.go:1649 +0x871
testing.runTests.func1(0x0?)
/usr/lib/golang/src/testing/testing.go:2054 +0x85
testing.tRunner(0xc000007a00, 0xc0002b1af8)
/usr/lib/golang/src/testing/testing.go:1595 +0x262
testing.runTests(0xc0000b7720?, {0x13ee6e0, 0x1, 0x1}, {0x1c?, 0x1e?, 0x1459fe0?})
/usr/lib/golang/src/testing/testing.go:2052 +0x8ae
testing.(*M).Run(0xc0000b7720)
/usr/lib/golang/src/testing/testing.go:1925 +0xcd8
main.main()
_testmain.go:47 +0x2be
goroutine 19 [chan receive]:
github.com/golang/glog.(*loggingT).flushDaemon(0x0?)
/home/joseph/go/pkg/mod/github.com/golang/[email protected]/glog.go:882 +0x8b
created by github.com/golang/glog.init.0 in goroutine 1
/home/joseph/go/pkg/mod/github.com/golang/[email protected]/glog.go:410 +0x258
goroutine 35 [sleep]:
time.Sleep(0x2faf080)
/usr/lib/golang/src/runtime/time.go:195 +0x125
github.com/matrix-org/dendrite/syncapi/sync.(*RequestPool).cleanPresence(0xc000230000, {0xffd378?, 0x23c0620}, 0x2faf080)
/home/joseph/jendrite/syncapi/sync/requestpool.go:119 +0x152
created by github.com/matrix-org/dendrite/syncapi/sync.TestRequestPool_updatePresence in goroutine 6
/home/joseph/jendrite/syncapi/sync/requestpool_test.go:129 +0x45a
goroutine 6 [chan receive]:
testing.(*T).Run(0xc000007ba0, {0xefb650, 0x25}, 0xc0002281a0)
/usr/lib/golang/src/testing/testing.go:1649 +0x871
github.com/matrix-org/dendrite/syncapi/sync.TestRequestPool_updatePresence(0x0?)
/home/joseph/jendrite/syncapi/sync/requestpool_test.go:131 +0x49b
testing.tRunner(0xc000007ba0, 0xf2ddb0)
/usr/lib/golang/src/testing/testing.go:1595 +0x262
created by testing.(*T).Run in goroutine 1
/usr/lib/golang/src/testing/testing.go:1648 +0x846
goroutine 40 [sync.Mutex.Lock]:
sync.runtime_SemacquireMutex(0x12eb090?, 0x80?, 0xc00022e540?)
/usr/lib/golang/src/runtime/sema.go:77 +0x25
sync.(*Mutex).lockSlow(0xc00022c010)
/usr/lib/golang/src/sync/mutex.go:171 +0x213
sync.(*Mutex).Lock(0xc00022c010)
/usr/lib/golang/src/sync/mutex.go:90 +0x55
github.com/matrix-org/dendrite/syncapi/sync.TestRequestPool_updatePresence.func1(0xc0002269c0)
/home/joseph/jendrite/syncapi/sync/requestpool_test.go:132 +0x59
testing.tRunner(0xc0002269c0, 0xc0002281a0)
/usr/lib/golang/src/testing/testing.go:1595 +0x262
created by testing.(*T).Run in goroutine 6
/usr/lib/golang/src/testing/testing.go:1648 +0x846
FAIL github.com/matrix-org/dendrite/syncapi/sync 600.041s
ok github.com/matrix-org/dendrite/syncapi/synctypes (cached)
ok github.com/matrix-org/dendrite/syncapi/types (cached)
ok github.com/matrix-org/dendrite/userapi (cached)
ok github.com/matrix-org/dendrite/userapi/consumers (cached)
ok github.com/matrix-org/dendrite/userapi/internal (cached)
ok github.com/matrix-org/dendrite/userapi/storage (cached)
ok github.com/matrix-org/dendrite/userapi/storage/tables (cached)
ok github.com/matrix-org/dendrite/userapi/util (cached)
FAIL
⏎
joseph@f ~/jendrite (presence-optimization) [1]>
Oh I should also mention don't look too closely at the commit history, this one is a squash and merge (if it gets merged)
eh im in over my head and not in a mental state where it makes sense to work on this. bit of a waste of a day but :shrug: i got out of bed
I've hopefully fixed any potential race conditions, thanks DiamondBurned for the assistance
I could be seeing things, but this seems to have made presence with maunium bridges work correctly.
one interesting limitation ive discovered is this means presence set through sync doesnt override explicitly set presence through the api.
the only time this matters is for some reason element ios explicitly sets presence to offline when you swipe out of the app to the homescreen. im marking this down as a client bug more than anything because why the hell would it do that
I'm not a Dendrite user myself, but I'm very interested in seeing some progress on this, as my homeserver is getting presence spam from a few Dendrite instances from time to time.
I did not review your code (I don't usually write or even read go), but thank you for trying and keeping this up to date with the main branch. Have you been running this since February ? Is it going well ?
If a maintainer reads this : how can I help with making this move forward ?
PS : Disabling presence on my side does not seem to prevent receiving a deluge of presence updates.
Yes I have been running this patch since I made it, and till has recently for a couple weeks been running it too. I’m not sure what we can do to move it forward.
On Mon, Jul 15, 2024 at 10:06 PM Pierre Couy @.***> wrote:
I'm not a Dendrite user myself, but I'm very interested in seeing some progress on this, as my homeserver is getting presence spam from a few Dendrite instances from time to time.
I did not review your code (I don't usually write or even read go), but thank you for trying and keeping this up to date with the main branch. Have you been running this since February ? Is it going well ?
If a maintainer reads this : how can I help with making this move forward ?
PS : Disabling presence on my side does not seem to prevent receiving a deluge of presence updates.
— Reply to this email directly, view it on GitHub https://github.com/matrix-org/dendrite/pull/3320#issuecomment-2230278341, or unsubscribe https://github.com/notifications/unsubscribe-auth/AWNJYMG6IKFGRIINF56QLW3ZMTIAXAVCNFSM6AAAAABC3LURX2VHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZDEMZQGI3TQMZUGE . You are receiving this because you modified the open/close state.Message ID: @.***>
Codecov Report
Attention: Patch coverage is 65.38462% with 9 lines in your changes missing coverage. Please review.
Project coverage is 68.24%. Comparing base (
4d116ff) to head (ed1f7cb). Report is 1 commits behind head on main.
| Files | Patch % | Lines |
|---|---|---|
| syncapi/sync/requestpool.go | 64.00% | 8 Missing and 1 partial :warning: |
Additional details and impacted files
@@ Coverage Diff @@
## main #3320 +/- ##
=======================================
Coverage 68.24% 68.24%
=======================================
Files 513 513
Lines 46865 46888 +23
=======================================
+ Hits 31984 32000 +16
- Misses 10892 10899 +7
Partials 3989 3989
| Flag | Coverage Δ | |
|---|---|---|
| unittests | 53.41% <65.38%> (+0.03%) |
:arrow_up: |
Flags with carried forward coverage won't be shown. Click here to find out more.
:umbrella: View full report in Codecov by Sentry.
:loudspeaker: Have feedback on the report? Share it here.
![codecov[bot] avatar](https://avatars.githubusercontent.com/in/254?v=4 "Published by a pub.dev verified publisher"){kind=small}
This sounds useful for me, hopefully the maintainers can spend some of their time on reviewing this soon
I'm basically happy with it, I just want CI to be green before merging, this needs a few other PRs to be merged before.
