chatterino2
chatterino2 copied to clipboard
Chatterino
refactor: load Twitch emotes from Helix
It's probably not a fix, but for a user it's a fix.
This PR refactors how Twitch emotes are loaded - using the new Helix API.
Although the API returns decent data, it's slow. For my user (two subscriptions), I need to make 28 requests to get all emotes for one channel. One request takes about 100-250ms.
To help reduce latency, the emotes are fetched once for the user (by leaving broadcaster_id empty) and for each channel, the follow-status is checked and follower emotes are loaded. This requires (user:read:follows, https://github.com/Chatterino/website/pull/527).
Related twitchdev issues:
- https://github.com/twitchdev/issues/issues/921
- https://github.com/twitchdev/issues/issues/922
- Fixes #3583
- Fixes #4043
- Fixes #2341
With this approach we're going to need to use owner_id over emote_set_id, at least for a specific list of channel IDs, otherwise the global tab is going to be even worse than before.
image
notes:
a cavalcade of funni errors
f5-ing a single channel will label the `Twitch` section of global emotes incorrectly:
loaded naturally
f5 #1
f5 #2
emote menu can populate gif emotes twice in some channels (maybe specifically an issue with perma subs)
this does not effect the : menu
a cavalcade of funni errors
I'm hoping for Twitch to utilize the pages better. But we'll still need some rate-limiting.
f5-ing a single channel will label the
Twitchsection of global emotes incorrectly:
Fixed by grouping everything to Twitch (no owner-id).
emote menu can populate gif emotes twice in some channels (maybe specifically an issue with perma subs)
Seems to be https://github.com/twitchdev/issues/issues/922 - I'm hoping this gets fixed - having yet another map to detect duplicates seems pointless. Btw, it's only in the popup.
https://github.com/Chatterino/chatterino2/discussions/5289 - should make sure the emotes listed as well ordered
So this is functionally "complete" but still really slow. It tries to cache emote sets for channels that haven't resolved yet. But from my testing, all channels resolved equally slow, so there wouldn't be any benefit.
I'm waiting for Twitch to reply to the issues mentioned in the description and for them to potentially use better page sizes (some are tiny, costing a few RTTs - this adds up).
Although the API returns decent data, it's slow. For my user (two subscriptions), I need to make 28 requests to get all emotes for one channel. One request takes about 100-250ms. Now imagine how long this would take if a user has 50 (or 100) channels open and more subscriptions. I don't think it's my implementation (this would generate a report though the slow HTTP handler alerts).
Are you able to retest this? It doesn't seem right; I have 4 active subscriptions right now and it fetched everything in 6 calls (no broadcaster_id specified)
Are you able to retest this? It doesn't seem right; I have 4 active subscriptions right now and it fetched everything in 6 calls (no broadcaster_id specified)
It seems weird that no broadcaster_id was specified...
I've added some additional logging for each request and for the total (I'll probably remove this if this gets merged). For my user, it's actually 28 requests. They're probably from event emotes that have accumulated throughout the years (as you can see, the size of each page is small).
Logs
chatterino.twitch: [TwitchChannel "nerixyz"] Loading Twitch emotes - userID: "129546453", broadcasterID: "129546453", manualRefresh: false
chatterino.twitch: [TwitchChannel "nymn"] Loading Twitch emotes - userID: "129546453", broadcasterID: "62300805", manualRefresh: false
chatterino.twitch: [TwitchChannel "pajlada"] Loading Twitch emotes - userID: "129546453", broadcasterID: "11148817", manualRefresh: false
chatterino.twitch: [TwitchChannel "uint128"] Loading Twitch emotes - userID: "129546453", broadcasterID: "489584266", manualRefresh: false
chatterino.twitch: [TwitchChannel "nerixyz"] Got 16 more emote(s)
chatterino.twitch: [TwitchChannel "nymn"] Got 21 more emote(s)
chatterino.twitch: [TwitchChannel "pajlada"] Got 21 more emote(s)
chatterino.twitch: [TwitchChannel "uint128"] Got 16 more emote(s)
chatterino.twitch: [TwitchChannel "nerixyz"] Got 5 more emote(s)
chatterino.twitch: [TwitchChannel "nymn"] Got 5 more emote(s)
chatterino.twitch: [TwitchChannel "pajlada"] Got 5 more emote(s)
chatterino.twitch: [TwitchChannel "uint128"] Got 5 more emote(s)
chatterino.twitch: [TwitchChannel "nerixyz"] Got 5 more emote(s)
chatterino.twitch: [TwitchChannel "nymn"] Got 5 more emote(s)
chatterino.twitch: [TwitchChannel "uint128"] Got 5 more emote(s)
chatterino.twitch: [TwitchChannel "pajlada"] Got 5 more emote(s)
chatterino.twitch: [TwitchChannel "nymn"] Got 5 more emote(s)
chatterino.twitch: [TwitchChannel "nerixyz"] Got 5 more emote(s)
chatterino.twitch: [TwitchChannel "pajlada"] Got 5 more emote(s)
chatterino.twitch: [TwitchChannel "uint128"] Got 5 more emote(s)
chatterino.twitch: [TwitchChannel "nymn"] Got 5 more emote(s)
chatterino.twitch: [TwitchChannel "nerixyz"] Got 5 more emote(s)
chatterino.twitch: [TwitchChannel "pajlada"] Got 5 more emote(s)
chatterino.twitch: [TwitchChannel "nymn"] Got 5 more emote(s)
chatterino.twitch: [TwitchChannel "nerixyz"] Got 5 more emote(s)
chatterino.twitch: [TwitchChannel "uint128"] Got 5 more emote(s)
chatterino.twitch: [TwitchChannel "nerixyz"] Got 5 more emote(s)
chatterino.twitch: [TwitchChannel "nymn"] Got 5 more emote(s)
chatterino.twitch: [TwitchChannel "pajlada"] Got 5 more emote(s)
chatterino.twitch: [TwitchChannel "uint128"] Got 5 more emote(s)
chatterino.twitch: [TwitchChannel "nerixyz"] Got 5 more emote(s)
chatterino.twitch: [TwitchChannel "pajlada"] Got 5 more emote(s)
chatterino.twitch: [TwitchChannel "uint128"] Got 5 more emote(s)
chatterino.twitch: [TwitchChannel "nymn"] Got 5 more emote(s)
chatterino.twitch: [TwitchChannel "nerixyz"] Got 5 more emote(s)
chatterino.twitch: [TwitchChannel "pajlada"] Got 5 more emote(s)
chatterino.twitch: [TwitchChannel "nymn"] Got 5 more emote(s)
chatterino.twitch: [TwitchChannel "uint128"] Got 5 more emote(s)
chatterino.twitch: [TwitchChannel "pajlada"] Got 5 more emote(s)
chatterino.twitch: [TwitchChannel "nerixyz"] Got 10 more emote(s)
chatterino.twitch: [TwitchChannel "nymn"] Got 10 more emote(s)
chatterino.twitch: [TwitchChannel "uint128"] Got 5 more emote(s)
chatterino.twitch: [TwitchChannel "pajlada"] Got 10 more emote(s)
chatterino.twitch: [TwitchChannel "nerixyz"] Got 5 more emote(s)
chatterino.twitch: [TwitchChannel "nymn"] Got 5 more emote(s)
chatterino.twitch: [TwitchChannel "uint128"] Got 10 more emote(s)
chatterino.twitch: [TwitchChannel "pajlada"] Got 5 more emote(s)
chatterino.twitch: [TwitchChannel "nerixyz"] Got 5 more emote(s)
chatterino.twitch: [TwitchChannel "nymn"] Got 5 more emote(s)
chatterino.twitch: [TwitchChannel "uint128"] Got 5 more emote(s)
chatterino.twitch: [TwitchChannel "pajlada"] Got 5 more emote(s)
chatterino.twitch: [TwitchChannel "nerixyz"] Got 47 more emote(s)
chatterino.twitch: [TwitchChannel "nymn"] Got 47 more emote(s)
chatterino.twitch: [TwitchChannel "uint128"] Got 5 more emote(s)
chatterino.twitch: [TwitchChannel "pajlada"] Got 47 more emote(s)
chatterino.twitch: [TwitchChannel "nerixyz"] Got 5 more emote(s)
chatterino.twitch: [TwitchChannel "nymn"] Got 5 more emote(s)
chatterino.twitch: [TwitchChannel "uint128"] Got 47 more emote(s)
chatterino.twitch: [TwitchChannel "pajlada"] Got 5 more emote(s)
chatterino.twitch: [TwitchChannel "nerixyz"] Got 5 more emote(s)
chatterino.twitch: [TwitchChannel "nymn"] Got 5 more emote(s)
chatterino.twitch: [TwitchChannel "uint128"] Got 5 more emote(s)
chatterino.twitch: [TwitchChannel "pajlada"] Got 5 more emote(s)
chatterino.twitch: [TwitchChannel "nerixyz"] Got 5 more emote(s)
chatterino.twitch: [TwitchChannel "nymn"] Got 5 more emote(s)
chatterino.twitch: [TwitchChannel "uint128"] Got 5 more emote(s)
chatterino.twitch: [TwitchChannel "pajlada"] Got 5 more emote(s)
chatterino.twitch: [TwitchChannel "nerixyz"] Got 5 more emote(s)
chatterino.twitch: [TwitchChannel "nymn"] Got 5 more emote(s)
chatterino.twitch: [TwitchChannel "uint128"] Got 5 more emote(s)
chatterino.twitch: [TwitchChannel "pajlada"] Got 5 more emote(s)
chatterino.twitch: [TwitchChannel "nerixyz"] Got 4 more emote(s)
chatterino.twitch: [TwitchChannel "nymn"] Got 4 more emote(s)
chatterino.twitch: [TwitchChannel "uint128"] Got 5 more emote(s)
chatterino.twitch: [TwitchChannel "pajlada"] Got 4 more emote(s)
chatterino.twitch: [TwitchChannel "nymn"] Got 5 more emote(s)
chatterino.twitch: [TwitchChannel "nerixyz"] Got 5 more emote(s)
chatterino.twitch: [TwitchChannel "uint128"] Got 4 more emote(s)
chatterino.twitch: [TwitchChannel "pajlada"] Got 5 more emote(s)
chatterino.twitch: [TwitchChannel "nymn"] Got 46 more emote(s)
chatterino.twitch: [TwitchChannel "nerixyz"] Got 46 more emote(s)
chatterino.twitch: [TwitchChannel "uint128"] Got 5 more emote(s)
chatterino.twitch: [TwitchChannel "pajlada"] Got 46 more emote(s)
chatterino.twitch: [TwitchChannel "nymn"] Got 21 more emote(s)
chatterino.twitch: [TwitchChannel "nerixyz"] Got 21 more emote(s)
chatterino.twitch: [TwitchChannel "uint128"] Got 46 more emote(s)
chatterino.twitch: [TwitchChannel "pajlada"] Got 21 more emote(s)
chatterino.twitch: [TwitchChannel "nymn"] Got 5 more emote(s)
chatterino.twitch: [TwitchChannel "nerixyz"] Got 5 more emote(s)
chatterino.twitch: [TwitchChannel "uint128"] Got 21 more emote(s)
chatterino.twitch: [TwitchChannel "pajlada"] Got 5 more emote(s)
chatterino.twitch: [TwitchChannel "nymn"] Got 9 more emote(s)
chatterino.twitch: [TwitchChannel "nerixyz"] Got 9 more emote(s)
chatterino.twitch: [TwitchChannel "uint128"] Got 5 more emote(s)
chatterino.twitch: [TwitchChannel "pajlada"] Got 9 more emote(s)
chatterino.twitch: [TwitchChannel "nymn"] Got 5 more emote(s)
chatterino.twitch: [TwitchChannel "nerixyz"] Got 5 more emote(s)
chatterino.twitch: [TwitchChannel "uint128"] Got 9 more emote(s)
chatterino.twitch: [TwitchChannel "pajlada"] Got 5 more emote(s)
chatterino.twitch: [TwitchChannel "nymn"] Got 5 more emote(s)
chatterino.twitch: [TwitchChannel "nerixyz"] Got 5 more emote(s)
chatterino.twitch: [TwitchChannel "uint128"] Got 5 more emote(s)
chatterino.twitch: [TwitchChannel "pajlada"] Got 5 more emote(s)
chatterino.twitch: [TwitchChannel "nymn"] Got 5 more emote(s)
chatterino.twitch: [TwitchChannel "nerixyz"] Got 5 more emote(s)
chatterino.twitch: [TwitchChannel "uint128"] Got 5 more emote(s)
chatterino.twitch: [TwitchChannel "pajlada"] Got 5 more emote(s)
chatterino.twitch: [TwitchChannel "nymn"] Got 5 more emote(s)
chatterino.twitch: [TwitchChannel "nerixyz"] Got 5 more emote(s)
chatterino.twitch: [TwitchChannel "uint128"] Got 5 more emote(s)
chatterino.twitch: [TwitchChannel "pajlada"] Got 5 more emote(s)
chatterino.twitch: [TwitchChannel "nymn"] Got 275 more emote(s)
chatterino.twitch: [TwitchChannel "nymn"] Loaded 533 Twitch emotes (28 requests)
chatterino.twitch: [TwitchChannel "nerixyz"] Got 275 more emote(s)
chatterino.twitch: [TwitchChannel "nerixyz"] Loaded 528 Twitch emotes (28 requests)
chatterino.twitch: [TwitchChannel "uint128"] Got 5 more emote(s)
chatterino.twitch: [TwitchChannel "pajlada"] Got 275 more emote(s)
chatterino.twitch: [TwitchChannel "pajlada"] Loaded 528 Twitch emotes (28 requests)
chatterino.twitch: [TwitchChannel "uint128"] Got 275 more emote(s)
chatterino.twitch: [TwitchChannel "uint128"] Loaded 528 Twitch emotes (28 requests)
It seems weird that no broadcaster_id was specified...
Yeah we can paginate once without broadcaster_id, but we need to specify broadcaster_id for the channels we have joined but are not subscribed (can leverage userstate for this logic)
It seems weird that no broadcaster_id was specified...
Yeah we can paginate once without broadcaster_id, but we need to specify broadcaster_id for the channels we have joined but are not subscribed (can leverage userstate for this logic)
I believe 'Get Channel Emotes' would work better for this, as it requires no pagination to get all emotes
I believe 'Get Channel Emotes' would work better for this, as it requires no pagination to get all emotes
good point, can use Get Channel Emotes, just have to do separate api call to check if user is following the channel
good point, can use
Get Channel Emotes, just have to do separate api call to check if user is following the channel
I don't like this, but it's probably the best idea looking at the request count. My main concern is that Twitch could add more emotes that are local, and Get User Emotes would handle that just fine. There's a uservoice to add a parameter that would only return local emotes, which would be the ideal API.