Ganymede Doesn't See Logged-In-Only 4K Quality

[VXsz]

Expected Behavior

I went to check a vod the other day after it got archived, and I was surprised to see this (Sign up or log in to view higher resolutions. Log In) on 4K quality, I realized Ganymede only archived the 720p60 quality which is the highest publicly available quality:

Is there a way to grab them automatically? Ganymede has all the login info already, I think it is a simple fix in that case but not sure, it could pose issues with ratelimits if live status checks are made with auth.

It's AVC btw, here's yt-dlp output:

ID         EXT   RESOLUTION FPS │   FILESIZE   TBR PROTO │ VCODEC      ACODEC     ABR MORE INFO
────────────────────────────────────────────────────────────────────────────────────────────────
sb1        mhtml 110x62       0 │                  mhtml │ images                     storyboard
sb0        mhtml 220x124      0 │                  mhtml │ images                     storyboard
Audio_Only mp4   audio only     │ ~245.70MiB  322k m3u8  │ audio only  mp4a.40.2 322k
160p       mp4   284x160     30 │ ~220.23MiB  289k m3u8  │ avc1.4D000C mp4a.40.2
360p       mp4   640x360     30 │ ~540.90MiB  709k m3u8  │ avc1.4D001E mp4a.40.2
480p       mp4   852x480     30 │ ~  1.16GiB 1558k m3u8  │ avc1.4D001F mp4a.40.2
720p60     mp4   1280x720    60 │ ~  2.63GiB 3534k m3u8  │ avc1.4D0020 mp4a.40.2
2160p60    mp4   3840x2160   60 │ ~  6.36GiB 8543k m3u8  │ avc1.640034 mp4a.40.2      Source

This can be easily solved with those sub-only bypass methods.

VOD: https://www.twitch.tv/videos/2477176808

Current Behavior

Doesn't grab (see?) the 4K quality.

Possible Solution

Ganymede already requires login data and such, I think it would trivial to implement this in that case. However ratelimits need to be watched, maybe they are better or worse. Maybe there's a better way down the toolchain with TwitchDownloaderCLI, I have opened an issue there https://github.com/lay295/TwitchDownloader/issues/1445. But I think ganymede can resolve this without going that way.

Steps to Reproduce

1. Wait for a 4K streamer to stream in 4K
2. Ganymede only sees 720p60 (or the highest available to logged out users)
3. ???
4. Loss

Version

v4.3.1 (v4.4.0 doesn't have any relevant patches for this)

Server logs

[cli][info] Found matching plugin hls for URL hls://https://lb-eu2.cdn-perfprod.com/playlist/m5aoi.m3u8%3Fplayer%3Dtwitchweb%26type%3Dany%26allow_source%3Dtrue%26allow_audio_only%3Dtrue%26allow_spectre%3Dfalse%26fast_bread%3Dtrue
[cli][info] Available streams: audio_only, 160p (worst), 360p, 480p, 720p60 (best)
[cli][info] Opening stream: 720p60 (hls)
[cli][info] Writing output to
/data/temp/321224350585_85f527c4-4184-11f0-bf41-0242ac100d03-video.mp4
...
stream ended etc

[Zibbp]

Interesting that the 2160p quality is locked behind logging in for you. For me I'm able to view it in an incognito window.

Additionally my unauthenticated streamlink download (which is used for live stream and VOD download) seems to work fine.

zibbp@desktop:~$ streamlink https://www.twitch.tv/videos/2477176808 best -o foo.mp4
[cli][info] Found matching plugin twitch for URL https://www.twitch.tv/videos/2477176808
[cli][info] Available streams: audio, 160p (worst), 360p, 480p, 720p60, 2160p60 (best)
[cli][info] Opening stream: 2160p60 (hls)
[cli][info] Writing output to

My development instance also downloads it in 2160p fine.

[cli][info] Found matching plugin twitch for URL https://twitch.tv/videos/2477176808
[cli][info] Available streams: audio, 160p (worst), 360p, 480p, 720p60, 2160p60 (best)
[cli][info] Opening stream: 2160p60 (hls)
[cli][info] Writing output to
/data/temp/2477176808_c73675fb-448c-11f0-aa65-16c20346aef9-video.mp4

Maybe locking the higher quality is some new feature Twitch is demoing out? Are you using a proxy at all? And do you have your Twitch token in Admin > Settings?

[VXsz]

Huh that's weird, is twitch really trolling me with these A/B tests? lol. Actually I just tested out with a clean "random" IP with my mobile, and a clean browser environment and it still shows me that sign and locks the 4K on their site. I even asked a couple of friends from a completely different cities in the same country, and it shows the same lock on incognito. Could this be region/country-level? that'd be nuts ngl. I'm a quality maniac so this kinda triggers me.

do you have your Twitch token in Admin > Settings?

I don't, I kinda thought I had it, maybe that'd solve it. But this also shows another issue I've been wanting to ask which prevents me from testing this right now. Is there a way to save vod that also have their lives archived? It gives me a (slightly cryptic) error message about it already existing since I have the livestream archived. Usually VODs are nicer to have due to more complete streams (lives sometimes fail, start a bit late or get cut up) and the ad-less proxies sometimes fail, and the ads are pretty obnoxious.

[Zibbp]

Maybe A/B tests, or bandwidth is crazy expensive where you live 🤷 ?

The external ID is used as the identifier when checking iv the VOD has already been archived. So it's unlikely you can archive both. You can try editing the External ID of the live stream you have archived then archive the VOD. Usually users prefer the live streams because of dual audio tracks not being present in VODs.

[ccKep]

Just thought I'd add: I'm getting the same lock on the VOD linked above in incognito mode (Germany)

[VXsz]

Fix is out :) https://github.com/lay295/TwitchDownloader/releases/tag/1.55.9

(Thanks for the quick fix @ScrubN)

[Zibbp]

I use Streamlink for downloading streams and videos so we'll need to wait for the maintainers over there to support HEVC/fragmented MP4.

[BigFlubba]

Just thought I'd add: I'm getting the same lock on the VOD linked above in incognito mode (Germany)

Same here (USA)

[Zibbp]

I will likely need to replace streamlink with yt-dlp (see why here https://github.com/Zibbp/ganymede/pull/805). yt-dlp already supports enhanced broadcasting support. Though in my experience you need to provide your twitch token to even see >1080p quality.

[BigFlubba]

When I rebuilt it last night Streamlink updated to 7.5.0. I noticed your fix fir it, but now I have 9 VODs that are out of sync. I wish they had the option to disable it rather then it being force enabled. I don't think having to supply your token is that high of a bar because most already do for ad removal.

[Zibbp]

https://github.com/Zibbp/ganymede/pull/812 has been merged which replaces streamlink with yt-dlp/ffmpeg. Everything in my testing worked fine but I'm sure something else will pop up. This means enhanced broadcasting/4K support is here. You can test this now in the :dev image. No breaking changes except for any custom streamlink arguments no longer working.

Important Information about yt-dlp/Enhanced Broadcasting

Enhanced broadcasting (4K/hevc) moves away from mpeg-ts to fragmented MP4 containers as the container in the HLS playlist. This is extremely annoying as fMP4 is super sensitive to disruption, compared to mpeg-ts. Ultimately this means if you archive an enhanced broadcast stream/4K stream without an ad-free stream (subscriber or turbo) it will not play in the browser if an ad is served. The resulting video stream is "corrupted" enough that web players cannot handle it. If you have turbo or a a subscriber with your Twitch token then this is not an issue. This is also not an issue if you use a proxy, though that will not be 4K. I've found that channels that participate in the enhanced broadcasting only serve fMP4 and not mpeg-ts even for lower resolutions.

I've tried to work around the "corrupted/broken" live streams with ads for enhanced broadcasts but I cannot find a suitable solution. Open to ideas if anyone has any!

[ScrubN]

Open to ideas if anyone has any!

I may or may not have been experimenting with the Twitch API and may have found a way to get the VOD link of currently live streams (even if the VOD is private). With enough work, it would be possible for TD to incrementally download segments as the stream is being broadcasted.

The main issues with this idea are:

1. If the VOD is immediately deleted after the stream ends, there would be some missing segments at the end.
2. With TwitchDownloader being such a large project, it might provoke Twitch into patching the method.
3. The method involves bruteforcing the Amazon IVS subdomain, which might be a problem for users monitoring a large number of channels that go live with private VODs around the same time

[Zibbp]

I may or may not have been experimenting with the Twitch API and may have found a way to get the VOD link of currently live streams (even if the VOD is private). With enough work, it would be possible for TD to incrementally download segments as the stream is being broadcasted.

I've thought about doing something similar but issue with private VODs or even disabled VODs always prevented that. Incrementally downloading would fix the fMP4 but introduces some other issue such as presumably being X time behind the "edge" of the live stream. It's likely a workaround like you suggested would get patched, though Twitch seems to generally not care or is slow to fix pretty glaring issues.

I've also thought about downloading the mp4 chunks for the playlist and processing them kind of in real time to see if it's a "corrupt/advertisement" chunk and simply generating a blank video to replace the location in the stream. Really hacky but functionally it would be the same as current non-enhanced broadcast behavior without an ad-free stream of seeing the "ads in progress" screen.

[Zibbp]

I saw that Twitch is also (finally) releasing a rewind feature (behind a paywall lol). I wonder if that could be used to our advantage. I would think the old segments are demoted back to their VOD/cloudfront setup rather than the edge servers, kinda like how visiting a VOD of a livestream works now.