yt-dlp icon indicating copy to clipboard operation
yt-dlp copied to clipboard

[extractor/twitch] Skip errors in GQL response

Open elyse0 opened this issue 2 years ago • 3 comments

IMPORTANT: PRs without the template will be CLOSED

Description of your pull request and other information

This PR fixes the problem where Twitch's GQL API sometimes returns an error before the actual data, breaking the extractor that assumed the video data would be at index 0.

Full json response: https://gist.github.com/elyse0/3b1679b6a0b8d7aa8112f75c1eb52403

Trimmed json

[
  {
    "errors": [
      {
        "message": "PersistedQueryNotFound"
      }
    ],
    "extensions": {
      "durationMilliseconds": 3,
      "operationName": "VideoMetadata",
      "requestID": "01GSTXCZDDA9GN7ZHBRSB9MP4Y"
    }
  },
  {
    "data": {
      "video": {
        "id": "1697007802",
        "moments": {
          ...
          "__typename": "VideoMomentConnection"
        },
        "__typename": "Video"
      }
    },
    "extensions": {
      "durationMilliseconds": 61,
      "operationName": "VideoPlayer_ChapterSelectButtonVideo",
      "requestID": "01GSTXCZDDA9GN7ZHBRSB9MP4Y"
    }
  },
  {
    "data": {
      "video": {
        "id": "1697007802",
        "seekPreviewsURL": "https://d1m7jfoe9zdc1j.cloudfront.net/66e41701aa009d198649_vickypalami_40717337225_1672796209/storyboards/1697007802-info.json",
        "__typename": "Video"
      }
    },
    "extensions": {
      "durationMilliseconds": 25,
      "operationName": "VideoPlayer_VODSeekbarPreviewVideo",
      "requestID": "01GSTXCZDDA9GN7ZHBRSB9MP4Y"
    }
  }
]

Fixes #6308

Template

Before submitting a pull request make sure you have:

In order to be accepted and merged into yt-dlp each piece of code must be in public domain or released under Unlicense. Check all of the following options that apply:

  • [x] I am the original author of this code and I am willing to release it under Unlicense
  • [ ] I am not the original author of this code but it is in public domain or released under Unlicense (provide reliable evidence)

What is the purpose of your pull request?

elyse0 avatar Feb 21 '23 21:02 elyse0

With this patch applied, stream titles will randomly fail to be fetched. Which VOD fails to title fetch varies on every run of the for loop below:

> for /f %a in (vid) do @yt-dlp --get-title %a
Untitled Broadcast
The game in the year of our lord 2023 without FOV options 🤣🤣🤣
BIRTHDAY SPEEDRUNS
Returnal
Untitled Broadcast
SPOOKY SUNDAY - The Karaoke
Untitled Broadcast
Any Goblin Caves Enjoyers?

[ R:\ ]
> for /f %a in (vid2) do @yt-dlp --get-title %a
Untitled Broadcast
Untitled Broadcast
Untitled Broadcast
Elden Ring Any% Glitchless WR Attempts | Atomic Hearts/Rivals Later w/ !team
Untitled Broadcast
Untitled Broadcast
PURE DARKNESS Elden Ring in FIRST PERSON | Horror Mode | !Vods (I won the !KemonoQuest)
Beating Elden Ring in FIRST PERSON | All Remembrances | !Vods (I won the !KemonoQuest)

vid contents: https://0x0.st/Hs3g.txt vid2 contents: https://0x0.st/Hs3E.txt

My yt-dlp environment:

[debug] Command-line config: ['-v']
[debug] User config "C:\Users\garoto\yt-dlp.conf": ['--extractor-args', 'youtube:skip=translated_subs', '--external-downloader-args', '-hide_banner -loglevel warning -stats', '--no-sponsorblock', '--no-mtime', '--no-warnings', '--ignore-errors', '--force-ipv4', '--restrict-filenames', '--merge-output-format', 'mkv', '--sub-lang', 'en.*,pt,pt-BR', '--write-sub', '--embed-subs', '--convert-subs', 'srt', '--format', '(bestvideo[width<=1920][fps<=60][vcodec^=vp9]+bestaudio[acodec=opus])/299+140/137+251/best/bestvideo+bestaudio', '--output', '%(title).70s-%(upload_date)s-%(uploader)s-%(extractor_key)s_%(id).19s.%(ext)s']
[debug] Encodings: locale cp1252, fs utf-8, pref cp1252, out utf-8 (No VT), error utf-8 (No VT), screen utf-8 (No VT)
[debug] yt-dlp version 2023.02.17 [a0a7c01] (pip)
[debug] Python 3.8.2 (CPython AMD64 64bit) - Windows-8.1-6.3.9600-SP0 (OpenSSL 1.1.1d  10 Sep 2019)
[debug] exe versions: ffmpeg 4.0.2, ffprobe 4.0.2
[debug] Optional libraries: Cryptodome-3.17, brotli-1.0.9, certifi-2022.12.07, mutagen-1.46.0, sqlite3-2.6.0, websockets-10.4
[debug] Proxy map: {}
[debug] Loaded 1782 extractors

garoto avatar Feb 24 '23 15:02 garoto

Looking at the extractor and error more closely, the problem was that some _OPERATION_HASHES were not up to date

I've updated these hashes and updated some tests

This shoud fix the titles @garoto

elyse0 avatar Feb 24 '23 21:02 elyse0

This shoud fix the titles @garoto

Tested a bunch of VODs and it seems to be fixed indeed. Thank you!

garoto avatar Feb 25 '23 13:02 garoto