Temporary error "Invalid data found when processing input"

[edrozenberg]

Describe the bug On a Twitter space that recently ended, saw this error for some minutes. Retrying the same download again a bit later worked fine. Maybe just a transient Twitter issue with the data they provided for a recently ended space, and not something twspace-dl needs to worry about.

To Reproduce Tried to download a recently ended space (ended a few minutes prior):

$ twspace_dl -m -i https://twitter.com/i/spaces/1DXxyDqgdbNJM?s=20

Expected behavior Download the space successfully

Output

./tmppau4qu6q/(Mriya Report)MriyaReport - Russian Invasion of 🇺🇦 Crimea Actions ⛑️ via MriyaAid.org-1DXxyDqgdbNJM.m3u8: Invalid data found when processing input
Error: ffmpeg -protocol_whitelist file,https,tls,tcp -y -stats -v warning -i ./tmppau4qu6q/(Mriya Report)MriyaReport - Russian Invasion of 🇺🇦 Crimea Actions ⛑️ via MriyaAid.org-1DXxyDqgdbNJM.m3u8 -c copy -metadata title=MriyaReport - Russian Invasion of 🇺🇦 Crimea Actions ⛑️ via MriyaAid.org -metadata artist=Mriya Report -metadata episode_id=1DXxyDqgdbNJM ./tmppau4qu6q/(Mriya Report)MriyaReport - Russian Invasion of 🇺🇦 Crimea Actions ⛑️ via MriyaAid.org-1DXxyDqgdbNJM.m4a
Retry with -v to see more details
[eduardr@lion ~/Desktop/2022-08-10_1]$  twspace_dl -m -v -i https://twitter.com/i/spaces/1DXxyDqgdbNJM?s=20
2022-08-10 11:55:37,119 [DEBUG] Starting new HTTPS connection (1): api.twitter.com:443
2022-08-10 11:55:37,208 [DEBUG] https://api.twitter.com:443 "POST /1.1/guest/activate.json HTTP/1.1" 200 63
2022-08-10 11:55:37,213 [DEBUG] Starting new HTTPS connection (1): twitter.com:443
2022-08-10 11:55:37,473 [DEBUG] https://twitter.com:443 "GET /i/api/graphql/jyQ0_DEMZHeoluCgHJ-U5Q/AudioSpaceById?variables=%7B%22id%22%3A%221DXxyDqgdbNJM%22%2C%22isMetatagsQuery%22%3Afalse%2C%22withSuperFollowsUserFields%22%3Atrue%2C%22withUserResults%22%3Atrue%2C%22withBirdwatchPivots%22%3Afalse%2C%22withReactionsMetadata%22%3Afalse%2C%22withReactionsPerspective%22%3Afalse%2C%22withSuperFollowsTweetFields%22%3Atrue%2C%22withReplays%22%3Atrue%2C%22withScheduledSpaces%22%3Atrue%7D HTTP/1.1" 200 8031
2022-08-10 11:55:37,476 [DEBUG] Media Key: 28_1557383083459952641
2022-08-10 11:55:37,480 [DEBUG] Starting new HTTPS connection (1): cdn.syndication.twimg.com:443
2022-08-10 11:55:37,549 [DEBUG] https://cdn.syndication.twimg.com:443 "GET /widgets/followbutton/info.json?screen_names=MriyaReport HTTP/1.1" 200 156
2022-08-10 11:55:37,559 [DEBUG] Starting new HTTPS connection (1): twitter.com:443
2022-08-10 11:55:37,769 [DEBUG] https://twitter.com:443 "GET /i/api/1.1/live_video_stream/status/28_1557383083459952641 HTTP/1.1" 200 1498
2022-08-10 11:55:37,775 [DEBUG] Starting new HTTPS connection (1): prod-fastly-eu-central-1.video.pscp.tv:443
2022-08-10 11:55:38,020 [DEBUG] https://prod-fastly-eu-central-1.video.pscp.tv:443 "GET /Transcoding/v1/hls/zEg3Tz98sqlLU67tjex73hlyWdJsIfdcR11UO7Dj41CTBXT_Wsma_HszidusGs0qbpeGcwyJvswODSeDyJcmTQ/non_transcode/eu-central-1/periscope-replay-direct-prod-eu-central-1-public/audio-space/master_playlist.m3u8 HTTP/1.1" 200 550
2022-08-10 11:55:38,025 [DEBUG] Starting new HTTPS connection (1): prod-fastly-eu-central-1.video.pscp.tv:443
2022-08-10 11:55:38,282 [DEBUG] https://prod-fastly-eu-central-1.video.pscp.tv:443 "GET /Transcoding/v1/hls/zEg3Tz98sqlLU67tjex73hlyWdJsIfdcR11UO7Dj41CTBXT_Wsma_HszidusGs0qbpeGcwyJvswODSeDyJcmTQ/transcode/eu-central-1/periscope-replay-direct-prod-eu-central-1-public/eyJhbGciOiJIUzI1NiIsInR5cCI6IkpXVCIsInZlcnNpb24iOiIyIn0.eyJIZWlnaHQiOjgsIkticHMiOjM1MCwiVHJhbnNjb2RlQXVkaW8iOnRydWUsIldpZHRoIjo4fQ.j9A1sJbguxIY34C17U6iN7yTx8JM_ZNYJhfxaSmwGKQ/audio-space/playlist_16786592591867834374.m3u8 HTTP/1.1" 404 10
2022-08-10 11:55:38,285 [DEBUG] ./tmp0d075lpp/(Mriya Report)MriyaReport - Russian Invasion of 🇺🇦 Crimea Actions ⛑️ via MriyaAid.org-1DXxyDqgdbNJM.m3u8 written to disk
2022-08-10 11:55:38,285 [DEBUG] Command for the old part: ffmpeg -protocol_whitelist file,https,tls,tcp -y -stats -v warning -i ./tmp0d075lpp/(Mriya Report)MriyaReport - Russian Invasion of 🇺🇦 Crimea Actions ⛑️ via MriyaAid.org-1DXxyDqgdbNJM.m3u8 -c copy -metadata title=MriyaReport - Russian Invasion of 🇺🇦 Crimea Actions ⛑️ via MriyaAid.org -metadata artist=Mriya Report -metadata episode_id=1DXxyDqgdbNJM ./tmp0d075lpp/(Mriya Report)MriyaReport - Russian Invasion of 🇺🇦 Crimea Actions ⛑️ via MriyaAid.org-1DXxyDqgdbNJM.m4a
./tmp0d075lpp/(Mriya Report)MriyaReport - Russian Invasion of 🇺🇦 Crimea Actions ⛑️ via MriyaAid.org-1DXxyDqgdbNJM.m3u8: Invalid data found when processing input
Traceback (most recent call last):
  File "/usr/local/lib/python3.9/site-packages/twspace_dl/twspace_dl.py", line 171, in download
    subprocess.run(cmd_old, check=True)
  File "/usr/local/Cellar/[email protected]/3.9.13_2/Frameworks/Python.framework/Versions/3.9/lib/python3.9/subprocess.py", line 528, in run
    raise CalledProcessError(retcode, process.args,
subprocess.CalledProcessError: Command '['ffmpeg', '-protocol_whitelist', 'file,https,tls,tcp', '-y', '-stats', '-v', 'warning', '-i', './tmp0d075lpp/(Mriya Report)MriyaReport - Russian Invasion of 🇺🇦 Crimea Actions ⛑️ via MriyaAid.org-1DXxyDqgdbNJM.m3u8', '-c', 'copy', '-metadata', 'title=MriyaReport - Russian Invasion of 🇺🇦 Crimea Actions ⛑️ via MriyaAid.org', '-metadata', 'artist=Mriya Report', '-metadata', 'episode_id=1DXxyDqgdbNJM', './tmp0d075lpp/(Mriya Report)MriyaReport - Russian Invasion of 🇺🇦 Crimea Actions ⛑️ via MriyaAid.org-1DXxyDqgdbNJM.m4a']' returned non-zero exit status 1.

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "/usr/local/bin/twspace_dl", line 8, in <module>
    sys.exit(main())
  File "/usr/local/lib/python3.9/site-packages/twspace_dl/__main__.py", line 231, in main
    args.func(args)
  File "/usr/local/lib/python3.9/site-packages/twspace_dl/__main__.py", line 123, in space
    twspace_dl.download()
  File "/usr/local/lib/python3.9/site-packages/twspace_dl/twspace_dl.py", line 173, in download
    raise RuntimeError(" ".join(err.cmd)) from err
RuntimeError: ffmpeg -protocol_whitelist file,https,tls,tcp -y -stats -v warning -i ./tmp0d075lpp/(Mriya Report)MriyaReport - Russian Invasion of 🇺🇦 Crimea Actions ⛑️ via MriyaAid.org-1DXxyDqgdbNJM.m3u8 -c copy -metadata title=MriyaReport - Russian Invasion of 🇺🇦 Crimea Actions ⛑️ via MriyaAid.org -metadata artist=Mriya Report -metadata episode_id=1DXxyDqgdbNJM ./tmp0d075lpp/(Mriya Report)MriyaReport - Russian Invasion of 🇺🇦 Crimea Actions ⛑️ via MriyaAid.org-1DXxyDqgdbNJM.m4a

Desktop (please complete the following information):

• OS: MacOS 11.6.8 (20G730) (x86)
• Version: 2022.6.6.1
• Installation method: pip (python3)

Additional context Workaround: wait some time and retry the same download. Appears to be a temporary Twitter issue with the data Twitter returns shortly after a space ends. Wait a few minutes or more and try the same download again, it should work.

[Ryu1845]

This is weird, hmm. Maybe I should add some kind of auto retry in these cases.

[edrozenberg]

Auto-retry would probably help, but don't know if it would better retry every 1 minute say, or every 5 minutes.

[Ryu1845]

#70 should add a maximum of 5 retry every 30 seconds if necessary. Could you confirm @exzhawk?

[exzhawk]

#70 should add a maximum of 5 retry every 30 seconds if necessary. Could you confirm @exzhawk?

maybe. but this retry has nothing to do with the retry in requests (in #70 ).

A retry mechanism can be implemented. I wonder if the script can know that it's encountering an error that can be solved by retrying later. Probing before invoking FFmpeg might be helpful. just my two cents.

[Ryu1845]

I added an error message telling the user to try again later.