[Panopto] Add Panopto extractors

[kmark]

Please follow the guide below

• You will be asked some questions, please read them carefully and answer honestly
• Put an x into all the boxes [ ] relevant to your pull request (like that [x])
• Use Preview tab to see how your pull request will actually look like

---

Before submitting a pull request make sure you have:

• [x] At least skimmed through adding new extractor tutorial and youtube-dl coding conventions sections
• [x] Searched the bugtracker for similar pull requests

In order to be accepted and merged into youtube-dl each piece of code must be in public domain or released under Unlicense. Check one of the following options:

• [x] I am the original author of this code and I am willing to release it under Unlicense
• [ ] I am not the original author of this code but it is in public domain or released under Unlicense (provide reliable evidence)

What is the purpose of your pull request?

• [ ] Bug fix
• [ ] Improvement
• [x] New extractor
• [ ] New feature

---

Description of your pull request and other information

No test cases are included as I am not aware of any publicly available Panopto recordings that this extractor will work with.

Supports downloading individual recordings or entire folders recursively. Folders are seperated with a ' -- ' in the playlist title.

Cookies are likely required to use this extractor specifically their .ASPXAUTH cookie which can be obtained from your browser after logging in.

--write-all-thumbnails can be used to download PowerPoint slides if they are not included as a video stream.

Suggested output format is 'out/%(playlist)s/%(title)s.%(ext)s'

[dstftw]

1. Read

youtube-dl coding conventions

and fix all issues. 2. You have to provide account credentials and concrete example URL for testing.

[kmark]

I've fixed all the issues noted by QuantifiedCode. I've also added tests for the Panopto extractor that both work on my machine but fail the Travis test. I have no idea why this is. Please let me know if there are any further places where conventions aren't being followed.

As noted in one of my commits:

Testing may be impossible for the Folder extractor, or I'm just doing it wrong. With the current test we enter a catch-22 where it claims we need an 'ext' entry to continue testing but upon adding that it claims it expected None.

I've still included the faulty PanoptoFolder test for review and reproduction of this issue.

Thanks!

[kmark]

The Travis errors appear to be from ffmpeg not being available:

DownloadError: ERROR: m3u8 download detected but ffmpeg or avconv could not be found. Please install one.

Once ffmpeg is installed it fixed the above error but the tests still failed with MD5 mismatches. It looks like the --test option isn't working properly on Linux:

$ python2.7 -m youtube_dl --test https://demo.hosted.panopto.com/Panopto/Pages/Viewer.aspx?id=26b3ae9e-4a48-4dcc-96ba-0befba08a0fb
$ wc -c Panopto\ for\ Business-26b3ae9e-4a48-4dcc-96ba-0befba08a0fb.mp4
13663 Panopto for Business-26b3ae9e-4a48-4dcc-96ba-0befba08a0fb.mp4
$ md5sum Panopto\ for\ Business-26b3ae9e-4a48-4dcc-96ba-0befba08a0fb.mp4
76d6bc8e500b1e47b53541514e3d1ea6  Panopto for Business-26b3ae9e-4a48-4dcc-96ba-0befba08a0fb.mp4

or macOS:

$ python2.7 -m youtube_dl --test https://demo.hosted.panopto.com/Panopto/Pages/Viewer.aspx?id=26b3ae9e-4a48-4dcc-96ba-0befba08a0fb
$ gwc -c Panopto\ for\ Business-26b3ae9e-4a48-4dcc-96ba-0befba08a0fb.mp4
13679 Panopto for Business-26b3ae9e-4a48-4dcc-96ba-0befba08a0fb.mp4
$ gmd5sum Panopto\ for\ Business-26b3ae9e-4a48-4dcc-96ba-0befba08a0fb.mp4
06fb292a3510aa5bc5f0c950fe58c9f7  Panopto for Business-26b3ae9e-4a48-4dcc-96ba-0befba08a0fb.mp4

Different lengths means both platforms will generate different MD5 sums during testing. If I truncate both to 10241 bytes I get the same MD5 but neither platform will test correctly against that MD5.

It's worth adding that if I select -f 0 to avoid M3U8/ffmpeg stuff then both platforms properly truncate to 10241 bytes.

[kmark]

Looks like the issue is with youtube-dl assuming that the -fs option of FFmpeg will limit the file size to 10241 bytes in a deterministic fashion. According to the documentation it doesn't guarantee a final file size and even claims the download will be larger by some undefined degree:

Set the file size limit, expressed in bytes. No further chunk of bytes is written after the limit is exceeded. The size of the output file is slightly more than the requested file size.

So we have two different versions of FFmpeg on two different platforms with FFmpeg's -fs option producing two different final file sizes. When I bring the FFmpeg versions closer together on both platforms (latest or near-latest from master) the file sizes are equal again. Unless I'm misunderstanding something about youtube-dl, I would put this in the bug category.

Edit: Even when I brought the FFmpeg versions closer together (roughly 19 days apart) the MD5s were still different. I brought both up to N-86744-gfe92422 and only then was I able to get identical MD5s.

[manuel-uberti]

Hello, any update on this? Thank you for working on it.

[kmark]

Hey Manuel, I haven't touched this or tried to use it since 2017. I imagine that it may need to be updated if Panopto has changed their (semi-private) API. This was waiting on another round of reviews, I believe I had addressed all of the outstanding comments.

As I mentioned earlier there seemed to be a bug in the way youtube-dl was doing its tests against FFmpeg, relying on the -fs option to behave deterministically across versions, which it does not. I opened up a PR to try and address this but did not receive any feedback on the second iteration of the patch. I am not sure if this is still an outstanding issue.

If there are users willing to test this patch and perhaps even work on it I would certainly appreciate it, and I'm sure the maintainers would too.

[rgreenblatt]

I tested this PR and I was able to get things working (just test/test_download.py and an example private video) after a few changes (see review).

[jxu]

https://github.com/jstrieb/panopto-download I just found about this that might help. I looked briefly through the PR and saw the links looked like they were hardcoded. They are available through the RSS feed of each folder currently.

[rgreenblatt]

https://github.com/jstrieb/panopto-download I just found about this that might help. I looked briefly through the PR and saw the links looked like they were hardcoded. They are available through the RSS feed of each folder currently.

Interestingly, the links in the RSS feed don't require any authentication, so it should be possible to convert from the folder URL to the RSS feed url and download all the videos without requiring cookies. Unfortunately, I don't think its possible to get the RSS feed url directly from a viewer video url, so cookies are needed when downloading a single video.

For my personal use cases, using the RSS feed directly is perfect. I set up a daemon which periodically runs youtube-dl on the RSS feed and sends notifications when a new video is downloaded. See this script from my dotfiles. This doesn't require this PR at all.

For the use case of downloading a single video, I think that using the RSS feed would increase complexity.

For the folder use case, I think that using the RSS should be considerably cleaner, but I don't know how that would work for recursive folder downloads. Also, I haven't tested recursive folder downloads, so it is possible that they are broken right now. I probably won't write this change for a few reasons, but I think this would be reasonably easy to do (aside from subfolders perhaps).

[coletdjnz]

Reviving this in yt-dlp: https://github.com/yt-dlp/yt-dlp/pull/2908

[lebdron]

@dirkf Hi! Any chance to get this merged? Thanks!

[jxu]

it's merged in yt-dlp btw

[dirkf]

We should back-port the yt-dlp extractor unless it does something weird wrt yt-dl.

[coletdjnz]

iirc there were some bugs in core that had to be fixed for panopto to work completely, so those will need to be backported too

[jxu]

at the risk of sounding like a broken record, if you compare youtube-dl commits and bufixes/new features to yt-dlp, youtube-dl as (mostly) controlled by @dirkf will always be behind in a project that has to keep up with constantly changing unofficial APIs

[lebdron]

@jxu no worries, I am working on a backport https://github.com/ytdl-org/youtube-dl/pull/31097 so that there will be at least some implementation

[jxu]

@lebdron I digress (rant): from what I can tell, unnecessary energy is spent backporting, but that is the reality of having a better maintained fork compete with the formerly undisputed project and fracturing the community. Especially when the project insists on supporting python2 and the 0.1% of people who choose run youtube-dl on an ancient embedded device that somehow doesn't have python3 but those people put the burden on contributors.

[dirkf]

@jxu can feel free not to spend any effort back-porting code to yt-dl, and should probably ignore yt-dl altogether.

Meanwhile thx @lebdron: please include "closes # 13449" in your PR text (but without the space after the #).