youtube-dl
youtube-dl copied to clipboard
ytdl-org
Add build workflow
Please follow the guide below
- You will be asked some questions, please read them carefully and answer honestly
- Put an
xinto all the boxes [ ] relevant to your pull request (like that [x]) - Use Preview tab to see how your pull request will actually look like
Before submitting a pull request make sure you have:
- [x] Searched the bugtracker for similar pull requests
- [ ] Read adding new extractor tutorial
- [ ] Read youtube-dl coding conventions and adjusted the code to meet them
- [ ] Covered the code with tests (note that PRs without tests will be REJECTED)
- [x] Checked the code with flake8
(Forgive me for not checking some of them)
In order to be accepted and merged into youtube-dl each piece of code must be in public domain or released under Unlicense. Check one of the following options:
- [x] I am the original author of this code and I am willing to release it under Unlicense
- [x] I am not the original author of this code but it is in public domain or released under Unlicense (provide reliable evidence)
yt-dlp build workflow and devscripts/update-version.py for latter.
What is the purpose of your pull request?
- [ ] Bug fix
- [x] Improvement
- [ ] New extractor
- [ ] New feature
Description of your pull request and other information
The goal for this PR is to give this repository a build workflow, enough to create a release except signatures (aka .sig).
It also reproduces how the youtube-dl.exe was built.
The files are reduced version to create a daily releases, so I need some fixes.
Big thanks for yt-dlp build workflow which this PR is based on. ~~(forgive me for being lazy while ESA is in session)~~ ESA ended.
TODO
- [x] Fix version manipulation (currently it appends
.114514) - [x] Create changelogs (untested)
- [x] Final test (https://github.com/Lesmiscore/youtube-dl/releases/tag/2022.02.20)
Known Notes:
- Has to set a secret with key
PUSH_VERSION_COMMIT. Value can be "anything". - To fully reproduce Releases as it were, you need the private key to sign binaries. That's not what I can have. Either dstftw or remitamine absolutely has it.
Closes #31067.
GH-flavored markdown render test:
version 2021.12.17
Core
* [postprocessor/ffmpeg] Show ffmpeg output on error (#22680, #29336)
Extractors
* [youtube] Update signature function patterns (#30363, #30366)
* [peertube] Only call description endpoint if necessary (#29383)
* [periscope] Pass referer to HLS requests (#29419)
- [liveleak] Remove extractor (#17625, #24222, #29331)
+ [pornhub] Add support for pornhubthbh7ap3u.onion
* [pornhub] Detect geo restriction
* [pornhub] Dismiss tbr extracted from download URLs (#28927)
* [curiositystream:collection] Extend _VALID_URL (#26326, #29117)
* [youtube] Make get_video_info processing more robust (#29333)
* [youtube] Workaround for get_video_info request (#29333)
* [bilibili] Strip uploader name (#29202)
* [youtube] Update invidious instance list (#29281)
* [umg:de] Update GraphQL API URL (#29304)
* [nrk] Switch psapi URL to https (#29344)
+ [egghead] Add support for app.egghead.io (#28404, #29303)
* [appleconnect] Fix extraction (#29208)
+ [orf:tvthek] Add support for MPD formats (#28672, #29236)
No. Great work by everyone, and thanks, and apologies for taking so long to follow this up.
To do:
- [ ] signing key
- [ ] complete testing in my dev repo
- [ ] add bundled build for Pythonless MacOS and any other Pythonly challenged Unixes, following yt-dlp
add bundled build for Pythonless MacOS and any other Pythonly challenged Unixes, following yt-dlp
You'll need to use pyinstaller for this and not py2exe
add bundled build for Pythonless MacOS and any other Pythonly challenged Unixes, following yt-dlp
IMO that's unnecessary, because afaik all those devices will be compatible with yt-dlp (at least the macbooks, I'm not sure about the other unixes).
signing key
and speaking of the signing key, I wonder if @dirkf would be willing to share the signing key with @pukkandan, so that yt-dlp can also be signed (and hence won't be flagged by any antiviruses, although that's not a major issue nowadays)
the signing key with
@pukkandan, so that yt-dlp can also be signed
The signing part implies .sig files like there
For a signing keys for EXE files (which you meant) must NEVER be done as you're asking to share private keys (possibly) containing dirkf's personal info!
edit: youtube-dl.exe from version 2021.12.17 isn't signed, for a PE file
@MrRawes, please feel free to make your comment against the relevant line in the PR.
Arguably, unless you write <fromfile command args ..., the cat fromfile | command args ... pattern is clearer. No doubt that discussion is in the WP article that I couldn't be bothered to read (again).
... Sorry for barging in :wink:, I not being a python coder :blush:, but,
- Does this PR aim to replicate "in a way" the automated
youtube-dlreleases to be found here? - I'm asking because those binaries (
youtube-dl.exe), as much as they resemble the "older" official releases by the previous team of maintainers, they still are compiled without nativeCryptosupport, which, though not a deal breaker in itself, necessitates the downloading of a certain type of streams (mainly HLSe) via another dependency,FFmpeg... - I have verified that the latest, as it stands, official release of
2021.12.17has been built with thepycrypto-2.6.1module:
so my hope/wish/suggestion 😉 is any future binary release (based on this PR) by the new management also contain "native" Crypto support...
PS: The abandoned pycrypto module is sufficient for yt-dl purposes, however PyPI doesn't hold any wheels for it; the module contains C++ extensions, thus "installing it" from source requires compilation via a supported C++ compiler, compatible with the CPython version it's about to be used with...
Another option is to use the pycryptodome (not -x) module, for which PyPI wheels are available; but the end binary filesize will increase this way...
Kind regards 😃
Another option is to use the
pycryptodome(not -x) module, for which PyPI wheels are available; but the end binary filesize will increase this way...
As far as I remember, py2exe doesn't work well with it. Hence why yt-dlp_min doesn't use it
While ffmpeg isn't a hard requirement, so much yt-dl[p] functionality depends on it that I can't imagine not having it, except in some niche application of yt-dl.
Also, it solves the problem for the old Pythons that yt-dl is trying to support but which are not supported by pycryptodome. A Unix self-extracting build will use an installed Python and so find any pre-installed optional modules but I don't think that's true in the Windows version?
FWIW, there have been cases in the past where I wanted to fetch geo-fenced media content served over HLSe (AES-128 encrypted), and the only geo-location circumvention means available to me was a SOCKS5 proxy (or a SSH tunnel).
youtube-dl does carry native support for SOCKS proxies via the proxy switch:
--proxy "socks5://proxyhost:proxyport"
so it passes this down to [hlsnative] which, when crypto is available, can successfully fetch HLSe over SOCKS...
FFmpeg, to my knowledge so far, still doesn't support the SOCKS proxy scheme, only HTTP[S]...
PS: It wasn't obvious to me back then, but, in the absence of crypto support, I could have used proxy-chaining (e.g. "The Web"<=> SOCKS5 proxy <=> Privoxy[HTTP] => FFmpeg), but why make things so complicated for the average, layman, user :wink:?
but I don't think that's true in the Windows version?
Nope... The standalone Windows binaries built either by py2exe/pyinstaller come with the python interpreter itself and required extra python modules embedded:
[debug] yt-dlp version 2023.02.14 [8b37c58f8] (win_x86_exe)
[debug] Python 3.7.16 (CPython x86 32bit) - Windows-Vista-6.0.6003-SP2 (OpenSSL 1.1.1t 7 Feb 2023)
[debug] exe versions: ffmpeg n5.2-dev-2245-N-109649-gab8cde6 (setts), ffprobe n5.2-dev-2245-N-109649-gab8cde6, phantomjs 2.1.1, rtmpdump 2.4
[debug] Optional libraries: Cryptodome-3.17, brotli-1.0.9, certifi-2022.12.07, mutagen-1.46.0, sqlite3-2.6.0, websockets-10.4
i.e. no type of "installation" is required at all...
PS: The abandoned
pycryptomodule is sufficient foryt-dlpurposes,
Do you think that's wise? Buffer Overflow in pycrypto .
https://nvd.nist.gov/vuln/detail/CVE-2013-7459
Base Score: 9.8 CRITICAL
The official response is to "Switch to pycryptodome", as a quick fix.
https://github.com/pycrypto/pycrypto/issues/290#issuecomment-533131496 pycryptotodome is incompatible with arm64 as of writing this comment as per below issue.
But it seems like a good quick fix to release something.
A better fix would be to move to the cryptography library, but i am not aware of what this would imply.
"most of the python ecosystem has moved on to using cryptography,
https://github.com/pyca/cryptography"
https://github.com/tencentyun/cos-python-sdk-v5/issues/231
https://github.com/ytdl-org/youtube-dl/pull/30644
I would go with the better fix. But that's just me. Wdyt?
yt-dl can't use cryptography because of missing legacy platform support.
It can use the common API of Pycrypto and Pycryptodome, relying on the user having installed the latter if suitable, or the former, or bundling the former otherwise. Though I'm not sure that Pycrypto will help ARM64 since it expects Py<=3.3. If there is a known security issue in Pycrypto it should be possible to fix it, though someone (maybe a Bellingcat-type open source investigator) who is really concerned about such issues should be running yt-dl in a sandbox.
If there is a known security issue in Pycrypto it should be possible to fix it
Debian still maintains pycrypto and have fixed the CVE-2013-7459 vulnerability (#849495). See also the changelog.
python-crypto (2.6.1-7) unstable; urgency=high [ Salvatore Bonaccorso ] * Throw exception when IV is used with ECB or CTR (CVE-2013-7459) (Closes: #849495)
Someone should be able to build from source, I guess?
Source package available here.
python-crypto_2.6.1.orig.tar.gz
Imho, this is being unnecessarily delayed and side tracked. Getting any release out, even one without deps or even unsigned will cover most people's needs and should be the priority imo. Automation, signing, pycryptodome, standalone unix binaries are all useful to have, but is not essential to a release and can be added later incrementally.
Do you think that's wise? Buffer Overflow in pycrypto .
https://nvd.nist.gov/vuln/detail/CVE-2013-7459
Well, indeed, both pycrypto tags v2.6.1 and v2.7a1 (the second was cut from master on 20131021 but never made it to PyPI) are vulnerable to that CVE...
Be that as it may, in the linked article it's stated that CVE has been patched in the pycrypto master branch:
https://github.com/dlitz/pycrypto/issues/176 => https://github.com/dlitz/pycrypto/commit/8dbe0dc3eea5c689d4f76b37b93fe216cf1f00d4
Additionally, the pycrypto GH repo has seen in 2022 some minor activity, after a long period of inertia, but has now been made a Public archive... https://github.com/dlitz/pycrypto/commit/8dbe0dc3eea5c689d4f76b37b93fe216cf1f00d4 is the very last code snapshot (with CVE-2013-7459 patched 👍 ).
pycrypto suffers from another publicly disclosed CVE, CVE-2018-6594, and that hasn't been patched in pycrypto master, but a fix is public here; doesn't seem to affect the purposes for which the module is used under youtube-dl ...
The "official" Win-binaries produced by the former team of devs seem to contain only a subset of the full pycrypto package, namely ONLY AES-related stuff:
Crypto
-- Cipher
-- __init__.pyo
-- _AES.pyd
-- AES.pyo
-- blockalgo.pyo
-- Util
-- __init__.pyo
-- py3compat.pyo
-- __init__.pyo
which is understandable, because "we" only need AES-128 decrypting support; if this stuff is still vulnerable to the mentioned CVEs, I'm not sure...
however PyPI doesn't hold any wheels for it;
For those interested and willing to trust third-party compiled wheels, GitHub holds:
py3.5 wheels of v2.6.1:
https://github.com/sfbahr/PyCrypto-Wheels/raw/master/pycrypto-2.6.1-cp35-none-win32.whl
https://github.com/hex-in/pycrypto-2.6.1/raw/master/dist/pycrypto-2.6.1-cp35-cp35m-win32.whl
py3.6 wheels of v2.6.1/v2.7a1:
https://github.com/hex-in/pycrypto-2.6.1/raw/master/dist/pycrypto-2.6.1-cp36-cp36m-win32.whl
https://github.com/hex-in/pycrypto/raw/master/dist/pycrypto-2.7a1-cp36-cp36m-win32.whl
py3.7 wheels of v2.6.1:
https://github.com/hex-in/pycrypto-2.6.1/raw/master/dist/pycrypto-2.6.1-cp37-cp37m-win32.whl
py3.8 wheels of v2.6.1:
https://github.com/hex-in/pycrypto-2.6.1/raw/master/dist/pycrypto-2.6.1-cp38-cp38-win32.whl
... and py3.4 wheels for v2.6.1 were available on www.voidspace.org.uk, but now that host has shut down; web archive link below:
https://web.archive.org/web/20200627032153/http://www.voidspace.org.uk/python/pycrypto-2.6.1/pycrypto-2.6.1-cp34-none-win32.whl
For the reasons I described in my previous post, I'd prefer youtube-dl.exe binaries, when other pending issues get resolved, to be Crypto enabled...
@nicolaasjan: You routinely compile your own, py3.4-based, Windows builds, with py2exe; can you please confirm that substituting pycrypto for pycryptodome-v3.9.9 results in non-functional builds (if the compilation succeeds at all, that is...) ?
Thanks in advance 😄 ...
pycryptosuffers from another publicly disclosed CVE, CVE-2018-6594, and that hasn't been patched in pycryptomaster, but a fix is public here; doesn't seem to affect the purposes for which the module is used underyoutube-dl...
Is also fixed by Debian. 🙂
can you please confirm that substituting pycrypto for pycryptodome-v3.9.9 results in non-functional builds (if the compilation succeeds at all, that is...) ?
Pip couldn't find 3.9.9, but I found a wheel on a Chinese site.
Youtube-dl compiles fine, but fails to run:
Traceback (most recent call last):
File "c:\Program Files (x86)\Python3.4.4\lib\site-packages\Crypto\Util\_raw_api.py", line 82, in <module>
raise ImportError("CFFI with optimize=2 fails due to pycparser bug.")
ImportError: CFFI with optimize=2 fails due to pycparser bug.
https://github.com/Legrandin/pycryptodome/issues/228 (?)
I give up...
Pip couldn't find (pycryptodome) 3.9.9
Why not? It's certainly there, in wheel format for py3.4 x86:
https://files.pythonhosted.org/packages/3f/64/c096ccf34ed329c4a17cce65d38aa8b598e5f1e9275f53520cf3e68bfd53/pycryptodome-3.9.9-cp34-cp34m-win32.whl
especially since I specifically requested its creation and the module's author indulged 👍 ...
https://github.com/Legrandin/pycryptodome/issues/228 (?)
Well, this comment 😉 from there describes an identical situation... This appears related, which, in turn, leads to this workaround; does it change things at all?
BTW, and I think it might be an interesting read for the current maintainer, there's a related discussion between the two main former devs (dstftw and remitamine) on the topic of crypto libs and youtube-dl, as comments on 1bba88efc 😉 ...
Why not? It's certainly there, in wheel format for py3.4 x86:
You're right, I made a typing error (pycrytodome). Sorry...
After installing pycparser 2.14, same issue...:
Traceback (most recent call last):
File "c:\Program Files (x86)\Python3.4.4\lib\site-packages\Crypto\Util\_raw_api.py", line 82, in <module>
raise ImportError("CFFI with optimize=2 fails due to pycparser bug.")
ImportError: CFFI with optimize=2 fails due to pycparser bug.
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "__main__.py", line 16, in <module>
File "<frozen importlib._bootstrap>", line 2237, in _find_and_load
File "<frozen importlib._bootstrap>", line 2226, in _find_and_load_unlocked
File "<frozen importlib._bootstrap>", line 1191, in _load_unlocked
File "<frozen importlib._bootstrap>", line 1161, in _load_backward_compatible
File "c:\Program Files (x86)\Python3.4.4\lib\site-packages\zipextimporter.py", line 86, in load_module
return zipimport.zipimporter.load_module(self, fullname)
File "C:\Users\nico\Desktop\youtube-dl_source\youtube_dl\__init__.py", line 15, in <module>
from .options import (
File "<frozen importlib._bootstrap>", line 2237, in _find_and_load
File "<frozen importlib._bootstrap>", line 2226, in _find_and_load_unlocked
File "<frozen importlib._bootstrap>", line 1191, in _load_unlocked
File "<frozen importlib._bootstrap>", line 1161, in _load_backward_compatible
File "c:\Program Files (x86)\Python3.4.4\lib\site-packages\zipextimporter.py", line 86, in load_module
return zipimport.zipimporter.load_module(self, fullname)
File "C:\Users\nico\Desktop\youtube-dl_source\youtube_dl\options.py", line 8, in <module>
from .downloader.external import list_external_downloaders
File "<frozen importlib._bootstrap>", line 2237, in _find_and_load
File "<frozen importlib._bootstrap>", line 2226, in _find_and_load_unlocked
File "<frozen importlib._bootstrap>", line 1191, in _load_unlocked
File "<frozen importlib._bootstrap>", line 1161, in _load_backward_compatible
File "c:\Program Files (x86)\Python3.4.4\lib\site-packages\zipextimporter.py", line 86, in load_module
return zipimport.zipimporter.load_module(self, fullname)
File "C:\Users\nico\Desktop\youtube-dl_source\youtube_dl\downloader\__init__.py", line 18, in <module>
from .hls import HlsFD
File "<frozen importlib._bootstrap>", line 2237, in _find_and_load
File "<frozen importlib._bootstrap>", line 2226, in _find_and_load_unlocked
File "<frozen importlib._bootstrap>", line 1191, in _load_unlocked
File "<frozen importlib._bootstrap>", line 1161, in _load_backward_compatible
File "c:\Program Files (x86)\Python3.4.4\lib\site-packages\zipextimporter.py", line 86, in load_module
return zipimport.zipimporter.load_module(self, fullname)
File "C:\Users\nico\Desktop\youtube-dl_source\youtube_dl\downloader\hls.py", line 6, in <module>
from Crypto.Cipher import AES
File "<frozen importlib._bootstrap>", line 2237, in _find_and_load
File "<frozen importlib._bootstrap>", line 2226, in _find_and_load_unlocked
File "<frozen importlib._bootstrap>", line 1191, in _load_unlocked
File "<frozen importlib._bootstrap>", line 1161, in _load_backward_compatible
File "c:\Program Files (x86)\Python3.4.4\lib\site-packages\zipextimporter.py", line 86, in load_module
return zipimport.zipimporter.load_module(self, fullname)
File "c:\Program Files (x86)\Python3.4.4\lib\site-packages\Crypto\Cipher\__init__.py", line 27, in <module>
from Crypto.Cipher._mode_ecb import _create_ecb_cipher
File "<frozen importlib._bootstrap>", line 2237, in _find_and_load
File "<frozen importlib._bootstrap>", line 2226, in _find_and_load_unlocked
File "<frozen importlib._bootstrap>", line 1191, in _load_unlocked
File "<frozen importlib._bootstrap>", line 1161, in _load_backward_compatible
File "c:\Program Files (x86)\Python3.4.4\lib\site-packages\zipextimporter.py", line 86, in load_module
return zipimport.zipimporter.load_module(self, fullname)
File "c:\Program Files (x86)\Python3.4.4\lib\site-packages\Crypto\Cipher\_mode_ecb.py", line 29, in <module>
from Crypto.Util._raw_api import (load_pycryptodome_raw_lib,
File "<frozen importlib._bootstrap>", line 2237, in _find_and_load
File "<frozen importlib._bootstrap>", line 2226, in _find_and_load_unlocked
File "<frozen importlib._bootstrap>", line 1191, in _load_unlocked
File "<frozen importlib._bootstrap>", line 1161, in _load_backward_compatible
File "c:\Program Files (x86)\Python3.4.4\lib\site-packages\zipextimporter.py", line 86, in load_module
return zipimport.zipimporter.load_module(self, fullname)
File "c:\Program Files (x86)\Python3.4.4\lib\site-packages\Crypto\Util\_raw_api.py", line 204, in <module>
_PyObject_GetBuffer = ctypes.pythonapi.PyObject_GetBuffer
File "c:\Program Files (x86)\Python3.4.4\lib\ctypes\__init__.py", line 364, in __getattr__
func = self.__getitem__(name)
File "c:\Program Files (x86)\Python3.4.4\lib\ctypes\__init__.py", line 369, in __getitem__
func = self._FuncPtr((name_or_ordinal, self))
AttributeError: function 'PyObject_GetBuffer' not found
pip list:
Package Version
-------------- ---------
altgraph 0.17
cachetools 3.1.1
future 0.18.2
mutagen 1.42.0
pefile 2019.4.18
pip 19.1.1
py2exe 0.9.2.2
pycparser 2.14
pycryptodome 3.9.9
PyInstaller 3.5
pywin32-ctypes 0.2.0
setuptools 43.0.0
wheel 0.33.6
@Lesmiscore
Apologies for contacting you here/this way, but 😉 ...
... You appear to be the person behind the unofficial "nightly/daily" youtube-dl automated builds published below:
https://github.com/ytdl-patched/youtube-dl/tags
and produced via GHA; most sadly, the Rebase on Upstream workflow runs have been constantly failing for the past 23 (!) consecutive days:
https://github.com/ytdl-patched/youtube-dl/actions/workflows/rebase-on-upstream.yml
thus even the latest tag offered (2023.04.28.334) is built on stale source code from Mar 14th 2023 😭 ...
As I'm sure you're aware by now, your kind service is currently the source for updated yt-dl EXEs for many Windows users, but in master at least 17 commits "happened" after Mar 14th:
https://github.com/ytdl-org/youtube-dl/compare/6fece0a...HEAD
among which the latest "fix" for Google/YT's throttling issue (which, thankfully, doesn't appear to still be an "issue" with latest yt-player 0c487f05 😜 ) ... I'm not asking for myself, mind you 😉 , I can build my own EXEs locally, but the "audience" you are targeting with your builds probably can't...
Huge thanks :heart: in advance for an eventual resolution/fix...
Best regards 😄
I suppose this arises from the following chain of unpleasantness:
- GH withdraws Ubuntu 18.04 runners
- yt-dl test workflow is changed to skip test sets that needed it
- nightly workflow fails because GH needs a special "password" when modifying workflow.
But now seems to be running, many thanks.
Thanks Lesmiscore for your immediate action on the reported issue 🥇 ... Latest "nightly" tag as of this writing, 2023.04.29.1919, is indeed built on 211cbfd 👍 !
However, to futureproof the service, it's MHO the relevant .yml files should be modified to integrate some "safety" checks, so that when:
a) a Rebase on Upstream workfow fails, the ensuing Build workflow shouldn't be run; it'd be a useless consumption of GHA resources, as the resulting artifacts would be identical, code-wise, to the ones produced in the immediately previous (successful) Build WF run; in addition, the repo owner should be somehow alerted about the Rebase on Upstream WF failure...
b) a Rebase on Upstream workfow succeeds, but the ensuing Build workflow fails (has indeed happened in the past); again, the repo owner should be made aware of such a failed scenario...
As I'm totally clueless myself about GHA, this is purely academic on my part, but I'm confident savvy people monitoring this PR would come up with applicable solutions (hopefully 😜 ) ...
dirkf: The https://github.com/ytdl-patched/youtube-dl repo doesn't (by design) have its issue tracker enabled; as the administrator of this repo, what would be the recommended place/way one should report a future "breakage" of the unofficial "daily" youtube-dl builds service?
... one should report a future "breakage" of the unofficial "daily" youtube-dl builds service?
Today's "daily" youtube-dl build, v2023.05.26.810, is still built on d1c6c5c4d618fa950813c0c71aede34a5ac851e9 from May 11th, whereas master is currently on 2389c7cbd30813435c50848a9b276bcfe2a810db from May 23rd 😞 ; that's because the relevant Rebase on Upstream workflow run failed; excerpt from the corresponding log:
+ git rebase upstream/master
Auto-merging youtube_dl/YoutubeDL.py
CONFLICT (content): Merge conflict in youtube_dl/YoutubeDL.py
Rebasing (1/1)
error: could not apply 3d8d94e6b... Automated Daily Builds
hint: Resolve all conflicts manually, mark them as resolved with
hint: "git add/rm <conflicted_files>", then run "git rebase --continue".
hint: You can instead skip this commit: run "git rebase --skip".
hint: To abort and get back to the state before "git rebase", run "git rebase --abort".
Could not apply 3d8d94e6b... Automated Daily Builds
Error: Process completed with exit code 1.
Regards.