youtube-dl New Release?

Checklist

[x] I'm asking a question
[x] I've looked through the README and FAQ for similar questions
[x] I've searched the bugtracker for similar questions including closed ones

Question

Preface: There have been a number of fixes done since the last release (2021-12-17) {over a years worth} The error messages refer using youtube-dl -U (which is already in place). All the errors I am experiencing are already closed issues, but no no release so youtube-dl -U won't fix it. Issue #31583

Question: Can either the instructions be updated (and the error message refering to youtube-dl -U be dropped) to "install" via a github pull from Master

or

Can a new release be cut?

Thanks

Feb 18 '23 21:02 aarondvail

A new release is overdue but depends on solving certain cross-platform issues in the CI build process. Meanwhile for the main platforms, see #31530 for update hints.

Feb 19 '23 00:02 dirkf

What are those issues that need sorting out? Is there an issue/PR tracking this? Creating a PR could bring in people that are willing to help with the process (like myself)

Feb 19 '23 12:02 Grub4K

#30644, but it should probably be refreshed now.

This is very far down my stack today, but as I recall there were various possible blockers:

change to pytest for Py3.10+ (mostly achieved in my test branch with a pytest build that should run on the supported platforms)
especially Jython was tricky, if anyone cares (but we have found a Py2.6 user today)
various workflow upgrades because of GH dependencies
the usual maze of twisty build system dependencies, all alike.

Obviously I need to dive back into this especially in view of https://github.com/yt-dlp/yt-dlp/pull/6220, which I would also like to acquire, if only for more regular versioned releases.

Feb 19 '23 14:02 dirkf

But isn't the auto-update installer already there, depending on the how the yt-dl instance was installed? Once a build has been deployed to the release page and to PyPi, it can be pulled by almost all installation types.

The problem being addressed is really to get the small number of builds in the release tested on all the different platforms.

Feb 21 '23 00:02 dirkf

My understanding is that PyPi and other options would only update on release, and not necessarily auto update at that (i.e. without running pip install again separately).

My suggestion would auto update on either release or master branch updates before each run of the program. That way you don't need to rush to cut release, at the same time users could stay up to date if they wish.

One thing I didn't really point out is that youtube_dl itself would also be initiated through this script, so it would run the checks to update and then pass the argument to youtube_dl and run.

Basically it's a simple hack to make updating against the master branch painless and easy to the user.

Feb 21 '23 02:02 J7a4s0m5ine

Will #30644 keep tracking this and should I post my remarks on there?

I think maybe instead of perfecting the release process it might make sense to release something so as to get updates to the end user. If releases were not automated, whats problematic about building like that, then passing those executables to the user to be tried on various platforms? The related release can be marked as pre-release if need be, but that way there at least are executables which will work for most people while we can get reports for the edge cases and squash those specifically.

I would also advise to look into the release.yml/release-nightly.yml workflow in https://github.com/yt-dlp/yt-dlp/pull/6220. It is crafted so that it creates a release in the yt-dlp style regardless of build output, adding all of the build artifacts from build.yml, it's checksums and the _update_spec. A build.yml satisfying simplistic conditions can be created at first, which would build some more general artifacts, gathering info about possible incompatibilities.

These are the repository configuration options of the release and nighly action:

vars.PUSH_VERSION_COMMIT: Update master branch with a push
vars.BUILD_NIGHTLY: Build a nightly release on every commit

# Both need to be set to push build to archive repo as release
vars.ARCHIVE_REPO: repo like `yt-dlp/yt-dlp-builds`
secrets.ARCHIVE_REPO_TOKEN: token with contents:write

# Push to PyPI and brew if each is set
secrets.PYPI_TOKEN: Token for PyPI
secrets.BREW_TOKEN: Token for brew

update-version.py would have to be backported since the format changed and it's output is now a bit different. Maybe the naming could also be changed to update_version.py?

make_changelog.py, while configurable, is most likely not configurable enough to be able to support the ydl changelog. The author assumption would have to be corrected (Authored by: ) or the changelog would have to be crafted and read manually for now (would be simple to do a shim that just reads from ChangeLog for now).

change to pytest for Py3.10+ (mostly achieved in my test branch with a pytest build that should run on the supported platforms)

Are the test adjustments needed for the release for now? Is that preparation to be testing if the builds would work as expected? If so, I'd think that can be moved back for now?

especially Jython was tricky, if anyone cares (but we have found a Py2.6 user today)

Do we need to supply youtube-dl builds that embed JPython? Otherwise I can't really see the direct relation of that and a release.

various workflow upgrades because of GH dependencies

This should be handled by 6220. I can help with the remaining blockers there.

the usual maze of twisty build system dependencies, all alike.

Yes, understandable. If we encounter anything in particular we might need to push that back until somebody with experience on that system or with those dependencies can chime in. I think those issues will become apparent once we create/try creating the actual build.yml.

The steps that I would propose for now are:

[ ] Create the release.yml and adjust to the format we would want
[ ] Fix/port update_version.py and make_changelog.py
[ ] Create a simplified build.yml handling the most common targets.
[ ] Publish a pre-release with the created executables (people will be _{^{sort of}} happy here)
[ ] Fix problems, improve and extend build
[ ] Employ extended testing and the sort

Feel free to contact me via E-Mail if you would like to move to a different communication channel for this. Alternatively let me know of anything else I can do.

Feb 21 '23 03:02 Grub4K

This fork works and has updated releases: https://github.com/yt-dlp/yt-dlp

Mar 13 '23 21:03 iBobik

Just to show that a release might appear quite soon, and not in the far future.

From https://github.com/ytdl-org/youtube-dl/issues/31530#issuecomment-1472127323

... I was wonder why so big deal to create new release. I came across to #30568 and #31535 and I hope I understand. @dirkf decided to support old unsupported versions of python and probably this is reason why is not possible to use continuous integration to build and release new version.

No, although it's true that the GH workflow ecosystem forces you into an upgrade race as various dependencies are moved to new versions, much as in the rest of the s/w world. Actually, for yt-dl itself, Py2.7 is fine and dandy for the most part: the #31530 fix did reveal some 2.6 hold-outs but also surprisingly many verbose logs showing 2.7 still in use.

The previous build and release system, not well documented, depended on some local setup that had been passed down through the former maintainers. Lured by the gleaming promise of online builds, I completely failed to reproduce the necessary local setup. That brings us to last month.

Now, I find that GH is about to junk ("deprecated") versions of Linux that support the Pythons needed for yt-dl's CI testing. So we'll have to rework that completely using our own runners, etc. This is too hard for now and the existing tests will have to do: boo to Python >= 3.10 (though I don't expect significant issues there). Therefore the build and release workflow doesn't depend on the test workflow.

I took the yt-dlp scheme as a pattern, as suggested, rather than building directly on the excellent work in #30644.

Thus build.yml is a simplified script merging the one from #30644 with the yt-dlp-derived version, removing the bits that are handled elsewhere in the yt-dlp-derived workflows. For now, only the existing targets will be built, ie: POSIX (Linux/macOS/BSD/etc) self-extracting executable, matching .tar.gz, Windows self-extracting executable with built-in Python 3.4.4. As the unofficial nightly builds seem unproblematic I don't expect to see a lot of build issues.

Users of new macOSes that have no Python will still have to install a Python and arrange for it to be found by the shebang of the POSIX build.

Currently a tricky part is make_changelog.py. yt-dl wants a plain-text changelog, but the Changelog class in the script can be specialised and the punch-line becomes:

print((Changelog if args.markdown else PlainChangelog)(commits.groups(), args.repo))

Less simple, the 200 commits since the last release aren't as disciplined as the yt-dlp script expected. I think the only practical design for grouping these commits automatically is to use git commands to identify what each commit changed and then map back from there. For instance:

git log --format= --dirstat=files commit gives a count of changed files by directory that can be sorted to pick the significant directory, eg if the top directory in the commit is youtube_dl/extractors, the group should be CommitGroup.EXTRACTOR
git log --format= --numstat commit gives a count of lines changed in each file that can be sorted to give the most changed file (should be relative to the line count of the file) and the unsuffixed basename of that file is a useful input to classification.

Given this, the solution is to take some output of an as-good-as-can-be-expected hack on the existing script and modify the CI process to accept some manually tweaked changelog delta that will be prepended with its version ID to the ChangeLog file and uploaded in the release notes.

Obviously any other suggestions would be happily considered.

Mar 16 '23 23:03 dirkf

Good to hear there is a path forward!

Now, I find that GH is about to junk ("deprecated") versions of Linux

Yes, I believe the 18.04 runner is being removed in April if I remember correctly. I assume you're aware of this action to setup Python? https://github.com/actions/setup-python ... Example: https://github.com/LizardByte/Themerr-plex/blob/master/.github/workflows/CI.yml#L36-L49

Ubuntu 20.04 and 22.04 also still have python 2.7 that can be manually installed. Here's a build in an Ubuntu 22.04 dockerfile that uses Python 2.7 for reference: https://github.com/LizardByte/PlexyGlass/blob/nightly/Dockerfile#L4-L24 ... Debian/Ubuntu has too many packages that still rely on Python 2.x to completely remove it.

I didn't see a mention of publishing to PyPi in your comment. Are you still going to publish there?

Mar 17 '23 00:03 ReenigneArcher

Is there a new PR, some branch or similar that I can take a look at and implement missing factors to help out?

I think the only practical design for grouping these commits automatically is to use git commands to identify what each commit changed and then map back from there.

This is why I suggested shimming it for now, as manual mapping will be required anyways. Regarding future commits, it might make sense to adjust the commit message to be closer to dlp format OR (I thought about it before but scrapped it since its easier to just rely on message only) search for the files using file or text search if the format should be kept the same ([YouTube] => extractor/youtube.py => EXTRACTOR). Let me know of any changes to that script that I could do.

I didn't see a mention of publishing to PyPi in your comment. Are you still going to publish there?

Assuming that this is building on the release workflow of dlp the PyPI publish is integrated in those steps

It might make sense to diff with dlp build workflow as some new changes were coming. A relevant branch can be found here (though many changes were directly related to nightly and --update-to and are therefore irrelevant)

Mar 17 '23 00:03 Grub4K

Homebrew project leader here 👋🏻. I think even getting a tagged prerelease here with failing CI will allow Homebrew to update to a version that will work for downloading from YouTube and stop you getting so many issues filed by our users.

Mar 18 '23 22:03 MikeMcQuaid

Thanks for your interest!

Almost any commit to master is meant to be reliable and its CI test will have succeeded, having been exercised on another branch first.

If you would like a special tag applied to some commit, that's easy. If you're able to take an unofficial nightly build (from a downstream), which has the advantage of reporting a different version and showing a disclaimer, that's even easier.

Mar 19 '23 00:03 dirkf

Is there a new PR, some branch or similar that I can take a look at and implement missing factors to help out?

Upcoming.

This is why I suggested shimming it for now, as manual mapping will be required anyways. ...

I suppose that a manually adjusted changelog delta could be offered to the workflow as a var, even if it's ~15kB?

I didn't see a mention of publishing to PyPi in your comment. Are you still going to publish there?

Assuming that this is building on the release workflow of dlp the PyPI publish is integrated in those steps

For now I commented out the package management interfaces. The existing PyPi update process would have been manual AFAIK with any account details held by former maintainers; presumably we can regenerate them if they can't be recalled.

It might make sense to diff with dlp build workflow as some new changes were coming. A relevant branch can be found here (though many changes were directly related to nightly and --update-to and are therefore irrelevant)

The workflow files

	new file:   .github/workflows/build.yml
	new file:   .github/workflows/publish.yml
	new file:   .github/workflows/release.yml
	new file:   devscripts/make_changelog.py
	modified:   devscripts/make_issue_template.py
	new file:   devscripts/update_formulae.py
	new file:   devscripts/update_version.py
	new file:   devscripts/utils.py

all started out from the yt-dlp equivalents in a meld session. So yes.

Mar 19 '23 00:03 dirkf

Thanks for your interest!

Thanks for your great software that I personally use!

If you would like a special tag applied to some commit, that's easy.

Yes, our hope would be any sort of tag applied to a release that would show up in https://github.com/ytdl-org/youtube-dl/releases that we could update to 🙇🏻

Mar 19 '23 04:03 MikeMcQuaid

Which release asset is used to create the Homebrew package?

Mar 19 '23 20:03 dirkf

The pypi URL: https://github.com/Homebrew/homebrew-core/blob/a44ce7e234fddf054ba589cc682d73d9d8499771/Formula/youtube-dl.rb#L6

Mar 19 '23 22:03 MikeMcQuaid

Which release asset is used to create the Homebrew package?

This is also reflected in the publish_pypi_homebrew job from the yt-dlp release workflow. It publishes to PyPI, then updates the respective Ruby script in the homebrew-taps repo using its wheels.

A PyPI publish only requires the PYPI_TOKEN secret on the repo to be updated and then should function automatically (this can be tested on test.pypi.org, like we did using yt-dlp-test). Certain changes might be required for adjusting readme and similar meta files.

Mar 19 '23 22:03 Grub4K

In that case the answer is either linking something like this

url "https://github.com/ytdl-patched/youtube-dl/releases/download/2023.03.19.43044/youtube-dl-2023.03.19.43044.tar.gz"
sha256 "56888be194b5c1f2f54b527a31d93fb34719d3f0f8265f7bc7d2d98c870d1858"

in the formula, or using --head.

Mar 19 '23 23:03 dirkf

@dirkf We're unlikely to change our upstream to a repo titled "Unofficial daily builds for youtube-dl". Our policy (not for youtube-dl but for all projects) is to require tagged (or otherwise versioned) official upstream releases. --head is not a good solution for our users; it's slower, untested by us and more error-prone so we don't support any failures that result from using it.

Mar 20 '23 14:03 MikeMcQuaid

@dirkf hey, do you have approx possible terms for the new release?

Mar 25 '23 12:03 DmytroUsenko

I didn't read everything here, but how about having just a patch release 2021.12.17.2b or so from a ad-hoc branch with minimal changes? For me it's kind of a blocker that the tool regularly fails with youtube links (#31530). Patch releases are quite standard procedure for other projects. And note that even distribution packages like ubuntu's depend on these official releases

Mar 27 '23 21:03 daald

Just to show that a release might appear quite soon, and not in the far future.

Any update on getting a release published. It's been one month since this comment, and 2 months since the latest release is unusable (https://github.com/ytdl-org/youtube-dl/issues/31530).

Like many others it's not feasible for me to install from git when using this library as a dependency in other projects.

Almost any commit to master is meant to be reliable

If this is the case, perhaps you should automate GitHub and PyPi releases on push events.

... Anything you need help with specifically to get automated releases working? Or this is just not a priority for this project?

Apr 16 '23 16:04 ReenigneArcher

Just to weigh in on the release cycle debate... I'm a devops engineer and so I do a lot of stuff with pipelines and releases. At work, it's important to pin my dependencies (usually libraries). But with an app like youtube-dl I'm just a user, I'm not in work mode. I install it via brew and I want it to "just work" when updates are available via package manager. I don't want to be backporting fixes or building my own version. So a patch release would be great, just to fix the current problem with YouTube. Thanks for your efforts

Apr 17 '23 09:04 djjudas21

Just a heads-up: Google plans to start deleting inactive accounts later this year, and that includes YouTube content. https://www.blog.google/technology/safety-security/updating-our-inactive-account-policies/ So, saving historic videos from YouTube is now more important than ever. If we can get a new release pushed soon that works with YouTube, that would be great. Thanks for all your work, everyone!

May 16 '23 20:05 TinaRussell

Just a heads-up: Google plans to start deleting inactive accounts later this year, and that includes YouTube content. https://www.blog.google/technology/safety-security/updating-our-inactive-account-policies/ So, saving historic videos from YouTube is now more important than ever. If we can get a new release pushed soon that works with YouTube, that would be great. Thanks for all your work, everyone!

I agree with archiving historic videos. Just wanted to point out that this information is incorrect, though. In the announcement linked, it includes this sentence:

Additionally, we do not have plans to delete accounts with YouTube videos at this time.

So YouTube content is still safe.

May 19 '23 09:05 adeeb1

It wasn't incorrect at the time, that was a later addition, as noted at the end

We've updated this post to clarify how this announcement impacts creator accounts and accounts with YouTube videos.

and as visible archived

In fact they explictly said they would delete youtube

To reduce this risk, we are updating our inactivity policy for Google Accounts to 2 years across our products. Starting later this year, if a Google Account has not been used or signed into for at least 2 years, we may delete the account and its contents – including content within Google Workspace (Gmail, Docs, Drive, Meet, Calendar), YouTube and Google Photos.

May 19 '23 16:05 DianaNites

I have been using youtube-dl for a long time and now it's just broken. Yeah, I can use it via Git but it's complicated and I'd like to just be able to use it. I suggest people switch to https://github.com/unrud/video-downloader Why not just release a new version so it's at least not broken for everybody? Thanks

Jun 10 '23 07:06 steckerhalter

While the build and release workflow for the main repo is under construction, we have this, thanks @Lesmiscore: https://github.com/ytdl-org/ytdl-nightly/releases.

Jul 03 '23 03:07 dirkf

btw here is the "patch release" bug report for Ubuntu: https://bugs.launchpad.net/ubuntu/+source/youtube-dl/+bug/2008009

Jul 03 '23 14:07 daald

we have this, thanks @Lesmiscore:

https://github.com/ytdl-org/ytdl-nightly/releases.

As an old-time an avid yt-dl plain user, it's not a hyperbole to say I'm elated over this new development 🎉 ...

As a yt-dl tester though, and without any trace of entitlement/ungratefulness 😉 , I have to point out the results of my nit-picking:

Like the ytdl-patched/youtube-dl "daily" releases, this one is also built without native crypto support 😞 ; I have detailed this "problem" in the relevant PR #30644, specifically in comments 1, 2, 3; since the building environment is py3.4-based, it may simply be a case of

python -m pip install -U setuptools
python -m pip install -U pycrypto

however, in that case, VS2010 (32-bit?) also has to be pre-installed, so that GHA can build the binary parts of pycrypto (.pyd files) under Windows; or simply deploy:

python -m pip install "https://web.archive.org/web/20200627032153/http://www.voidspace.org.uk/python/pycrypto-2.6.1/pycrypto-2.6.1-cp34-none-win32.whl"

Currently, it's not easily evident the master branch code snapshot the ytdl-nightly binary is built from (unlike the nightly "downstream" builds, e.g. [debug] yt-dlp version [email protected] [4dc4d8473] (win_x86_exe)); for the latest nightly version, 2023.07.04.114514, I had to visually compare https://github.com/ytdl-org/youtube-dl/commits/master to https://github.com/ytdl-org/ytdl-nightly/commits/master to establish that nightly v2023.07.04.114514 reflects master branch code snapshot fa7f0effbe4e14fcf70e1dc4496371c9862b64b9 ...
Despite the rebrand to "nightly", --verbose mode still speaks of "unofficial daily" builds:

youtube-dl -v

[debug] System config: []
[debug] User config: []
[debug] Custom config: []
[debug] Command-line args: ['-v']
[debug] Encodings: locale cp1253, fs mbcs, out cp737, pref cp1253
[debug] youtube-dl version 2023.07.04.810 (single file build)
** This build is unofficial daily builds, provided for ease of use.
** Please do not ask for any support.
[debug] Python 3.4.4 (CPython x86 32bit) - Windows-Vista-6.0.6003-SP2 - OpenSSL 1.0.2d 9 Jul 2015
[debug] exe versions: none
[debug] Proxy map: {}
Usage: youtube-dl [OPTIONS] URL [URL...]

youtube-dl: error: You must provide at least one URL.
Type youtube-dl --help to see a list of all options.

The ytdl-nightly builds are still on the same update channel as the fork they "came" from ; STR: a. Download previous ytdl-nightly binary release, i.e. v2023.07.03.19419:

https://github.com/ytdl-org/ytdl-nightly/releases/download/2023.07.03.19419/youtube-dl.exe

b. Issue youtube-dl -vU:

debug] System config: []
[debug] User config: []
[debug] Custom config: []
[debug] Command-line args: ['-vU']
[debug] Encodings: locale cp1253, fs mbcs, out cp737, pref cp1253
[debug] youtube-dl version 2023.07.03.19419 (single file build)
** This build is unofficial daily builds, provided for ease of use.
** Please do not ask for any support.
[debug] Python 3.4.4 (CPython x86 32bit) - Windows-Vista-6.0.6003-SP2 - OpenSSL 1.0.2d 9 Jul 2015
[debug] exe versions: none
[debug] Proxy map: {}
Latest version: 2023.07.04.810, Current version: 2023.07.03.19419
Current Build Hash f6dfe7cff1211ada6e63b982c0dc429e5d3fc22b54cb7744087879ea80743fbe
Updating to version 2023.07.04.810 ...
Updated youtube-dl to version 2023.07.04.810

However, v2023.07.04.810 is the latest release on "ytdl-patched/youtube-dl":

https://github.com/ytdl-patched/youtube-dl/releases/tag/2023.07.04.810

NOT the latest "ytdl-org/ytdl-nightly" one, which is v2023.07.04.114514:

https://github.com/ytdl-org/ytdl-nightly/releases/tag/2023.07.04.114514

Incidentally, the "updated" build v2023.07.04.810 contains stale yt-dl code, thus the "update" is actually a "downgrade" 😉 ...

I propose that the ytdl-nightly update mechanism follow what is currently being implemented in the yt-dlp-nightly update channel: Build workflow runs should be triggered only when the actual master code gets updated; in the ytdl-patched/youtube-dl configuration, we had consecutive daily builds irrespective of yt-dl's master branch "activity": if no commit was pushed for, say, a period of a month, ytdl-patched/youtube-dl would have produced 30 (!) "daily" builds, all built on identical source code... Perhaps pukkandan 😄 could be kind enough to lend his GHA expertise to that goal..

In conclusion, please take my suggestions above as incentive to improve the ytdl-nightly "service" itself, not as unreasonable demands raised by an entitled non-coder 😜 ; far from that...

Kindest regards 😄 !

Jul 04 '23 18:07 Vangelis66

youtube-dl youtube-dl copied to clipboard

New Release?

Checklist

Question

youtube-dl
youtube-dl copied to clipboard