release tarballs miss required sources

[umlaeute]

Operating System Info

Other

Other OS

Debian

OBS Studio Version

28.0.1

OBS Studio Version (Other)

No response

OBS Studio Log URL

https://github.com/obsproject/obs-studio/releases/tag/28.0.1

OBS Studio Crash Log URL

No response

Expected Behavior

downloading the release tarballs from https://github.com/obsproject/obs-studio/releases/tag/28.0.1 should include the entire source required to build obs-studio

Current Behavior

the release tarballs do not contain submodules, e.g. plugins/obs-websocket is missing.

this means that obs-studio cannot be built from the source tarballs.

Steps to Reproduce

1. download https://github.com/obsproject/obs-studio/archive/refs/tags/28.0.1.tar.gz and extract it
2. install all the 3rd party libraries required
3. attempt to build the build fails with:

CMake Error at cmake/Modules/ObsHelpers.cmake:446 (message):
  OBS: obs-websocket submodule not available.
Call Stack (most recent call first):
  plugins/CMakeLists.txt:86 (obs_status)

Anything else we should know?

This is for creating the Debian-official packages of obs-studio (which will also be part of Ubuntu and other derivatives). On Debian, we use release tarballs for creating our packages.

Since github does not include any gitmodules in their generated "Source Downloads" (which is just a wrapper around git archive), crucial parts of the source code are missing. The typical way to solve this (on the upstream (that is your) side) is to manually create a release tarball that contains all the necessary sources.

[tytan652]

Maybe just download tarballs with the right commit hashes for each sub-modules.

[kkartaltepe]

There is no way to disable github's broken tarballs or we would have. You can assume OBS does not distribute tarballs and it is incumbent on packagers to use git.

[kkartaltepe]

We have added error messages to the missing submodules so that package managers know that they are disabling functionality if they refuse to use git. But you can continue to disable the functionality as it was silently disabled if you previously used github tarballs.

[umlaeute]

@tytan652 wrone:

Maybe just download tarballs with the right commit hashes for each sub-modules.

ah well, we have infrastructure to download tarballs for us, but unfortunately that doesn't work well if we have to assemble upstream sources from multiple tarballs.

@kkartaltepe wrote

But you can continue to disable the functionality as it was silently disabled if you previously used github tarballs.

as of 4ce3a729fd63f8aa8f32ef715347bc18e6c5b9f2, it doesn't seem like you can disable obs-websockets, it just fails if it is missing: https://github.com/obsproject/obs-studio/blob/615728fa3be71e928842285ce3ba8f7133d1ad22/plugins/CMakeLists.txt#L88

so it would be great if there was some (hopefully automated) job that is just run whenever you create a new "Release" that creates a release-asset containing all the sources (including submodules). using github actions, this might be a one-time coding exercise (and the CI does the rest)

[kkartaltepe]

as of https://github.com/obsproject/obs-studio/commit/4ce3a729fd63f8aa8f32ef715347bc18e6c5b9f2, it doesn't seem like you can disable obs-websockets, it just fails if it is missing:

Yes you are right, i thought we had pushed this outside the submodule as described but i was wrong.

[umlaeute]

here's a script that creates a tar-file with all the submodules included:

prefix=obs-studio-HEAD
rm -f release.tar
git archive --format tar -o release.tar  --prefix=$prefix/ HEAD
PREFIX=${prefix} TOPDIR=$(pwd) git submodule --quiet foreach --recursive \
   'git archive --format tar -o submodule.tar --prefix=$PREFIX/$displaypath/ $sha1; tar -Af $TOPDIR/release.tar submodule.tar; rm -f submodule.tar'
gzip release.tar

[PatTheMav]

@umlaeute The best place for this would be to extend our Linux workflow and the release workflow to include a proper tarball in addition to the deb package generated - we welcome a PR for this.

[umlaeute]

I should be able to provide a PR that adds the workflow.

however, before I did into that: I haven't found a workflow (for obs-studio) yet that actually creates release assets. the linux_build job in the BUILD workflow seems to create some build artifacts https://github.com/obsproject/obs-studio/blob/743117f08060d3d79f92740f46a60cbeed2e2699/.github/workflows/main.yml#L270-L275, (but as is usual with my relation to github, I'm not even able to locate the artifacts; EDIT: obviously they are not always created, so that's why I didn't find them...)

so my first question is: what is your release workflow?

I could (one or more of the following):

• add a job to the BUILD workflow that just creates the tarballs and makes them available via the artifacts (but I think this is actually of limited use, except for testing)
• extend an existing release workflow (but both Steam Upload and Flatpack) seem to be wrong...
• create an extra source.yml workflow that acts on:release and builds the source tarball and attaches it to the release.

your pick :-)

[PatTheMav]

I would just amend the package script to also generate a tarball and run the upload build artefact step twice, once for the deb file and once for the tarball - you can just output two different variables to be used then.

So just duplicate this step:

https://github.com/obsproject/obs-studio/blob/743117f08060d3d79f92740f46a60cbeed2e2699/.github/workflows/main.yml#L277

And use a different variable for env.FILE_NAME.

[PatTheMav]

Will be fixed by #9787.