Add support for indexing/downloading streams

[mikeweyandt]

Add support for indexing/downloading previously streamed content for channel and channel_id sources (under the live tab)

Added toggles to enable indexing and downloading streams under source, defaulting to false to match the current logic today.

Saw others also were running into this under https://github.com/meeb/tubesync/issues/322

[meeb]

Excellent, thanks for the contribution! I'll review this shortly.

[tonytonyistony]

Any updates on this? Definitely waiting on this feature.

[FaySmash]

What's the current state? I'm missing out of ≈90% of the videos from my subscribed channels and the issue will only grow. What needs to be done besides this PR to get this working?

[meeb]

It introduced some issues when testing it, so tweaks and retesting mostly.

[FaySmash]

It introduced some issues when testing it, so tweaks and retesting mostly.

Thanks for the quick response. Glad it's still being worked on. I'm thinking about building an image from the master branch and merging this PR, can I somehow support your testing efforts?

[meeb]

Sure, go for it. Don't merge the update ffmpeg and update s6 patches though they're now out of date. The main patch (https://github.com/meeb/tubesync/pull/453/commits/a40e3e715e55764085ab3dbe8746a57d79a61f0d) should apply fine still.

The issues I encountered were mostly due to a/b testing stuff at YouTube where downloads would fail because things like URLs were different for some channels. This isn't anything to do with the patch, it assume it worked perfectly for @mikeweyandt

I was waiting until YouTube stopped fiddling with whatever they were fiddling with and also I personally don't use this feature so it was more of a pain to test with a real deployment.

[FaySmash]

I finished merging the changes in my fork but I'm unable to test it. I tried for hours now but something about the current Dockerfile is broken. When I build the image with docker build -t tubesync . and start it afterwards, it has a permission issue and won't run the app:

s6-rc: info: service s6rc-oneshot-runner: starting
s6-rc: info: service s6rc-oneshot-runner successfully started
s6-rc: info: service fix-attrs: starting
s6-rc: info: service tubesync-init: starting
s6-rc-oneshot-run: fatal: unable to exec /etc/s6-overlay/s6-rc.d/tubesync-init/run: Permission denied
s6-rc: warning: unable to start service tubesync-init: command exited 126

Any idea what to change about the Dockerfile to make it run again?

[meeb]

For some reason you've reset or otherwise lost execute permissions on the init scripts.

$ chmod +x tubesync/config/root/etc/s6-overlay/s6-rc.d/*/run

or similar.

[FaySmash]

Thanks for the quick response! I see, the perms must be set on the local filesystem before building the image. I thought all permissions would be taken care of in the Dockerfile. I use windows and copied the folder with it, no wonder that the perms got lost. Now it aborts on a different file

s6-rc: info: service s6rc-oneshot-runner: starting
s6-rc: info: service s6rc-oneshot-runner successfully started
s6-rc: info: service fix-attrs: starting
s6-rc: info: service tubesync-init: starting
exec: fatal: unable to exec bash
: No such file or directory
s6-rc: warning: unable to start service tubesync-init: command exited 127

Would it be safe to do something like $ chmod -R +x tubesync/? Or is there any chance to identify the files/folder which need execution permissions?

[meeb]

The permissions are all set correctly in the Dockerfile other than the /config dir which is copied verbatim. You'll really want to preserve the permissions of /config as they are in the repo. Resetting everything to 0755 or something might work, but I've never tried it. Your error now is also related to permissions (something like s6 can't execute the init script wrapper shebang so is throwing a weird error).

[FaySmash]

It works 🥳 I'll test it and report back. I plan to keep my fork up to date with the upstream so I or @mikeweyandt can open a PR anytime when you deem it ready.

[FaySmash]

So far I downloaded a few live stream vods... all without any issues

I also added an option to only download Live VODs and exclude regular videos. I can open a PR if you think it's ready to integrate, or @mikeweyandt if he wants to continue to work on this.

[meeb]

Thanks for the report, it's entirely possible whatever wasn't working with URLs is working now. I'll given this test again and see if it's OK to merge.

[FaySmash]

Thanks for the report, it's entirely possible whatever wasn't working with URLs is working now. I'll given this test again and see if it's OK to merge.

Thanks for the consideration. I opened my own PR, with the updated code

[meeb]

@mikeweyandt I've merged the slightly refactored version of your PR from @FaySmash so I'll close this PR for now. If you want to be on the contributors list feel free to open a PR with a minor edit to the readme mentioning stream support and I'll merge it.

Thanks for your help!