[Enhancement] Add decryption of the sig parameter for bypassing throttling

[unixfox]

Currently we do not support fetching the streams using WEB client because by default the stream is too slow. So we are only relying on the ANDROID client. But recently the ANDROID client has started to be blocked from youtube unless you pass "droidguard" challenge: https://github.com/iv-org/invidious/issues/4584

This issue is to discuss how to implement the decryption of the sig parameter in order to avoid throttling.

An atempt was done back in https://github.com/iv-org/invidious/pull/2222 but was closed due to potential issues around missing isolation when executing untrusted JavaScript code from YouTube.

So there are other solutions:

• Execute a separate node process which is not part of the main invidious process. May be too permissive.
• Execute a separate deno process which is not part of the main invidious process. Good potential because by default deno lock down a lot of permissions.
• Create bindings for using a C library that allows to evaluate isolated javascript code. I do not know any library like this.
• Use the minimal JS interpreter from yt-dlp by executing it using python: https://github.com/dirkf/youtube-dl/blob/master/youtube_dl/jsinterp.py
• Other minimal JS interpreter from YouTube.js in Typescript: https://github.com/LuanRT/Jinter

More ressources:

• Calling python functions from crystal
• crython
• crystal-nodejs

[iBicha]

Worth mentioning https://github.com/LuanRT/Jinter used in https://github.com/LuanRT/YouTube.js

Because it implements ONLY what the decipher function needs (string, arrays, and Math.*) there's no risk with RCE (at least not that I can see). It can't read from disk, read env vars or make network requests, because the functionality doesn't exist.

I find it an interesting approach

Edit: I guess similar to youtube-dl's interpreter

[ChunkyProgrammer]

Another option could be: https://bun.sh/

[syeopite]

For C bindings there is https://github.com/jessedoyle/duktape.cr

[SamantazFox]

* Execute a separate [node](https://nodejs.org/en) process which is not part of the main invidious process. May be too permissive.

* Execute a separate [deno](https://deno.com/) process which is not part of the main invidious process. [Good potential because by default deno lock down a lot of permissions.](https://docs.deno.com/runtime/manual/basics/permissions)

Either of those is something I'd prefer, yep. And the more we can contain it, the better. We don't know what kind of API Youtube will use in the future.

* Create bindings for using a C library that allows to evaluate isolated javascript code. I do not know any library like this.

* Use the minimal JS interpreter from yt-dlp by executing it using python: https://github.com/dirkf/youtube-dl/blob/master/youtube_dl/jsinterp.py

Not an option, imo. This would have to run a lot of times for each request. We also need a solution which is caching the extracted JS for some time, so the function can be reused without making extra calls to Youtube.

[Theta-Dev]

What would be the security issue when running untrusted code in an embedded JS interpreter like Duktape or QuickJS? NodeJS and Deno add a lot of APIs for things like network and file IO, so they have to be locked down when running untrusted code.

None of this is possible with these embedded interpreters and they would be the fastest and most lightweight solution.

[SamantazFox]

What would be the security issue when running untrusted code in an embedded JS interpreter like Duktape or QuickJS? NodeJS and Deno add a lot of APIs for things like network and file IO, so they have to be locked down when running untrusted code.

None of this is possible with these embedded interpreters and they would be the fastest and most lightweight solution.

That's a good question! I have not looked at any of those yet, so I don't know their strengths and weaknesses.

Though, whatever the selected solution, I'd greatly prefer for it to be in a separate process, at least to benefit from running on a separate thread than invidious.

[techmetx11]

I'm making a helper for Invidious in C, It will use QuickJS and do the following:

• Fetch player script and extract functions independently from Invidious
• Communicate with Invidious via a UNIX or TCP socket
• Execute the functions from within the helper (this allows it to be isolated using kernel security measures like AppArmor/SELinux and also not be able to access Invidious' memory)
• (bonus) Store the extracted signature functions in a temporary location and perhaps, the pre-compiled bytecode version of it

In case of a RCE, syscalls can be isolated by the kernel using pledge (OpenBSD) or seccomp (Linux)

[Vizonex]

• Create bindings for using a C library that allows to evaluate isolated javascript code. I do not know any library like this.

Did you try duktape? It allows for compatibility with C and Javascript and there is less drag than using v8. It might be just what you guys were looking for.

[techmetx11]

Check out what I'm making (switched to Rust after some talk with SamantazFox): https://github.com/techmetx11/inv_sig_helper

[unixfox]

• Create bindings for using a C library that allows to evaluate isolated javascript code. I do not know any library like this.

Did you try duktape? It allows for compatibility with C and Javascript and there is less drag than using v8. It might be just what you guys were looking for.

Yes already said in https://github.com/iv-org/invidious/issues/4649#issuecomment-2080081724

And PR #2222 was implemented with duktape.

[techmetx11]

Check out what I'm making (switched to Rust after some talk with SamantazFox): https://github.com/techmetx11/inv_sig_helper

My program is done (along with the protocol documentation in README), all that's needed is writing some interface code in Invidious

[SamantazFox]

My program is done (along with the protocol documentation in README), all that's needed is writing some interface code in Invidious

Thanks a lot for your work! I'll try to check it (and try it) out soon!