YouTube search filters

[omarroth]

Hi! I wanted to provide some information on how YouTube's search filters work, as I noticed currently NewPipe uses hardcoded values. Sorry if this is the wrong place for this.

The &sp= parameter is a Base64'd protocol buffer that contains all the filters. I'd recommend looking here for a working implementation. It's pretty short, but essentially you have (about) two bytes for each filter, one (or more) bytes as a type, and then the other byte as a value, for example:

date month
0x08 0x04

Features (live, HD, ...) have a value of 1:

HD   true
0x20 0x01

HDR       true
0xc8 0x01 0x01

Once you've added all your filters, give it a header (which contains the only mandatory filter, sort, see the code above for details), Base64 encode, URI escape, and you're good to go.

Hopefully this is useful, and I want to thank the NewPipe team for providing their excellent work for others.

[theScrabi]

@omarroth great work. And yes this is the right place :)

[Bluesir9]

Hi, is anyone working on this currently? If not, I would like to pick this up.

[theScrabi]

No atm there is no one working on this.

[Bluesir9]

@theScrabi Cool, will get started then and keep things posted here.

[Bluesir9]

Small update on my end. I have implemented the existing content and sorting filters for YouTube search as per the implementation provided by @omarroth (thanks for that btw!). Have been testing it on my end and its working as per the original requirement. You can find those changes here.

At this point I have 2 primary issues 2 address:

1. Updating the existing test cases: In reference to the testWithContentfilter() test case implemented in YoutubeSearchQHTest.java this existing test case for checking if a sort filter is correctly applied will not work anymore since it was checking the resultant URL against a hardcoded url. The way I see it, a better way to implement the test case would be to run through the html and see if the expected filter was applied on the resultant URL by checking for whether it was highlighted, just like the "Playlist" filter has been highlighted in the below screenshot: I am not sure about the feasibility of the approach but I think using Jsoup this should be possible. Let me know what you guys think about this, or if you have any alternative approaches for the same.

2. Implementing the new content filters: As is obvious by the above provided screenshot, there are a lot of other content filters that YouTube provides out of the box, the implementation for which has been provided by @omarroth. But I am yet to implement them since I haven't quite narrowed down the best approach to do so. I was thinking of merging the concept of "sort filters" and "content filters" into a single "ListResultModifiers" which will contain "Sorters" and "Filters" with all of them having separate implementations for YouTube and Soundcloud. Building these modifiers would be the job of a Builder and the actual filters would be represented by enums. I am going to start coding this to see if this actually works out well but I am open to any suggestions that others may have.

Hopefully, this should be done over the weekend and then we can move to actually implementing it on the app :)

[theScrabi]

Sort filter and content filter are specifiable for each service already.

[theScrabi]

Why is the test against a url not possible anymore?

Also please open a pullrequest, so we can check the code :)

[Bluesir9]

Sort filter and content filter are specifiable for each service already

Yes. But for YouTube, the content filters look like this currently:

    public enum ContentFilter {
        all,
        videos,
        channels,
        playlists,
        movie,
        show
    }

Whereas now, it will become something like this:

public enum ContentFilter {
ContentType("Content",new String[]{"all","videos","channels","playlists","movie","show"}),        Length("Length",new String[]{"short","long"}),
Time("Time",new String[]{"hour","today","week","month","year"}),
...etc...
}

Why is the test against a url not possible anymore?

The existing test just validates if "our" logic to generate the URL for YouTube search is correctly implemented. But if the logic were to change on YouTube's end tomorrow, then that wouldn't be reflected accurately.

Also please open a pullrequest, so we can check the code :)

Will do once I am finished with my changes. 👍

[theScrabi]

The existing test just validates if "our" logic to generate the URL for YouTube search is correctly implemented. But if the logic were to change on YouTube's end tomorrow, then that wouldn't be reflected accurately.

But isent this the purpouse of tests? If it suddenly changes sentry will be the service to notify us.

[NitroNils]

Is this sorting issue the reason why NewPipe fails to sort the 'News' section chronologically? Or should I issue a new issue for this?

[theScrabi]

No, and No. The News section is a different issue, and yes there have been many threads about this problem. We are on it :)

[MD77MD]

what happened to this?

[B0pol]

There was a PR created, see why it has been closed https://github.com/TeamNewPipe/NewPipeExtractor/pull/124#issuecomment-616136364.

[FireMasterK]

This will be handy when writing the .proto files / an implementation: https://github.com/iv-org/invidious/blob/7ddab5b8cda8d673b2f4b5a482185fbf295c456d/src/invidious/search.cr#L273-L369