search-plugins
search-plugins copied to clipboard
qbittorrent
Discusion: Breaking changes in qBittorrent search-engine
Update: Proposal => qBittorrent.pdf
Hello everybody,
TLDR: I don't have time to maintain the plugins, the search-engine code needs a huge refactor. I'm proposing to remove all search plugins and Python code and make a native integration with https://github.com/Jackett/Jackett
First of all, I think this feature is used by thousands (take a look at Reddit) and qBittorrent is the only bittorrent client with this functionality. So, I don't want to remove it. Furthermore, in the latest releases is supported in the WebUI making it convenient for seedboxes.
The current implementation (code) is a mess, both, the C++ and Python parts. In this issue https://github.com/qbittorrent/search-plugins/issues/84 I list all required changes to make it clean and more maintainable. That are a lot of changes, breaks compatibility, I don't have the skills to make the changes in the C++ code, and more important, there aren't maintainers for the plugins.
Instead of doing the changes, I propose to remove all Python code, including official and unofficial plugins and make native integration with Jackett for the following reasons.
- https://github.com/Jackett/Jackett is a free and open source app that support more than 400 torrent sites. All official and unofficial plugins are supported. They are able to bypass captchas, cloudflare... We can't compete
- It's well maintained. I'm contributor and it provides releases every day. We don't have maintainers to update the plugins frequently and most unofficial plugins are not working. https://github.com/Jackett/Jackett/releases
- They provide a standard REST API based in Torznab. In simple words, it's a XML format well defined (never changes) and It's used by Radarr, Sonarr and many more software. https://github.com/Sonarr/Sonarr/wiki/Implementing-a-Torznab-indexer In case Jackett is discontinued, there are other projects that support Torznab and will be compatible also (see https://github.com/cardigann/cardigann). I'm talking about Jackett but we can make it generic to support all Torznab clients. Will be easy to request and parse the XML in C++ and keep the current UI/WebUI
- We already support Jackett and I know many people using it. They won't have to make big changes. https://github.com/qbittorrent/search-plugins/wiki/How-to-configure-Jackett-plugin
- Runs in the same platforms as qBittorrent does (win, linux, mac)
- We won't have external dependencies, Python.
- We won't have to worry about updating plugins nor resolving issues.
- We won't have legal problems.
- We can focus in doing the best bittorrent client, and let other projects to take care about the search-engines.
I want to know what do you think about and I would like you to try Jackett https://github.com/qbittorrent/search-plugins/wiki/How-to-configure-Jackett-plugin
ping @qbittorrent/demigods @qbittorrent/webdev @qbittorrent/frequent-contributors @qbittorrent/bug-handlers
UPDATE: Implementation draft
NOTE: In the first version I will try to make as few changes as possible.
1. Enable search engine
- Python is not required as external dependency. We can remove all code related to check Python version, download python, copy python code...
2. Install plugins / search configuration
- In a clean install, the list will be empty because we don't provide search engines anymore.
- Remove strikethrough parts in the screenshot
- When you click "add new indexer" or you "edit one" you will see a modal like this. We should add placeholders, tooltips or something to explain each field.
- All fields are mandatory and can be obtained from Jackett UI
3. Search UI
- By now we can let the UI as it is. Jackett provides additional information that we can add in the future.
4. Perform searches with Torznab
- GET <indexer_url>api?apikey=
&q=<search_term> - Eg GET http://127.0.0.1:9117/api/v2.0/indexers/1337x/results/torznab/api?apikey=ygm5k29tomyy11df265gio7iouhqxk4&q=ubuntu
- Eg response XML => https://pastebin.com/kK48jqAc
- Each item is a torrent. The field mapping with the current UI is: ** qBittorrent column => Torznab ** name => title ** size => size (in bytes) ** seeders => seeders ** leechers => peers - seeders ** search engine => torznab indexer name configured by the user (the real name of the indexer isin the response but it can confuse the user) ** description page => comments ** download link => link (can be a maget or a http link)
- Jackett also support categories. You must use this mapping (several categories are supported separated by comma) => https://github.com/qbittorrent/search-plugins/blob/master/nova3/engines/jackett.py#L64
- Eg GET http://127.0.0.1:9117/api/v2.0/indexers/1337x/results/torznab/api?apikey=ygm5k29tomyy11df265gio7iouhqxk4&q=ubuntu&cat=2000,5000
5. Future steps
- Add more columns in search results
- Move search engine to preferences ??
- Allow to select several categories in the same search (eg, tv and movies)
- Add all torznab categories. There are a lot.
- Group search results by infohash. Torznab provides a field with this information
We can focus in doing the best bittorrent client, and let other projects to take care about the search-engines.
👍
Updated the first message with the changes required.
@Chocobo1 Would you like to work in the development? I can test everything carefully. :)
@Chocobo1 Would you like to work in the development? I can test everything carefully. :)
I probably won't have much time to do anything significant.
If you don't mind I can help.
Of course you can. You have all the details in the first message. Just mention me when you have a PR or something. If at any point you are not longer interested in implementing this, please, notice me.
Imho the python integration has been there for a long time and it would not be ideal to drop it completely after all the work it required, just drop python2. Also, not all sites are supported and you can't use the "go to the description page" option.
These would be the sites that we have but jackett does not:
Academic Torrents
ali213.net
anidex.info
CineCalidad
HorribleSubs
Linux Tracker
small-games.info
UnionDHT
But yeah, it sounds like a sensible idea, and there is no reason why these sites above could not be implemented in jackett.
and you can't use the "go to the description page" option.
Jackett does support that
I opened some requests in Jackett to add them. anidex.info, HorribleSubs are in Jackett already.
@hannsen Maybe you can help porting some of them to Jackett, or just helping Jackett somehow. :)
Has anything already been done regarding this? I did upgrade my qBittorrent to 4.2.3 after which I still see all the plugins I had before, but "categories" only list "All categories". Also, sometimes plugins do not find anything anymore.
I agree that Jackett is a powerful plugin, but currently setting it up, running a separate process/server, configuring API keys and it being much slower than "normal" plugins, is not that good. I also see many errors in Jackett log, which implies that not everything works as reliably (of course, I have most of the indexers enabled for testing purposes, which might cause problems too - I will try to limit indexers).
Search functionality has been a killer feature for me so far when choosing a torrent client and if this changes then it needs to be easy/transparent to use (and Jackett is definitely not).
As far as I know no one is working in this. Most of the plugins included in qBittorrent are working (just make sure you have them updated). The categories are not tested well, just use "all categories".
I agree that Jackett is a powerful plugin, but currently setting it up, running a separate process/server, configuring API keys and it being much slower than "normal" plugins, is not that good. I also see many errors in Jackett log, which implies that not everything works as reliably (of course, I have most of the indexers enabled for testing purposes, which might cause problems too - I will try to limit indexers).
I'm Jackett developer too. These issues are real and are caused because Jackett has an API called "agregator" that sends the requests to all trackers at the same time.That API is not designed for use in production and its use is discouraged. It doesn't work well because it will wait until all trackers respond and if one tracker fails it causes problems in the others. In the current implementation of qBittorrent we are using that, but with the proposed changes the idea will be to list all indexers in qBittorrent and make separate requests. That is working really well in Jackett.
Search functionality has been a killer feature for me so far when choosing a torrent client and if this changes then it needs to be easy/transparent to use (and Jackett is definitely not).
I agree but qBittorent developers are not interested in maintaining this (an most of them use jackett anyway). In addition, it is increasingly difficult to maintain trackers because they have security measures such as re-captcha and cloudflare. Jackett has mechanisms to circumvent those protections. PRs are welcome but I don't see any.
@ngosang thank you for your thoughts.
The categories are not tested well, just use "all categories".
My point was that they were working before and stopped working after upgrading. That's why I asked. It doesn't really matter to me because I always have been using "all categories" anyway. Just an observation.
I'm Jackett developer too. These issues are real and are caused because Jackett has an API called "agregator" that sends the requests to all trackers at the same time.That API is not designed for use in production and its use is discouraged. It doesn't work well because it will wait until all trackers respond and if one tracker fails it causes problems in the others. In the current implementation of qBittorrent we are using that, but with the proposed changes the idea will be to list all indexers in qBittorrent and make separate requests. That is working really well in Jackett.
Looking at the Jackett logs that was my initial suspicion too, that it works exactly like you just described in here. Good that you confirmed it, since that explains a lot. When I did enable all (public) indexers in the beginning then I had to start looking at the log and disable some, which were failing thus causing entire search to not return any results. Using your proposed API does make sense and sounds like a good idea.
I agree but qBittorent developers are not interested in maintaining this (an most of them use jackett anyway). In addition, it is increasingly difficult to maintain trackers because they have security measures such as re-captcha and cloudflare. Jackett has mechanisms to circumvent those protections.
I totally understand this. One way of solving this problem would be to create better integration between Jackett and qBittorrent where Jackett is bundled with qBittorrent or can be installed somehow throught qBittorrent and it would start/stop Jackett server and configure API key automatically or whatever so that end-user would not need to know anything about Jackett or how to configure it etc. I'm quite sure that this is also not a small task and I totally understand that PRs are welcome since I'm developing some OSS stuff too.
Just my thoughts how things could be improved.
...> Just my thoughts how things could be improved.
Can you tell me which version of Qbittorent has working search for you? I'm talking about the regular search plugins not Jackett (Which I've never been able to get working) I'm on latest 4.2.5 but search has been broken for a long time now. All I get is a Yellow triangle with exclamation point. All search plugin URL's work in browser but do not work in Qbittorent. I tried both with proxy enabled and disabled. Nothing. I'm on a Mac running Catalina with python 3.7.7 installed.
The search feature is working in all versions of qBittorrent. A small number of users have problems because qBittorent can't find Python (it's an external program and we don't have control). Python 3.7.7 is supported too, just take a look at the qBittorrent log to be sure it's detecting the right Python version. Some users have several version of Python in the same machine and that can cause problems too.
That looks good. Could be a lot of things, even the torrent sites can be blocked in your country...I can't help much more. I recommend you Jackett by the way.
According to the log Python 3.7.7 is being detected. When I execute a search it just fails immediately with a yellow triangle and exclamation point but the log shows nothing regarding my failed searches. Is there another place to look for these failure logs other than the Execution log tab?
No sorry, you are blind. The search functionality is not used by most developers and they don't care about the search log.
The torrent sites are all up and running beautifully when I type each URL manually in my Safari browser. It's Qbittorent search that fails even when the URLS are working perfectly.
Windows and Linux. Definitely could be a Mac thing because we have few users.
Hello everybody.
For me, Jackett plugin in qBittorrent has been working excellent up until today. Jackett, plugin and qBittorrent have been updated to the latest versions. I've tried a search and it seems that the results cannot be parsed back into qBittorrent search window (log below). Any suggestions?
| 2020-05-04 09:40:16 | Error | System.NullReferenceException: Object reference not set to an instance of an object. at Jackett.Common.Models.ResultPage.<ToXml>b__13_0(ReleaseInfo r) in d:\a\1\s\src\Jackett.Common\Models\ResultPage.cs:line 62 at System.Linq.Enumerable.SelectListIterator`2.MoveNext() at System.Xml.Linq.XContainer.AddContentSkipNotify(Object content) at System.Xml.Linq.XContainer.AddContentSkipNotify(Object content) at System.Xml.Linq.XElement..ctor(XName name, Object[] content) at Jackett.Common.Models.ResultPage.ToXml(Uri selfAtom) in d:\a\1\s\src\Jackett.Common\Models\ResultPage.cs:line 38 at Jackett.Server.Controllers.ResultsController.Torznab(TorznabRequest request) in d:\a\1\s\src\Jackett.Server\Controllers\ResultsController.cs:line 420 |
|---|---|---|
| 2020-05-04 09:40:16 | Info | Found 3128 releases from AggregateSearch for: westworld |
Windows and Linux. Definitely could be a Mac thing because we have few users.
Thank you for providing that information. If anybody reading this is on a Mac with Jackett and native search plugins working please let me know version number as I have not had working search for many releases.
I think I solved my issue. It was my socks5 proxy that was blocking search. I did a fresh install and deleted all preferences and search works:-)
Hello,
quick question about Jackett. Well configure , i can get answer from jacket to QBT. no issue.
but seems there is a limit in term of item received. the search plugin from qbt return more item than the jacket one ( by using the same web site ). i can't get more than 60 item per indexer from jackett which ever indexer i use.
any idea ?
I've tried numerous times to update, but am having problems with UnionDHT - It returns results, but clicking on the result does not 'fire off' the torrent to download, I can go to the description page, but it seems no matter what I try I cannot get the torrent to start downloading?
Otherwise I've turned a few others off but have most of them working, ones I turned off were because returning jibberish (or maybe diff language?) or found everything was in a diff. language on the search,
I agree Jackett is neat, but was kinda a hassle to setup, and since has broken and I've just left it..
The only thing I would like to see would be to show new plugins available without having to manually search the plugin directory. Like from the 'update plugins' if it could search the directory and show new ones available that would be great! I know some of the sub only ones may not be able to do that with.
UnionDHT is unofficial. Reach the original author => https://github.com/nindogo/qbtSearchScripts
@ngosang to be honest, the official search engine plugins are perfect with the exception of EZTV imho, which I think doesn't deserve to be included. Let it be unofficial. It's fake, has illegal stuff and not generic. @nindogo if you want to replace it with something else then I recommend TorrentGalaxy which made it into the top 10 this year: https://torrentfreak.com/top-torrent-sites-2021-210103/ but I'm fine with 9 sites.
Installed a newer release of qBittorrent on a Windows machine the other week and Jackett was not available (previous installed version did not have support for Jackett IIRC). Presumably due to the previous install? Ended up going on to github, locating the plugin source and copy/pasting in plugin directory.
Just a heads up, that existing users may not get Jackett support when updating (not that it matters too much since they've still got to manually create a simple config too, unless the feature shown here arrives).
Is that going to be an issue if Jackett becomes the only option? Will the others disappear or not work, while the wiki page instructs that Jackett would be available by installing qBittorrent? May need a warning/notice to users when Jackett isn't installed/detected.
In fact, we have the Search subsystem in an unmaintainable state. The idea of getting rid of it in favor of implementing the support of the Jackett also looks unsuccessful, since firstly there are no people willing to implement it, and secondly it will replace one external dependency with another, which will require taking care of it throughout the entire life cycle (and this in turn will sooner or later lead it to the same state as the current one). We (core developers) are concerned about this problem and are looking for compromise solutions. Recently, we have been actively discussing the idea of redesigning the Search subsystem in such a way as to make it maintenance-free (to get rid of all direct dependencies and all responsibility for downloading, installing and updating plugins, even from the very essence of plugins).
Here is its concept:
- Drop direct support of "nova" and its plugins (including any cares for Python);
- Define a communication protocol (just a set of conventions about the format of command-line parameters and returned data) for interacting with so-called "search providers". The search provider can be any executable that supports the previously mentioned communication protocol.
- Implement a method for registering search providers in the app and the UI for it.
From the above, it can be seen that once implemented and debugged, the new Search subsystem will no longer require almost any maintenance on our part. But at the same time, third parties will be free to provide "search providers" with any degree of convenience, whether it is just an executable file of the provider that you need to download and register manually, or an installer that will install and register the provider and its dependencies (or several providers) for you. Providers can also be distributed through various package managers (apt etc.), which will allow you to update them in the same way as any other application/library. And given that the provider can be any executable file (not even a file, but anything that can act as an executable command), the provider can be written in any programming language that you prefer. In order not to upset existing users, I note that the transition process should not be too burdensome for them. We can take care of converting the installed "nova" plugins to a compatible format and register them automatically (and then let them float freely).
Comments are welcome. But do not forget that we are talking about choosing between an actually unmaintained feature and maintenance-free one.
In order not to upset existing users, I note that the transition process should not be too burdensome for them. We can take care of converting the installed "nova" plugins to a compatible format and register them automatically (and then let them float freely).
That was my main concern. Would not be great to update one day and find the search that was previously fine was no longer available/supported, eg Jackett support.
Implement a method for registering search providers in the app and the UI for it.
Each of these search provider executables could have a command to provide some config info which could be used for registering, perhaps a JSON/TOML file. The app stores a copy of that data and presents it in the UI? Is it possible to fall out of date if the search provider updates? eg if version info was included, or would you query/update this periodically like on restart of the app?
If they were instead running as small REST API servers, they could also receive a standard JSON file adhering to some schema that can be versioned that contains query parameters, or is this going to be split into several option flags for executables? (could alternatively stringify a JSON or similar serialized format?)