Feature Request: qBittorrent Hardlink Categories

[undaunt]

Hello,

Use Case 1 File A is downloaded into a qBittorrent category called "Downloads - ISO" which, with qBittorrent automatic management, points to /downloads/complete/iso/. Currently, the way I understand injection with daemon mode to work, the cross-seeding torrent from another tracker would also be inserted (with default tag cross-seed) into the same category and folder location. This category, or a generic Radarr or Sonarr category, may only be a placeholder category for new incoming files and not match my category organization for longer term storage.

If I move one of the copies of the torrent in the interface, the second copy is now pointing to missing data and I need to manually move that torrent's category as well. This inhibits my ability to leverage qBittorrent's automatic torrent management mode to manipulate where data lives on the server.

Use Case 2 Preexisting torrents that are scanned using scan mode instead of daemon mode will do the same behavior as use case 1. If I am storing files in a per tracker subfolder system, eg: /downloads/seed/BHD and I'm trying to cross seed to NBL in /downloads/seed/NBL right now both torrents will appear, seeding, in the BHD category and folder location. I'm then left with the use case 1 dilemma of missing data due to automatic management.

Use Case 3 Torrents are matched in daemon mode immediately upon completion. For users who have 1 category for Sonarr downloads (or similar) and a subsequent category for imported Sonarr downloads, if the cross-seeded torrents are injected into the first category, they may cause additional looping imports. Eg: A season of a show could be imported after cross-seed injection, thus it is imported, overwritten and imported by cross-seed 1, then imported and overwritten again by cross-seed 2.

In either case, a manual action must be taken to move one of the torrent's categories, hardlink the content of the torrent, then re-scan the data in the secondary category. It would be ideal to be able to pass through a custom qBittorrent category (or even better, a category per Prowlarr tracker optionally) that would auto hardlink the data instead of leaving it in place with the original data.

This would be an optional flag as not all users require hardlinks.

What are your thoughts on this? Thanks very much!

[mmgoodnow]

Thanks for the very detailed write up!

Hard links, symlinks, and partial data are an area of complexity that, when I wrote cross-seed initially, I struck a hard line stance against. That said, cross-seed has become a bit popular, and I will consider anything if enough people want it.

This feature proposal involves not only complexity in matching, but also complexity of configuration, which is made even more complicated by the fact that it would only work with qBittorrent. Please propose what the configuration would look like, both in the config.js format, and in CLI format (the hard one).

[undaunt]

Here is my first take on the CLI format. I'll circle around on the config.js format as soon as I can. Work has been extremely busy. If you have any thoughts on this though, I can update/amend before taking a crack at config.js. I've tried to add extra detail to each command flag to suggest possible functionality, defaults, etc.

Using my config as an example, here is my current compose state:

  qb-tv:
    image: lscr.io/linuxserver/qbittorrent:14.3.9
    container_name: qb-tv
    restart: unless-stopped
    network_mode: service:wireguard
    security_opt:
      - no-new-privileges:true
    depends_on:
      - wireguard
    volumes:
      - $DOCKERDIR/qb-tv:/config
      - $DLDIR:/downloads
    environment:
      - TZ
      - PUID
      - PGID
      - WEBUI_PORT=$WG_QB_PORT2 # Listening port 58477

  cs-tv:
    image: mmgoodnow/cross-seed
    container_name: cs-tv
    restart: unless-stopped
    networks:
      - $TRAEFIK_NETWORK
    security_opt:
      - no-new-privileges:true
    user: $PUID:$PGID
    #ports:
    #  - $CS_TV_PORT:2468
    volumes:
      - $DOCKERDIR/cross-seed-tv:/config
      - $DLDIR/torrents/tv:/torrents:ro
      - $DLDIR/watched/tv:/output
    #command: search -i /torrents -s /output -e -a -A inject --qbittorrent-url http://wireguard:$WG_QB_PORT2 -T http://prowlarr:$PROWLARR_PORT/25/api?apikey=$TRACKER1_KEY http://prowlarr:$PROWLARR_PORT/20/api?apikey=$TRACKER2_KEY http://prowlarr:$PROWLARR_PORT/19/api?apikey=$TRACKER3_KEY http://prowlarr:$PROWLARR_PORT/24/api?apikey=$TRACKER4_KEY
    command: daemon -i /torrents -s /output -e -a -A inject --qbittorrent-url http://wireguard:$WG_QB_PORT2 -T http://prowlarr:$PROWLARR_PORT/25/api?apikey=$TRACKER1_KEY http://prowlarr:$PROWLARR_PORT/20/api?apikey=$TRACKER2_KEY http://prowlarr:$PROWLARR_PORT/19/api?apikey=$TRACKER3_KEY http://prowlarr:$PROWLARR_PORT/24/api?apikey=$TRACKER4_KEY

I have the daemon command for regular usage of the container, however on first run prior to using the daemon I'd do a 1 time load of existing .torrent files using the search command. In either case, the subsequent flags would be the same. I'm anticipating any future hard/soft link flags, category flags, etc. would also mirror across commands for my use case.

Let's assume I'm grabbing most of my TV files from tracker 1, but in case I sometimes grab one from tracker 2 or 3, I want to compare all snatched files against all trackers for maximum cross-seeding. Perhaps we could define something like this:

-ln   Enables hardlinks for detected cross-seed torrents during injection. (Requires a directory or per-torznab directory linkdir via -ld or -ldn. Default: false. If enabled with -A of "save", does nothing.)
-lns   Enables symlinks for detected cross-seed torrents during injection. (Requires a directory or per-torznab directory linkdir via -ld or -ldn. Default: false. If enabled with -A of "save", does nothing.)

-ld, --linkdir    Specifies the folder path within qBittorrent that all cross-seeded torrents should be linked into for all trackers. (Fallback for blank Torznab specific folders. Requires hard or symlinks. If -ln/-lns not specified in conjunction with -ld, defaults to -ln. Default: false)

Using the above logic, I could improve current workflows by at least keeping all cross-seed torrents in a separate location, allowing me to still move data between various other qB categories via Automatic Torrent Management (or manually, I guess). Perhaps I intake net new files for Sonarr to /downloads/complete/tv/, then move them to /downloads/seed/tracker1/ or /downloads/seed/tracker2/ after the fact. By keeping the cross-seeded torrents separate from the gate, I'm already solving for all three use cases. Example new command below for my above container. My host OS is passing through a mounted NAS into the qB container at /downloads, so the path to the cross-seed hardlink location should also be relative to how qB sees it, aka in this case /downloads, not /mnt/nas/downloads.

I don't know if this would necessitate an additional volume mount being available to the cross-seed container itself, but I assume not since currently that isn't required for injection, but just thought I'd raise the concern.

command: daemon -i /torrents -s /output -e -a -A inject--qbittorrent-url http://wireguard:$WG_QB_PORT2 -ln -ld /downloads/seed/cross-seed/ -T http://prowlarr:$PROWLARR_PORT/25/api?apikey=$TRACKER1_KEY http://prowlarr:$PROWLARR_PORT/20/api?apikey=$TRACKER2_KEY http://prowlarr:$PROWLARR_PORT/19/api?apikey=$TRACKER3_KEY http://prowlarr:$PROWLARR_PORT/24/api?apikey=$TRACKER4_KEY

--

For a really fancy use case, getting to specify the cross-seeding location per tracker would allow me to actually leverage my entire structure of /downloads/seed/tracker1 through /downloads/seed/tracker4 natively, thus allowing me to leverage qBittorrent's automatic torrent management file controls with not only the original torrent, but all cross-seeded matches. The prior use case would not allow that if a file exists in more than 2 trackers, because then more than 1 cross-seeded torrent would be pointing at the same data in /downloads/seed/cross-seed/. For this, we'll add a few more commands:

-ldn, --linkdirn

Specifies the folder path within qBittorrent per provided Torznab url, eg -ld1 for the first, -ld2 for the second, and so on. (One directory per Torznab URL. Requires all Torznab URLs provided to have a dedicated folder path, or requires -ld directory for fallback. Requires hard or symlinks. If -ln/-lns not specified in conjunction with -ldn, defaults to -ln. Default: false)

That is really a mouthful for the config and I think I could clean it up more, but a command example probably provides easier context. Here are three:

command: daemon -i /torrents -s /output -e -a -A inject--qbittorrent-url http://wireguard:$WG_QB_PORT2 -ln -ld /downloads/seed/cross-seed/ -T http://prowlarr:$PROWLARR_PORT/25/api?apikey=$TRACKER1_KEY http://prowlarr:$PROWLARR_PORT/20/api?apikey=$TRACKER2_KEY http://prowlarr:$PROWLARR_PORT/19/api?apikey=$TRACKER3_KEY http://prowlarr:$PROWLARR_PORT/24/api?apikey=$TRACKER4_KEY -ld1 /downloads/seed/tracker1/ -ld2 /downloads/seed/tracker2/

command: daemon -i /torrents -s /output -e -a -A inject--qbittorrent-url http://wireguard:$WG_QB_PORT2 -ln -T http://prowlarr:$PROWLARR_PORT/25/api?apikey=$TRACKER1_KEY http://prowlarr:$PROWLARR_PORT/20/api?apikey=$TRACKER2_KEY http://prowlarr:$PROWLARR_PORT/19/api?apikey=$TRACKER3_KEY http://prowlarr:$PROWLARR_PORT/24/api?apikey=$TRACKER4_KEY -ld1 /downloads/seed/tracker1/ -ld2 /downloads/seed/tracker2/ -ld3 /downloads/seed/tracker3/ -ld4 /downloads/seed/tracker4/

command: daemon -i /torrents -s /output -e -a -A inject--qbittorrent-url http://wireguard:$WG_QB_PORT2 -ln -T http://prowlarr:$PROWLARR_PORT/25/api?apikey=$TRACKER1_KEY http://prowlarr:$PROWLARR_PORT/20/api?apikey=$TRACKER2_KEY http://prowlarr:$PROWLARR_PORT/19/api?apikey=$TRACKER3_KEY http://prowlarr:$PROWLARR_PORT/24/api?apikey=$TRACKER4_KEY -ld1 /downloads/seed/tracker1/ -ld2 /downloads/seed/tracker2/ -ld3 /downloads/seed/tracker3/

In the above examples here is what I believe should happen:

1st - trackers 1 and 2's found cross-seed files should land in their respective /downloads/seed/tracker1/ and /downloads/seed/tracker2/ folders. Trackers 3 and 4 files' should land in /downloads/seed/cross-seed/ 2nd - All four trackers files should land in their respective /downloads/seed/trackerx/ folders. 3rd - This would throw an error as the user did not specify an -ld directory but also did not provide a -ldn location for each tracker (missing the 4th tracker).

Looking at it, I don't know if a numerically increasing variable flag such as -ld1, -ld2, etc. is a good solution, or if simply one that does not change but is executed in the order received would be better, such as how we're just using multiple torznab URLs against just one -T call. If that's the case, perhaps -ldn is better renamed -Tld or --torznablinkdir.

[mmgoodnow]

Thank you for your dedication on this.

Looking at it, I don't know if a numerically increasing variable flag such as -ld1, -ld2, etc. is a good solution, or if simply one that does not change but is executed in the order received would be better, such as how we're just using multiple torznab URLs against just one -T call. If that's the case, perhaps -ldn is better renamed -Tld or --torznablinkdir.

I wonder if we could just use a sentinel value - something like

-T        torznab1 torznab2 torznab3
--linkdir    _        _      dir3

that would allow "fallback to default" behavior.

So far, I like this proposal. One concern I do have is that this might make the following harder:

• linking to separate directories based on a criterion other than tracker

because it sort of ties the link dir semantically to the tracker. I'm not sure there's really a good way to express pipelines of that complexity in a command line invocation, so maybe a config overhaul would be in order if we ever got there. (Or a webui but that would be a pretty big undertaking)

[undaunt]

@mmgoodnow This is still on my radar. Family just going through a protracted bout of COVID.

[undaunt]

I like the idea of allowing the fallback to default, so unless specified for a given torznab provider, they get the default directory - resolving my scenario 3 above.

Regarding your concern above, is there any criteria that would be relevant to using for different directories other than per tracker if not simply one default directory? I can't think of a use case where that wouldn't be the case, but I'm just one person.

[mmgoodnow]

does #257 (solution detailed in #254 ) solve your use case?

[undaunt]

This solves part of the issue in that use case 3 will no longer trigger causing a reimport for each tracker's copy of the content, which on its own is awesome. It however then presents a "new" problem:

I'll now have one copy of the data in folder A, tagged as both 'DL Movies' category and 'DL Movies.cross-seed' category, but I cannot leverage moving the content to a subsequent category that has a different folder path such as 'Seed - Tracker A' or 'Seed - Tracker B'. The second copy in 'DL Movies.cross-seed' would then be pointing to missing data - which is essentially the second paragraph of use case 1.

Edit: I should add that preventing the multiple import loop per tracker of use case 3 is huge, and will allow me to run the tool in daemon mode now instead of only in scan mode retroactively. The remaining use cases would be awesome to solve, but also for now I can manually triage by unselecting each torrent's automatic management flag, changing the category, doing a ln via terminal, etc. It's just a lot of steps per file.

[mmgoodnow]

Regarding your concern above, is there any criteria that would be relevant to using for different directories other than per tracker if not simply one default directory? I can't think of a use case where that wouldn't be the case, but I'm just one person.

per media type comes to mind (packs, movies, etc)

[SweetMNM]

Hi there! I will give my small suggestion on the matter... All of this "criterias" and deciding save paths based on tracker or based on season pack or not sounds way beyond the scope of the cross-seed project, mainly because there are other projects that already do it very well, and are already being used widely by people to manage and sort their torrents into categories.

Take a look at tqm for example. It allows you to filter based on many different properties, all in a clear and very nice config format. The only problem is that if tqm detects a torrent's files are being shared with another torrent (cross seeding) it ignores them and doesn't relabel them.... Relabeling refers to recategorizing, and thus getting qbit to move the files.

Here is a scenario: I have a tqm rule to put all torrents from the tracker T1 into a t1-permaseed category after 2 days. Same follows for T2 and t2-permaseed because I like everything neatly organized. Now a movie was grabbed from T1, and cross-seed crossseeded it from T2 as well. At this point they are both in the radarr category or uncategorized perhaps, doesn't really matter. What does matter is that tqm will now ignore them, and they will stay uncategorized or in radarr category for ever. Not only is it bad for organization, but more importantly it's bad for situations where you change categories that have save paths across different drives (like having a torrent added to a SSD initially for speed and then moved to an HDD for long term seeding).

I have looked at patching the issue from tqm side of things but came to the conclusion there is not a viable way to do so. There is no way to hardlink from tqm and then tell qbit to change a torrent's location, since that causes qbit to move the file, and it will be missing from the previous location for other torrents. Only sort of viable solution is to have tqm:

1. hardlink to a new path
2. remove the torrent from qbit without deleting files
3. add it again in the new path

but that will discard all of the stats that torrent had, which is not ideal. That why a solution needs to be implemented in cross-seed, when adding the torrents initially.

The solution I had in mind, that prevents cross-seed from having to implement from scratch what tqm does, and keep cross-seed as simple as possible is to provide a single --hard-links flag or config option, that when enabled does as follows (this solution and entire discussion is only viable when using qbit injection):

When ever a cross-seed is found, instead of adding it in the same path and to the same category, add it to a specific cross-seed category with autoTMM off. Specify the location for the cross-seeded torrent when adding to qbit to a specific configured cross-seed folder, with an increasing number added to it somehow (could be numbered subfolders inside the main cross-seed save folder). The increasing numbers will allow to cross-seed from more than one tracker, and still have the torrent files associated with each tracker hardlinked to a different path. At this point, since every cross-seeded torrent is hardlinked from it's own path, organization to neat folders based on whatever criterias you wish can be picked up by tqm, or any other tool that allows categorization based on conditions.

Sorry if this write up turned into a long incoherent mess. I tried my best to keep it organized and convey my suggestion as simply as possible, hopefully it wasn't too bad to bear through.

If this solution sounds nice to you, I also don't mind taking a shot at implementing it.

[mmgoodnow]

I definitely appreciate someone else other than me saying that this feature area is out of scope for cross-seed, I totally agree! 😅 @SweetMNM are you the tqm dev? What would you think about cross-seed calling out to TQM directly to perform the injection instead of injecting the torrent into qbit and then tqm finding out about it later?

[SweetMNM]

Hey @mmgoodnow, thanks for the awesome cross-seed project! I am a very happy user :) I am not the creator of tqm, but I am familiar with the codebase and am currently making updates and improvements in my fork that I will gladly merge back to the original tqm once the original author has time to review.

Your suggestion is interesting, but I still have some reasons to think it's better to just throw the torrents in a folder and let tqm find and organize them for several reasons.

1. configuration complexity

It will require users configuring cross-seed with the exact tqm command that takes care of the injection, specifying all of the cli flags that they need for tqm, and all the other possible communication overhead that it involves (tqm doesn't run as a daemon, rather usually configured to run as a cron), which complicates things for the user and leaves room for more user error.

2. versatility

not everyone uses tqm. By taking care of hardlinking and injecting torrents in a unique dir on the cross-seed side of things, we allow any other tool that users might prefer to handle things. Some people might use qbit_manage to tag torrents based on tracker, or qbittools tagging features. Even without said tools I assume it might be useful to some setups to have all cross-seeds seperated in a single cross-seed folder and category for organization purposes.

3. work to implement

the way tqm (and those other tools) work, they will by default (or with very minimal changes) interact just fine to having torrents simply appear uncategorized, that's their entire purpose infact. The original suggestion is optimistic on both sides and requires (I think) the least work from both the development side and user side of things to make work. Sometimes tqm might not have rules matched for a torrent (maybe a brand new tracker) what then? Tqm will still have to throw it in a specific cross-seed dir and add numbering to make sure paths are unique anyways, until the user configures rules for the new tracker and then tqm will relabel on the next run.

If there are upsides that I missed to having tqm do the injection I will gladly figure it out with you and we can perhaps find a solution that works with that, but I still think the original suggestion is better for simplicity and versatility sake.

Like I said once you give me a green light and we work out the details I will gladly pick up things on the cross-seed side and PR it :)

[mmgoodnow]

would action: "save" be enough in this case then? I just feel like this is something that would be better handled downstream entirely - if we're going to implement this feature in cross-seed for qbittorrent, we should also implement it for all the other clients, etc

[SweetMNM]

I see.... I believe the only issue with action: "save" is that there is no way to know where the target cross-seeded torrent is in. Would perhaps saving the torrent, and then saving a small info file next to it with the original torrent path be enough? Then any tool can pick it up whenever? Perhaps add a small, postSaveCommand that can pass the torrent path as an argument to speed things up and run tqm (or any other tool) instantly, without having to wait for the next cron run?