qBittorrent
qBittorrent copied to clipboard
qbittorrent
add a "trust me, it's all at 100%" option to skip the data check for seeding
I recently had to remove and re-add a 120GB torrent that I created and that I'm seeding. When I point the re-loaded torrent to the data folder containing the data, all the data is re-checked. I'd like the ability to skip this re-check and tell the program that all the data is fully intact and at 100% completion.
Doesn't the "skip hash check" checkbox in the add torrent dialog achieve what you want?
No. I don't have an "add torrent" dialog. I may add a hundred torrents in a few minutes, so that is off. There are several things I'd like to see available outside of that dialog that are part of that dialog, such as the ability to download magnet metadata without starting the actual data download, but I don't want drill down into settings twice to turn that box back on and then back off every time I have one particular torrent I'd like to make that exception for. I didn't know that was in that box because I always have it off, so it is effectively dead to me. Edit: might have come across harsh, but not meant that way, sorry
i'm opposed to skipping the data check for seeding in the official qBittorrent releases. If someone wants to do this they should have to recompile the client from source and then the client should announce itself with a radically different peer_id that doesn't claim to be qBittorrent in any way, shape or form.
This feature can be heavily abused to seed fake/poisoned torrents or files that have been intentionally changed and corrupted to send malicious data specially crafted to crash some remote clients or conduct various remote exploit attacks (including, but not limited to, potentially doing a remote code execution in the remote client). This "feature" is also a relatively common tactic used by anti-piracy enforcement companies.
Well-designed clients will drop any bad checksum data without crashing (or executing remote code while crashing), but poisoning torrents with bad checksummed files is enough of a low-effort way to waste the time of thousands of other users.
if qBittorrent implements in the official releases such features that contribute to torrent poisoning, it will then probably become blacklisted / banned from pretty much every tracker out there.
@R-Adrian Well according to @FranciscoPombal it apparently already has the feature (tucked away in a dialog box that some users prefer to disable) and it's not blacklisted everywhere, so...maybe read the thread prior to adding your two cents? Just a thought.
true, it has the checkbox for when adding a torrent file... which can have its contents be 0,001% finished or 99% finished downloading. That one can be useful when resuming a download.
But the client then does another full data check anyway when it finishes downloading and starting to seed. This check is what's at issue: full torrents being seeded with bad data. Partial torrents... meh.
You don't seem to understand how useless your argument is. All I have to do to poison the data in your scenario is have the legit data on hand first, then in-place overwrite the parts I want to taint after the data check finishes. Alternatively, the fastresume can be modified offline to contain the desired state, bypassing the check entirely. Also, the client doesn't do a full data check when it finishes downloading unless you check a box that explicitly forces that to happen.
You are arguing that the program should police the owner of the machine, much like DRM and software activation. This is an unacceptable stance, especially in a piece of open source software, and doubly so when the anti-user provision is hilariously easy to bypass. The only reason I don't do it instead of adding a feature request is because it's merely a matter of (in)convenience for me; I can wait for 121GB to transfer over a 2.5G ethernet connection from a RAID array, but I don't think I should have to when I'm the origin seeder and my copies are literally all of the source data for the torrent.
ok.. for particular cases where there are lots of unique data, it might make sense to have a checkbox for skipping all checks somewhere under tools -> options -> advanced ... preferably behind a warning that says "beware...you risk making a mess of things. Also, there's a leopard in here." Bad torrent data will happen anyway, either in official or custom-compiled builds, but its better to not make it too easy to happen (Murphy and his laws...) And it's not exactly useless.. it raises the knowledge bar a bit, being probably just enough of an inconvenience so that we don't see it everywhere. Not many people will venture to modifying those program configuration files.
Also, about the second data check, on finish... i didn't realize that... i probably enabled that setting long ago and got so used to it that i forgot about it... though i still like its use, especially when qBittorrent crashes randomly. (last such crash that i had was in... november last year... i had to redo all my qBittorrent settings.)
A possible way to mitigate your concerns is to have the program run a data integrity check on a piece before it is sent out. That not only prevents sending out bad data, but also catches any sort of corruption, intentional or otherwise, and brings it to the user's attention. That would actually be a nice thing.
indeed, a delayed just-in-time data check only when a piece is first requested seems useful
just like with manufacturing, instead of checking the entire stock before production, check it during production, when needed. https://en.wikipedia.org/wiki/Just-in-time_manufacturing
this might actually solve the problem of checking large files.
@R-Adrian @jbruchon
Guys, the situation is really not that bad. @an0n666 has done some work recently to prevent "fake" seeding and torrent poisoning (theses fixes made it to 4.3.0): https://github.com/qbittorrent/qBittorrent/pull/13206 and https://github.com/qbittorrent/qBittorrent/pull/12037. From my understanding, as soon as a piece is requested from a file that doesn't exist, the torrent will go into "missing files" state. Also, I'm pretty sure libtorrent doesn't let you seed bad pieces on purpose already (for example, if you create files with the same name but with random/bogus data for seeding). A sufficiently motivated attacker can of course modify the code to allow for this, but at least mainstream libtorrent-based clients have peer-banning heuristics built-in to defend against this.
So I really don't see a problem in adding a way to add multiple torrents with "skip hash check" at once. But please, @an0n666 @arvidn correct me if I'm wrong here.
I personally would want to make sure the data was at 100% before seeding. I know that I have downloaded some stuff, that obviously was seeded as a new torrent, but had only completed to maybe 99.5% from its original torrent. I currently sucks as of 4.3.0.1 with the torrents and rechecking and all that, because that's happened to me a lot, too, so I can see why you're so upset that you had to sit there, waiting for your 120 GB data to be checked.
@as-muncher My concern is when I know I have the data at 100%, particularly because I'm the person who made the torrent. In such a situation, being able to skip the hash checking process makes a lot of sense.
Yeah, I guess that makes sense if you're making a new torrent like that. I suppose qbittorrent might have to do something if there were, for some odd reason, already a torrent with the same name. Maybe it's making a hash of the torrent or something.
Sorry to bump this, but I would really love this feature.
Once a year or so a hick-up (or error on my part) causes the network share to disconnect resulting in it having to "check" over 3TB of torrent files which takes a good half a day or so.
I know for sure the data is correct, it's stored on ZFS which means it will auto-heal. Also the network storage itself never went down, just the mount, so no corruption could've happened to torrents that were at 100% already.
To be sure, this is not the "Skip hash check" at the torrent adding screen, the torrents have already been in my client, finished downloading sometimes as much as 2 years ago and just got marked as requires recheck because the storage got disconnected.
@smiba Have you tried perhaps BiglyBT, or maybe a different torrent client? I know with some of these torrent clients, they're supposed to just check the data that was in question at the time of a crash or what not, not the entire thing.
Switching is not really an option, I specifically choose qbittorrent so I could use qbittorrent-nox on linux and access it from the webbrowser. I used Transmission previously but that was just a miserable experience, qbittorrent does exactly what I want and has a ton of options, but this is the only real issue I'm struggling with.
Libtorrent is a lot less sensitive (and less trigger happy) to require a full check than it would as a few years ago. I would be interested in knowing what may have triggered the recheck. Is it enough to start up and all files are missing?
Iirc, one use case that was fixed a while back is when an entire mount point goes away. But these scenarios don't have great test coverage.
Do yo have logs from the time when the resume data was considered invalid? I assume by now the resume data has just been wiped and is just straight up missing.
What happened this time is that I rebooted the VM, qbittorrent autostarted but the file share didn't mount correctly (not mounted at all / missing)
This resulted in all files being reported as "Missing files", I shutdown qbittorrent again, fixed the mount, started qbittorrent again and this caused all the files to require rechecking.
The download directory / resume file is locally to the system (I move finished torrents manually through the "move" option), so the failure of this network share shouldn't cause issues with the resume file. The only thing I can imagine is that qbittorrent didn't shutdown safely the first time, this combined with the missing files might have pulled it over the edge? But fairly sure I did shutdown qbittorrent safely before the reboot.
The log looks like this (qbittorrent.log):
(N) 2022-05-24T16:22:15 - qBittorrent v4.2.5 started
(N) 2022-05-24T16:22:15 - Using config directory: /home/smiba/.config/qBittorrent/
(I) 2022-05-24T16:22:15 - Trying to listen on: 0.0.0.0:59232,[::]:59232
(N) 2022-05-24T16:22:16 - Peer ID: -qB4250-
(N) 2022-05-24T16:22:16 - HTTP User-Agent is 'qBittorrent/4.2.5'
(I) 2022-05-24T16:22:16 - DHT support [OFF]
(I) 2022-05-24T16:22:16 - Local Peer Discovery support [ON]
(I) 2022-05-24T16:22:16 - PeX support [ON]
(I) 2022-05-24T16:22:16 - Anonymous mode [OFF]
(I) 2022-05-24T16:22:16 - Encryption support [FORCED]
(I) 2022-05-24T16:22:16 - UPnP / NAT-PMP support [ON]
(I) 2022-05-24T16:22:16 - IP geolocation database loaded. Type: DBIP-Country-Lite. Build time: Fri Oct 1 00:06:22 2021.
(N) 2022-05-24T16:22:16 - Using built-in Web UI.
(W) 2022-05-24T16:22:16 - Couldn't load Web UI translation for selected locale (C).
(N) 2022-05-24T16:22:16 - Web UI: Now listening on IP: *, port: 8080
(I) 2022-05-24T16:22:16 - Successfully listening on IP: 127.0.0.1, port: TCP/59232
(I) 2022-05-24T16:22:16 - Successfully listening on IP: 127.0.0.1, port: UDP/59232
(I) 2022-05-24T16:22:16 - Successfully listening on IP: 192.168.2.153, port: TCP/59232
(I) 2022-05-24T16:22:16 - Successfully listening on IP: 192.168.2.153, port: UDP/59232
(I) 2022-05-24T16:22:16 - Successfully listening on IP: ::1, port: TCP/59232
(I) 2022-05-24T16:22:16 - Successfully listening on IP: ::1, port: UDP/59232
(I) 2022-05-24T16:22:16 - UPnP/NAT-PMP: Port mapping successful, message: successfully mapped port using UPnP. external port: TCP/59232
(I) 2022-05-24T16:22:16 - UPnP/NAT-PMP: Port mapping successful, message: successfully mapped port using UPnP. external port: UDP/59232
(N) 2022-05-24T16:22:16 - 'Torrent1' restored.
(N) 2022-05-24T16:22:16 - 'Torrent2' restored.
(N) 2022-05-24T16:22:16 - 'Torrent3' restored.
(N) 2022-05-24T16:22:16 - 'Torrent4' restored.
(N) 2022-05-24T16:22:16 - 'Torrent5' restored.
(N) 2022-05-24T16:22:16 - 'Torrent6' restored.
[...]
EDIT: I do realize I'm not running the latest version of qbittorrent, so maybe this has been fixed? I should probably update it at some point
@smiba I think that's what @arvidn was saying when he mentioned the latest version of qbittorrent, that qbittorrent used to be more prone to checking. You're using 4.2.5 when the latest now is 4.4.3. How about upgrading to 4.4.3 and then seeing if you keep getting this problem?
@smiba I think that's what @arvidn was saying when he mentioned the latest version of qbittorrent, that qbittorrent used to be more prone to checking. You're using 4.2.5 when the latest now is 4.4.3. How about upgrading to 4.4.3 and then seeing if you keep getting this problem?
Yeah only realized a moment ago it's been a good while since I've updated! I'll update soon and report back if it happens again, could take months though. Usually it just works without issues, as long as I don't mess around with my network or equipment too much :)
EDIT: 2024-04: I've not see this issue since updating, so this definitely seems to be resolved for me
For what it's worth, when my network drops with the 4.3.x series, I can close and re-open qBT after reconnecting the network shares and it picks up again without re-checking. This doesn't help with the original issue (removing and re-adding the torrent) but it helps with the "missing files" due to a missing device.
The download directory / resume file is locally to the system (I move finished torrents manually through the "move" option), so the failure of this network share shouldn't cause issues with the resume file.
My concern would be that once you get into this state, it's recorded in the resume files when you shut down. i.e. it's possible that it's just that one event of the the files missing that's interesting.
Once the resume files say the files need rechecking, it's not surprising that they need checking.
Network drives are much more error-prone, but an IO error does not necessarely mean data corruption. The moment a network hiccup happens, all read give an IO error and all torrents need to be rechecked. This is completely unfeasible. Just add the option already and let the user know this is very dangerous to do, chunks are already checked as they are seeded. (biglybt takes a lot of ram but i'm afraid i will switch to it)
@ElDavoo to someone who hasn't experienced this issue, your message is a bit cryptic. I think "the option" you're talking about is called "seed mode", is it not? but I wouldn't consider it dangerous
I was talking about "disable hash check" option. of course the client will check the hash "on-the-go", e.g. before sending it to the peer. Transmission does that, not sure about others
I'm actually really amazed by qbittorrent 4.4.3.1 on Windows 10 x64. I have my torrents on an external device, and my system has frozen, and so I initiated a manual blue screen. When I started up my computer again, started up qbittorrent, my torrents started downloading again right away without any rechecking, saving me a ton of time from all the rechecking. Great job to the developers!! Is this your doing @arvidn ? :)
@arvidn One thing I've noticed, though, and I haven't checked to see if someone has already made an issue about this, is that when I initiate a crash, and then reboot the computer and start up qbittorrent again, qbittorrent will start working right away without checking, but I'm sure glad that I have it set to add the .!qB extension to incomplete files and to recheck the torrent upon completion, because the torrent usually needs to be fixed. Maybe that data between the resume save state and the crash is not being checked, but at least that amount should be so that I don't have to recheck the torrent, and then download the missing pieces, and then check all over again. Maybe there could be some background checking. And with people not choosing to have the .!qB extension and not rechecking upon completion, I bet there are a lot of people who think they have completely downloaded the torrent, but it's actually faulty or incomplete.
when I initiate a crash, and then reboot the computer and start up qbittorrent again, qbittorrent will start working right away without checking
I assume by "initiate a crash" you mean cutting power to the computer. Not a graceful shutdown.
The intention is to not re-check files willy-nilly. If there is a correct-looking resume data, it will be trusted. There are some cheap checks to ensure all files the resume data says should exist actually do.
In the scenario where you cut power though, it's possible that data sent to the operating system never got to the disk before losing power, but the most recent resume data may still say we have it. In this scenario your downloaded file will be corrupt and should be re-checked.
Many years ago, libtorrent would save file timestamps in the resume data, and if they didn't match the files on disk it would run a full check. That ended up with a lot of false positives for checking. It was primarily needed when using "compact" mode storage, where pieces are moved as new ones are downloaded, to ensure the file on disk is contiguous (not sparse). That mode was removed from libtorrent a long time ago, which also weakened the justification for the timestamp checks.
The current resume data, with data written to sparse files to the correct location, makes the assumption that more data may have been downloaded since the last resume data was written, but as long as resume data is saved frequently enough, it's fine to re-download that.
Does qbt save resume data regularly in the background? If so, how often?
@arvidn https://docs.microsoft.com/en-us/windows-hardware/drivers/debugger/forcing-a-system-crash-from-the-keyboard set to save every 5 minutes. I wish there was option like that for linux distributions, in case the system hangs, as it sometimes does in Windows. It would be nice if qbittorrent background checked generally the area of data that may be at fault, while at the same time downloading more, because at the end, when the torrent has downloaded, those seeders who have the needed data may not have it for quite some time, if ever. But I suppose it could also be my responsibility to check the torrent manually just in case, anyways on my own time.
With the "Skip hash check" option now available for "Watched Folder"s under "Options > Downloads", should this issue be considered fixed? At the very least, it seems to handily cover the remove-then-readd-torrent workflow mentioned in OP's comment.
