cppcryptfs
cppcryptfs copied to clipboard
bailey27
Googel Drive / Files to be replaced are being downloaded and re-upped
Lately, when a file already exists and windows explorer asks if I want to replace it and I chose yes, explorer halts/freezes up for a while and then tells me that the file cannot be replaced because it's in use. I checked in the Google Drive client and it looks like the file is actually being downloaded (and since it's a large one windows explorer just pauses until it's completely downloaded) and then something happens and the replace fails.
I'm absolutely sure, it didnt use to do that. If a replace action was done, the file in question was just overwritten on the remote location and no download was necessary. I wouldnt even know why a download of the encrypted remote file would be necessary for it to be replaced with an identical one.....
My workaround for now is to tell windows explorer to not replace those files and skipping them. Then delete them on the remote drive and move stuff up again. Way faster but not really cool.
Google has finally allowed non-corporate/academic users to use Google Drive Stream!
Unfortunately, I was not able to reproduce this problem. Google mounted my drive stream on G:. I created a cppcryptfs filesystem under the "My Drive" folder. Then copied a 25MB pdf there. Then I renamed another large PDF (20MB) with the same name as the first one in a normal windows directory and copied over the one in my cppcryptfs that was using Google Drive stream for backing storage. It didn't hang.
Maybe Google is caching the first file locally in my case (because I just put it there), and maybe in your case the file is only offline when you try to copy over it?
I'll keep trying, but so far no luck.
Awww 20mb “large” file ;-) I’m talking 50-100GB files. And true, yours is most likely cached locally. I disabled caching/offline avail since that defeats the purpose for me to offload to the cloud. I’ll keep my eye on it too, I dont think it does it all the time. I need to figure out what triggers a download/reup cycle instead of just a purge and replace. I’ll update ccpcryptfs too, maybe it’ll solve the issue.
I tried again with a 2.59GB file, and I still can't reproduce it.
What I see happen is that when I paste the file with the same name, and select replace, I see it copy locally at 308 Mbytes/sec. While this is happening, there's no network activity.
As soon as the local copy finishes, I see my network go from zero to maxed out at 50Mbits/sec upload.
If I look here
%USERPROFILE%\AppData\Local\Google\DriveFS
I can see the 2.59GB file under there. It is definitely caching it locally.
I've got offline unchecked in the settings. Is there something else I need to change?
Hmm. I think that may have changed from way back. Nevermind. I upload massive amounts of data and the local cache gets rotated quickly. Just delete it manually after your upload so you’re sure it’s not using a cached instance. Log out of google drive client, delete the cache (mine is on a large fast ssd and not in userprofile but I guess I changed that manually), log in again and try to replace that file with an identical one (name/size). Whenever I replace a file with an ident one, the client downloads the file from drive and then fails. If I log out of drive and log in again I can then replace that file and no longer get that pesky “file in use” error. But it shouldnt download and it shouldnt have issues replacing the file. Also, if you’re playing around with drive anyway, every now and then check the google drive mount in explorer (the encrypted view). My encrypted folder structure starts 3 levels below My Drive (Google Drive W:/My Drive/Folder1/Folder2/EncryptedStructure). Since a few months (and google drive releases) ago, there are encrypted files and folders in Folder1 (2 levels above encryption level) I dont know where they’re coming from and if it’s the drive client dumping them there. I did a few checks over the months and they are duplicates of files and folders from under the proper Folder2 encryption structure. They dont get deleted automatically and I need to clean those manually. Since checking what those files actually are is annoyingly cumbersome, I just assume they are failed upload tries of the drive client but I cant be sure and it doesnt really instill a trusty feeling in the process. Google Drive support is useless in this specific case since they just assume it’s not their clients fault but the fault of the encryption tool. But I dont think so. cppcryptfs doesnt even know Folder1 exists, I mount a folder below Folder2 in cppcryptfs. It must be something the drive client has issues with.
If you try replacing a file that already exists on a cppcryptfs volume in Drive File Stream and it isn't cached, it'll download the entire thing first (even if the file is 100 GB). I've found the easiest solution is to scan for changed files based on date/size, delete the changed files in Drive File Stream, then copy the files over from the left. Then the files won't be download every time you try to replace something.
thanks @null-python at least I’m not imagining things lol.
@bailey27 is that a Google Drive or a cpp thing?
In my simple mind there is no reason whatsoever to download a file that’s being replaced with a new identical one....
In my simple mind there is no reason whatsoever to download a file that’s being replaced with a new identical one....
Could it be trying to download the file so it can calculate and upload the diff ? (Offloading the diff calculation to client?) Or it might be downloading random (or beginning) bytes of the file to determine: whether the file is edited, or the replacement is a completely different file.
I think this all goes away with the deterministic file name thing. now I need a migration path :-)