"Missing files"

[ctejada10]

Since maybe about a week ago I've been having unmounting issues every time I issued an upload command, which I do every night. I figured, no biggie, I'll double check if the drive is mounted before doing anything next morning. However, two days ago, everything went haywire. I was getting corrupt database issues, a --force sync wouldn't do anything, Bad address errors on some of my files when trying to list them, a total mess. And, as random as it came, it fixed. I didn't do anything.

Nonetheless, this hiccup left me with about 200GB in lost files. No idea how or why. But that's ok. I got all my stuff back and went to upload again, but that's when I got an error saying: Folder "super_awesome_stuff" already exists. Please sync., repeated about a million times with different folder names. Those are the files that went "missing" yesterday!

Since syncing doesn't fix the issue, I try checking my Amazon Drive to my surprise to find all the "missing" files safe and sound in there. Any pointers on how to proceed?

[kax0]

Same behavior on my side, i freaked out when i was looking for some data but it wasn't available on my computes but was on amazon.

I'm getting 27 out of 43 folders. I tried to determine if filename, size or tree structure was the cause but it seems to be completly random.

[ctejada10]

In my experience it were the older files that were missing.

A little update. I reverted back to version 0.3.1 and api version 0.8.7, which I had done before but never made sure that the reversion took place and ran a forced sync. The unmounting issues when uploading were fixed, as well as the missing files. Super random!

[cwallace]

Can confirm rolling back to 0.3.1 and api 0.8.7 resolves the missing file issue. Really sucks to have to downgrade though since 0.3.2 had so many fixes.

[pcjonathan]

I'm getting this too. It's a recent development. Not sure why, especially as I've not upgraded since it was working fine or added a huge amount of files in the past few days. Older files does seem to the theory I have too.

[ctejada10]

@cwallace Yeah, I agree. 0.3.2 added some cool stuff! Reverting also fixed the unmounting issues I've been having after issuing an upload command, not sure if I was the only one having these though.

@pcjonathan Can you check what version are you running? Try acd_cli version. The amount of files uploaded have never messed with the performance. I used to upload 1 TB every day for about a week and not a single glitch.

[pcjonathan]

@ctejada10 acd_cli 0.3.2, api 0.9.0, presumably same as above.

[minic00pers]

I noticed a problem like this last night. When I would run a sync, my main upload folder would no longer show. When I ran an old-sync, the files appeared as normal.

This might be a dumb question @ctejada10 but how do I revert to 0.3.1?

[kax0]

@minic00pers Specify the version on pip: pip3 install acdcli==0.3.1

[Demole]

The same happens to me

In a "Series" folder I have in ACD 242 folders, but there are only 232 in acd_cli

acd_cli 0.3.2, api 0.9.0

Sorry for my english

[ctejada10]

@minic00pers I specified the commit in the pip command, but as @kax0 says also works.

@pcjonathan reinstall version 0.3.1 to see if that improves the situation. Same for you @Demole.

[Demole]

Reinstall version 0.3.1 and arranged.

Thank you very much

[RazorII]

Having a few empty folders in my mounted drive. running: acd_cli 0.3.2, api 0.9.0

[yadayada]

Could anyone with missing files investigate the database entries? E.g. choose a missing file from the web interface and then use sqlitebrowser to check whether the node name is in the nodes table and the corresponding ID is in the files table?

[kax0]

@yadayada I queried two db's, a good and a bad one.

On the good one file exists on node table and has relationship on files table. On the missing files db, rows simply do not exists.

The db difference on my case is quite big, about 480kb beetween databases.

[ctejada10]

@kax0, @yadayada same here, though I did not measure the difference in dbs.

[ajkis]

Any updates on this issue ?

[l3uddz]

Is it correct in thinking this would be responsible for an episode appearing in the online cloud drive interface of amazon, but not appearing in the directory listing for acd_cli? Have an tv episode that for some reason is not appearing in acd_cli but is uploaded, this is after performing acd_cli sync numerous times.

[ajkis]

Yup thats the problem, but old acd version is so slow for browsing files compared to new one its just ...

I think something got borked with that db optimization.

[notDavid]

I'm also experiencing missing folders & files i recently uploaded - which are present in Amazons web interface...

acd_cli 0.3.2, api 0.9.0 ; Python 3.4.2 ; Linux-3.16.0-4-amd64-x86_64-with-debian-8.6

[notDavid]

So i'm not sure what my problem was, but i resolved it by updating both my machines to the latest acd_cli version, then i copied the entire .cache/acd_cli folder of the working host to the other one, and then i ran acd_cli -v init and noticed something about Migrating from schema version 1 to 2.

after that i can see all folders & files again in the mounted folder...

[ItsCompletelyYourFault]

Seems to exist within acd_cli 0.3.2 as no previous versions were installed on my box. Missing files exist online (webview), but acd_cli's sqlite db does not know it. "acd_cli sync" and "acd_cli -v init" do not recover the missing files (even after re-mount, given potential FUSE issues).

"acd_cli clear-cache && acd_cli sync" and fuse re-mount recovers the missing files.

Adding "resync [directory]"-ish might come in handy. However sync should be aware of count mismatches eventually.

[yadayada]

Since all the downgradees now should have done a clean (full) sync, I'd like to hear back from you in a few weeks whether files also go missing in 0.3.1 when you do subsequent "diff"-syncs.

"acd_cli clear-cache && acd_cli sync" and fuse re-mount recovers the missing files.

You could also just do a sync -f.

Adding "resync [directory]"-ish might come in handy. However sync should be aware of count mismatches eventually.

Due to public demand, I implemented a new partial sync command psync [-r] /dir

[ItsCompletelyYourFault]

I did not notice sync -f, thank you!

I (quick read) through your code and suspect the problem within sync_node_list.

(I am not a sqlite expert, especially on how it handles parallel access on the .db file.)

However sync_node_list seems not atomic. Whenever a user manages to run a sync while the other is not fully completed (hence LAST_SYNC_KEY is written after completion), we might end up with a black hole between two CHECKPOINT_KEYs.

Maybe flock while syncing?

Thanks for you work, its highly appreciated.

[reubendb]

I also just encountered this issue. I have two different VMs with acd_cli 0.3.2. On VM A, everything seems to work OK, but this is also the VM from which I upload most of the files (including the missing ones).

On VM B, I notice one directory with large number of files missing. I did "clear-cache" and "sync -f" multiple time and still didn't work. Finally I did "old-sync" and that seemed to have solved the issue for now. Not sure if/how this will resurface going forward.

[henfri]

Hello,

thanks for the hint. "acd_cli old-sync" did work for me as well.

Regards, Hendrik

[henfri]

.... but it did not work very long. The files are gone again after running duply once

[nickpdawson]

chiming in to say that I'm having the 'missing file' problem as well with acd_cli 0.3.2, api 0.9.2

I'm not nearly a proficient in python (or any language) as others on this tread. So I'm afraid I don't have much to offer.

I can say that I'm not having this problem with a box running acd_cli 0.3.1, api 0.9.0

To test, I'm mounting my ACD drive on Box A I write a file to /tmp I acd_cli upload /tmp /myfolder I've tried sync - and psync /myfolder and old-sync I check the ACD web site and see the file But the file is not in /mount/amazon/myfolder

On Box B running 0.3.1 I am able to see /mount/amazon/myfolder/file

Ditto on Macs running Expand Drive

Also, if I write file2 to /myfolder on a Mac with expand drive or linux box using 0.3.1 I can see it everywhere EXCEPT Box A even after: sync -f psync ... sync-old

[aidanharris]

I too am noticing missing files showing on Amazon but not in the fuse mount.

The only solution I've found so far is to unmount and re-sync everytime I change something e.g:

acd_cli umount && sudo umount /path/to/CloudDrive || true && acd_cli sync && acd_cli mount /path/to/CloudDrive

[KarmaPoliceT2]

I noticed missing files as well...

Noticed that if i PSYNC the folder that the file is missing from then it will re-appear, but if i do a SYNC (including with the -f) the file will not re-appear...

For now I've rolled back to acd_cli 0.3.1, api 0.8.7... cleared cache directory (except oauth)... and now i see all files again and things seem to be working steadily...

Will do some more debugging on 0.3.2 once i get the encryption layer removed and can duplicate the disappearing file issue again there... troubleshooting/debugging when all filenames are encrypted is a PITA...

[KarmaPoliceT2]

Following up, I experienced some files gone missing overnight last night against version 0.3.1/0.8.7... so the downgrade complete refresh of the db didn't work to solve the problem...