onedrive-d-old
onedrive-d-old copied to clipboard
xybu
Uncaught OneDriveAPIException exceptions
I'm trying to upload a large number of files (all my photos from G+ aprox 40G) to onedrive. The issue I encounter is that the upload task (not sure if that's the correct name) keeps stopping after a while. I used to restart onedrive-d and the upload went forward but after a while it stopped alltoghether. All the folders have synced (850) but not the files. In the log I only see added task "up" for files but no uploading block. One more thing worth mentioning is that the folder I'm trying to upload is symlinked inside ~/OneDrive.
How can I extract more debug information ? Is there a limit on the number of files onedrive-d can sync ?
For small files (size smaller than the BITS min file size parameter) the program uses HTTP PUT to upload the file and for large files the program uses MS BITS_POST which I found not as reliable as I assume.
HTTP PUT upload method does not have detailed log since it is assumed to succeed (unless IO Error). If you see a log saying "get" task of type "up" then some worker is uploading that file. If you see a line saying "add" log "up" then a new task is created and waits handling. You could try monitoring network traffic and see if things get sent out.
Another thing is that MS SkyDrive API (Live API) has a limit on how many requests can be made in a period of time. Current program in future branch could not handle it. It could crash workers one by one. 3rd party applications like onedrive-d suffer from more undocumented limits.
Next version of onedrive-d uses MS OneDrive API which was released about a month ago. While the new version is slowly in progress, the new OneDrive API is itself under testing. I can't guarantee an ETA for next version.
Sincerely, Xiangyu Bu
Sent from my Windows Phone
From: Emanuel Posescumailto:[email protected] Sent: 3/20/2015 3:16 AM To: xybu/onedrive-dmailto:[email protected] Subject: [onedrive-d] Syncing large number of files (#144)
I'm trying to upload a large number of files (all my photos from G+ aprox 40G) to onedrive. The issue I encounter is that the upload task (not sure if that's the correct name) keeps stopping after a while. I used to restart onedrive-d and the upload went forward but after a while it stopped alltoghether. All the folders have synced (850) but not the files. In the log I only see added task "up" for files but no uploading block. One more thing worth mentioning is that the folder I'm trying to upload is symlinked inside ~/OneDrive.
How can I extract more debug information ? Is there a limit on the number of files onedrive-d can sync ?
Reply to this email directly or view it on GitHub: https://github.com/xybu/onedrive-d/issues/144
That's the thing, I only have added task "up" on files and got task: sy on folders, so no got task: up on files. I will try to do a tcpdump and investigate the traffic.
Could you post some log? Preferably from MainThread start to the death of the program. Unnecessary stuff can be left out, but be sure to include anything about "exception". Don't use daemon mode btw.
Sincerely, Xiangyu Bu
Sent from my Windows Phone
From: Emanuel Posescumailto:[email protected] Sent: 3/20/2015 3:44 AM To: xybu/onedrive-dmailto:[email protected] Cc: Xiangyu Bumailto:[email protected] Subject: Re: [onedrive-d] Syncing large number of files (#144)
That's the thing, I only have added task "up" on files and got task: sy on folders, so no got task: up on files. I will try to do a tcpdump and investigate the traffic.
Reply to this email directly or view it on GitHub: https://github.com/xybu/onedrive-d/issues/144#issuecomment-83944172
I think I'm getting somewhere. All 4 worker threads seem to get stuck on this exception (just different folder/resource names)
Exception in thread worker3: Traceback (most recent call last): File "/usr/lib/python3.3/threading.py", line 637, in _bootstrap_inner self.run() File "/usr/local/lib/python3.3/dist-packages/onedrive_d-1.1.0dev-py3.3.egg/onedrive_d/od_worker_thread.py", line 443, in run self.make_remote_dir(task) File "/usr/local/lib/python3.3/dist-packages/onedrive_d-1.1.0dev-py3.3.egg/onedrive_d/od_worker_thread.py", line 338, in make_remote_dir new_entry = self.api.mkdir(name, task['remote_parent_id']) File "/usr/local/lib/python3.3/dist-packages/onedrive_d-1.1.0dev-py3.3.egg/onedrive_d/od_onedrive_api.py", line 328, in mkdir return self.parse_response(r, OneDriveAPIException, requests.codes.created) File "/usr/local/lib/python3.3/dist-packages/onedrive_d-1.1.0dev-py3.3.egg/onedrive_d/od_onedrive_api.py", line 115, in parse_response raise error(ret) onedrive_d.od_onedrive_api.OneDriveAPIException: The resource couldn't be created because a resource named '2013-09-14 -' already exists. (resource_already_exists)
So the remote dir did not exist when its parent was scanned, but was created before the "mkdir" task was performed. But at most one task can exist for a given local path.
May I know if any other onedrive client is running? It could be possible that another client created the dir before onedrive-d did.
But this is a problem. Should issue a fix to make the worker more robust.
I don't know how this happened but it seems that I have duplicate folder names. I have no idea how this happened but there are 2 folders named '2013-09-14 -' with different content. Something must have gone wrong during the unzip of the archives I downloaded from Google with my backups. So it appears it's not really a onedrive-d issue (tho gracefully treating that exception would be nice to have).
I have another exception that crashes the worker (I will rename the folder)
Exception in thread worker2: Traceback (most recent call last): File "/usr/lib/python3.3/threading.py", line 637, in bootstrap_inner self.run() File "/usr/local/lib/python3.3/dist-packages/onedrive_d-1.1.0dev-py3.3.egg/onedrive_d/od_worker_thread.py", line 443, in run self.make_remote_dir(task) File "/usr/local/lib/python3.3/dist-packages/onedrive_d-1.1.0dev-py3.3.egg/onedrive_d/od_worker_thread.py", line 338, in make_remote_dir new_entry = self.api.mkdir(name, task['remote_parent_id']) File "/usr/local/lib/python3.3/dist-packages/onedrive_d-1.1.0dev-py3.3.egg/onedrive_d/od_onedrive_api.py", line 328, in mkdir return self.parse_response(r, OneDriveAPIException, requests.codes.created) File "/usr/local/lib/python3.3/dist-packages/onedrive_d-1.1.0dev-py3.3.egg/onedrive_d/od_onedrive_api.py", line 115, in parse_response raise error(ret) onedrive_d.od_onedrive_api.OneDriveAPIException: The operation couldn't be completed because the requested folder name, 'Hangout Bogdan Postelnicu ● Bianca Stan ● Vali O.' isn't supported. A folder name must not start or end with '.', must not contain any unprintable characters or contain any of the following characters: '', '/', ':', ';', '*', '<', '>', '|', '?'. (request_parameter_invalid)
Ignore list not working?
Sincerely, Xiangyu Bu
Sent from my Windows Phone
From: Emanuel Posescumailto:[email protected] Sent: 3/23/2015 3:32 AM To: xybu/onedrive-dmailto:[email protected] Cc: Xiangyu Bumailto:[email protected] Subject: Re: [onedrive-d] Syncing large number of files (#144)
I have another exception that crashes the worker (I will rename the folder)
Exception in thread worker2: Traceback (most recent call last): File "/usr/lib/python3.3/threading.py", line 637, in bootstrap_inner self.run() File "/usr/local/lib/python3.3/dist-packages/onedrive_d-1.1.0dev-py3.3.egg/onedrive_d/od_worker_thread.py", line 443, in run self.make_remote_dir(task) File "/usr/local/lib/python3.3/dist-packages/onedrive_d-1.1.0dev-py3.3.egg/onedrive_d/od_worker_thread.py", line 338, in make_remote_dir new_entry = self.api.mkdir(name, task['remote_parent_id']) File "/usr/local/lib/python3.3/dist-packages/onedrive_d-1.1.0dev-py3.3.egg/onedrive_d/od_onedrive_api.py", line 328, in mkdir return self.parse_response(r, OneDriveAPIException, requests.codes.created) File "/usr/local/lib/python3.3/dist-packages/onedrive_d-1.1.0dev-py3.3.egg/onedrive_d/od_onedrive_api.py", line 115, in parse_response raise error(ret) onedrive_d.od_onedrive_api.OneDriveAPIException: The operation couldn't be completed because the requested folder name, 'Hangout Bogdan Postelnicu ● Bianca Stan ● Vali O.' isn't supported. A folder name must not start or end with '.', must not contain any unprintable characters or contain any of the following characters: '', '/', ':', ';', '*', '<', '>', '|', '?'. (request_parameter_invalid)
Reply to this email directly or view it on GitHub: https://github.com/xybu/onedrive-d/issues/144#issuecomment-84856893
Ignore list is working for the extra content that I added, but there is no ignore rule (as far as I can tell) for folders ending in '.' (dot), just for those begining with '.'
Thanks for catching this. It is a bug. The server response is not documented in Live API either. Should issue a fix. But can't guarantee when. Floods of homework coming...
If we could check if we've reached the limit on how many requests can be made in a period of time (some exception caught?), we can 'pause' the producer threads for a while. Or we can take some other rate control methords such as token bucket.
Might the mkdir conflict due to inconsistency? One worker has just made remote modification under the parent path while another just goes to scan same path. (remote change has not been observed yet?)
Also there is possibility that sync task for a path has returned while network IO under this path is on-going. I added another post on #153
@starrwang Bandwidth sounds like another topic. Please create a new issue for discussion. Not sure what kind of bandwidth you are talking about; not to mention how to detect it.
Also each local path can have at most one task (pending or running). This also holds for the new version.