gsync
gsync copied to clipboard
iwonbigbro
gsync only create directories
Hello,
I have a directory that I wish to sync with google drive. My directory, temp, contain one text file, with the name apan.txt and content "apan"
$ gsync -r --progress -v temp/ drive://temp ./ apan.txt 0 0% 0.00B/s 0:00:00 sent 0 bytes received 0 bytes 0.00 bytes/sec
However, only the directory temp is created on the google drive.
What do I do wrong?
/Richard
Hi, I also get same issue, some of the progress output :
$ time gsync --owner --group --progress --perms -u -d -r -vi /opt/ict/monday/ drive://ict/
<f+++++++++ 2012/work-jp.docx 0 0% 0.00B/s 0:00:00 <f+++++++++ 2012/~$noplanning.docx 0 0% 0.00B/s 0:00:00 <f+++++++++ 2012/PLANNING (edit).xlsx 0 0% 0.00B/s 0:00:00 <f+++++++++ 2012/Planned 2012.docx 0 0% 0.00B/s 0:00:00 <f+++++++++ 2012/strategic (c).docx 0 0% 0.00B/s 0:00:00 <f+++++++++ 2012/presentation LATEST.pptx 0 0% 0.00B/s 0:00:00
all shows 0.00B/s ..when checking through Google Drive webpage.. only folder is created but no content...
Hi.. I fixed it by using method as this thread issue : https://github.com/iwonbigbro/gsync/issues/66 thanks to @GyroLand ...but some error occured while syncing :
return req.execute()
File "/usr/lib/python2.6/site-packages/oauth2client/util.py", line 132, in positional_wrapper
return wrapped(*args, **kwargs)
File "/usr/lib/python2.6/site-packages/apiclient/http.py", line 723, in execute
raise HttpError(resp, content, uri=self.uri)
Error: <HttpError 500 when requesting https://www.googleapis.com/drive/v2/files/xxxxxxxxxxxxxxxxxxxxxx?alt=json&setModifiedDate=false&newRevision=true returned "Internal Error">
sent 16005692534 bytes received 0 bytes 4471186.31 bytes/sec
sent 0 bytes received 0 bytes 0.00 bytes/sec
sent 0 bytes received 0 bytes 0.00 bytes/sec
... also many files were not sync maybe because of above error? it takes about 7 hours+ to sync document file total size of 8GB office files .doc, ppt, pdf, etc (this is by using 12 Mbps leased line dedicated Internet)
Yes, i have the same, i think this is some kind of a time out error or something. Anyway, if you don't use the -u flag and run the program several times, it will upload also the failed files. And it takes much less than the first time, as it uploads only the missing files.
I have the same issue and can't get files uploaded(with or without the -u flag). I've tried many times and the files still get ignored(or seem to).
Thanks, Phil
On Sat, Jul 12, 2014 at 10:41 AM, GyroLand [email protected] wrote:
Yes, i have the same, i think this is some kind of a time out error or something. Anyway, if you don't use the -u flag and run the program several times, it will upload also the failed files. And it takes much less than the first time, as it uploads only the missing files.
— Reply to this email directly or view it on GitHub https://github.com/iwonbigbro/gsync/issues/69#issuecomment-48814026.
Also having the directories but no files issue. I don't think it's a timeout error, running with -v shows that it's not spending much time on any given file.
I've yet to decipher what's happening myself. In my case I found a workaround by uploading my files manually, killing the md5sum and running gsync. It's been working well ever since. Must be done strange semi-rare condition. On Jul 22, 2014 11:25 PM, "nmatthews" [email protected] wrote:
Also having the directories but no files issue. I don't think it's a timeout error, running with -v shows that it's not spending much time on any given file.
— Reply to this email directly or view it on GitHub https://github.com/iwonbigbro/gsync/issues/69#issuecomment-49829002.
$ gsync --version 0.1.14
I'm also having this problem. It seems to have started only recently; old files seem to update (I think), but new ones don't upload. Directories are created, but are unpopulated on Google Drive. I'm wondering if it has to do with having a directory path that's more than one level deep. Here's one example:
gsync -c -r -t -p -o -g -v --progress --delete -l -s /mnt/PERSONAL/Dane/Recipes/ drive://orchestrator/Recipes
I've tried this with just the "-r -v --progress" options, as well, and it still doesn't upload the files. (I refreshed the Google Drive page a few times, just to be sure.) My network activity applet reports that while (supposedly) uploading directories full of small text files, the download speed is ~12KiB/s, and the upload speed is ~14KiB/s. (My Internet connection is much, much faster than this, so I don't think it's an uplink issue.)
I've tried doing this sync multiple times, and the files haven't yet been uploaded, so the solution, above, about just trying it a few times seems not to be working for me.
Thanks for your work on this! It's truly an indespensible tool!
It looks like editing /usr/local/lib/python2.7/dist-packages/libgsync/drive/__init__.py, and changing:
body = {}
for k, v in properties.iteritems():
body[k] = _Drive.utf8(v)
...to:
body = {}
for k, v in properties.iteritems():
if v is not None:
body[k] = _Drive.utf8(v)
...solves the problem, per #66.
It does, however, produce this when it runs into a file that ends in a ~ character:
Monsters/Lvl. 3 - Undead Horrid Housecat~
0 0% 0.00B/s 0:00:00DEBUG: 'Exception': File "/usr/local/lib/python2.7/dist-packages/libgsync/drive/__init__.py", line 712, in update
status, res = req.next_chunk()
File "/usr/local/lib/python2.7/dist-packages/oauth2client/util.py", line 132, in positional_wrapper
return wrapped(*args, **kwargs)
File "/usr/local/lib/python2.7/dist-packages/apiclient/http.py", line 874, in next_chunk
return self._process_response(resp, content)
File "/usr/local/lib/python2.7/dist-packages/apiclient/http.py", line 901, in _process_response
raise HttpError(resp, content, uri=self.uri)
Worked for me.
Thanks, Phil
On Fri, Sep 19, 2014 at 1:05 PM, dmutters [email protected] wrote:
It looks like editing /usr/local/lib/python2.7/dist-packages/libgsync/drive/init.py, and changing:
body = {} for k, v in properties.iteritems(): body[k] = _Drive.utf8(v)
...to:
body = {} for k, v in properties.iteritems(): if v is not None: body[k] = _Drive.utf8(v)
...solves the problem, per #66 https://github.com/iwonbigbro/gsync/issues/66.
— Reply to this email directly or view it on GitHub https://github.com/iwonbigbro/gsync/issues/69#issuecomment-56205380.
It worked for a while, for me, and then broke. This is what I did to fix it (as root):
- (Install pip using your distro's package manager.)
- pip uninstall gsync
- rm -vfr /usr/local/lib/python2.7/dist-packages/libgsync /usr/local/bin/gsync
- pip install gsync
- pip install --upgrade gsync
It's once again working for me with only the occasional hiccup.
the fix worked for me for quite some time and just the other I realized it had stopped. On Oct 31, 2014 2:54 PM, "dmutters" [email protected] wrote:
It worked for a while, for me, and then broke. This is what I did to fix it (as root):
- (Install pip using your distro's package manager.)
- pip uninstall gsync
- rm -vfr /usr/local/lib/python2.7/dist-packages/libgsync /usr/local/bin/gsync
- pip install gsync
- pip install --upgrade gsync
It's once again working for me with only the occasional hiccup.
— Reply to this email directly or view it on GitHub https://github.com/iwonbigbro/gsync/issues/69#issuecomment-61311212.
I'm thinking this might be caused by something else.
Found a "Daily Limit for Unauthenticated Use Exceeded. Continued use requires signup." error in the cache files link for the files API.
If you
grep -R "selfLink" ~/.gsync/ | more
and look for the https://www.googleapis.com/drive/v2/files/.......... links and try them, I got the response from Google about a limit.
If you also get the key
https://www.googleapis.com/drive/v2/files/0B42Fqxu8L6YfYkZW.........
Then we are all using the same key.
Thank you,
bingo.
Thanks, Phil
On Fri, Oct 31, 2014 at 9:23 PM, johanels [email protected] wrote:
I'm thinking this might be caused by something else.
Found a "Daily Limit for Unauthenticated Use Exceeded. Continued use requires signup." error in the cache files link for the files API.
If you
grep -R "selfLink" ~/.gsync/ | more
and look for the https://www.googleapis.com/drive/v2/files/.......... links and try them, I got the response from Google about a limit.
If you also get the key
https://www.googleapis.com/drive/v2/files/0B42Fqxu8L6YfYkZW.........
Then we are all using the same key.
Thank you,
— Reply to this email directly or view it on GitHub https://github.com/iwonbigbro/gsync/issues/69#issuecomment-61352007.
My logs are full of this notice. Thanks for pointing it out.
How can we see what key we're using? That would be a pretty big problem if we're all using the same key, and there's a limit on how much data transfer can be ascribed to any one key.
Hi,
The Google JSON key information is loaded in /usr/local/lib/python2.7/dist-packages/libgsync/drive/client_json.py and are the same from the source, so everyone is using the same key.
https://github.com/iwonbigbro/gsync/blob/master/libgsync/drive/client_json.py
I created a new project on Google Developers site, created the JSON auth and changed the file on my machine. I see the requests in Google Developers Dashboard, but number of requests matches number of errors, and gsync still does not upload.
https://console.developers.google.com
Will keep trying.
Thanks, johanels. Let us know when you get it sorted! Hopefully, such a fix can be integrated into the gsync code...
As johanels indicated, the gsync utility is set to use pre-defined Google API key / credentials when shipped.
This means that by default all users will utilise the data limit (typically 10m API requests per day).
Therefore, you need to create your own Google API Project with associated key / credentials.
Method to resolve this:
- Goto: https://console.developers.google.com/project/
- Create a new project and go to that Projects Dashboard, then:
- Enable an API -> Set Drive API to On
- APIs & Auth -> Consent Screen. Configure the consent screen, set a project name 'gsync-someotherthing' (or suchlike) and any relevant information
- APIs & Auth -> Credentials -> Create new client ID. Select 'installed application'. Type: Other
- As root, edit /usr/local/lib/python2.7/dist-packages/libgsync/drive/client_json.py and modify the client_id and client_secret to reflect your newly generated client credentials
- Also apply fix #66
- As your required (suggest non-root..) user, run gsync --authenticate and follow the instructions as per the standard docs
- Uploading of files will work, against your own personal API requests quota
All the best,
Jez
Thank you Jez. Seems to be running and I can see Response codes in the Development Console.
Great. Have fun.
The solution is clearly to move the client_json requirement to within a per user .gsync folder.
I plan on submitting a patch for this in the near future.
Jez
On Tuesday, 6 January 2015, johanels [email protected] wrote:
Thank you Jez. Seems to be running and I can see Response codes in the Development Console.
— Reply to this email directly or view it on GitHub https://github.com/iwonbigbro/gsync/issues/69#issuecomment-68919177.
This email is confidential in that it is intended for the exclusive attention of the addressee(s) indicated. If you are not the intended recipient, this email should not be read or disclosed to any other person. Please notify the sender immediately and delete this email from your computer system. Any opinions expressed are not necessarily those of the company from which this email was sent and, whilst to the best of our knowledge no viruses or defects exist, no responsibility can be accepted for any loss or damage arising from its receipt or subsequent use of this email.
Hi Jez, I've been following this fix and am struggling as I don't seem to have the directory:
/usr/local/lib/python2.7/dist-packages/libgsync/drive/
I'm on Centos 6.6 server and have tried uninstalling and re-installing GSync following carrying out an installation of Python 2.7.
I would appreciate your guidance.
Many thanks, Simon
Hi
So I don't have access to a CentOS 6.6 right this second.
However, the issue is highly likely to be just that libgsync is in a different place to the one on my system.
To find it, you could do either:
jtucker@elmo:~/Desktop$ pip show -f gsync --- Name: gsync Version: 0.1.14 Location: /usr/local/lib/python2.7/dist-packages Requires: google-api-python-client, docopt, httplib2, oauth2client, python-dateutil, urllib3 Files: Cannot locate installed-files.txt
So libgsync on mine is under: /usr/local/lib/python2.7/dist-packages
(I'm presuming you've used pip, if you've used the git version, you should know where you've installed it ;-)
or
updatedb locate drive | grep libgsync
or
find / -name "drive" -type d -print | grep libgsync
All the best,
Jez
Thank you oh so much. The first command you issued showed me that instead of dist-packages, it was within site-packages.
I shall progress now. Many many thanks.
https://github.com/google/skicka already handles this, and appears to also meet my basic needs for syncing content up and down to google drive.
Hi there, I'm still getting the exceeded 30GB issue, but my client_json.py file has the google API secret in it as followed by the steps above. It's giving me the following error:
DEBUG: 'Exception': File "/usr/local/lib/python2.7/site-packages/libgsync/drive/init.py", line 703, in update return req.execute() File "/usr/local/lib/python2.7/site-packages/oauth2client/util.py", line 142, in positional_wrappe r return wrapped(_args, *_kwargs) File "/usr/local/lib/python2.7/site-packages/googleapiclient/http.py", line 729, in execute raise HttpError(resp, content, uri=self.uri)
DEBUG: <HttpError 500 when requesting https://www.googleapis.com/drive/v2/files/0B2oFXY6ToU6fNE04aUY wclZFMTg?alt=json&setModifiedDate=true&newRevision=true returned "Internal Error">: File "/usr/loc al/lib/python2.7/site-packages/libgsync/crawler.py", line 150, in run self._walk(srcpath, self._walkCallback, self._dev) File "/usr/local/lib/python2.7/site-packages/libgsync/crawler.py", line 124, in _walk self._sync(f) File "/usr/local/lib/python2.7/site-packages/libgsync/sync/init.py", line 31, in call self._sync(path) File "/usr/local/lib/python2.7/site-packages/libgsync/sync/init.py", line 197, in _sync self.dst.update(dstPath, srcFile) File "/usr/local/lib/python2.7/site-packages/libgsync/sync/file/init.py", line 325, in update self.updateFile(path, srcObj) File "/usr/local/lib/python2.7/site-packages/libgsync/sync/file/__init.py", line 228, in updat eFile self.__updateStats(path, src) File "/usr/local/lib/python2.7/site-packages/libgsync/sync/file/__init.py", line 273, in updat eStats self._updateStats(path, src, mode, uid, gid, mtime, atime) File "/usr/local/lib/python2.7/site-packages/libgsync/sync/file/remote/__init.py", line 148, in _updateStats 'setModifiedDate': GsyncOptions.times File "/usr/local/lib/python2.7/site-packages/libgsync/drive/init.py", line 703, in update return req.execute() File "/usr/local/lib/python2.7/site-packages/oauth2client/util.py", line 142, in positional_wrappe r return wrapped(_args, *_kwargs) File "/usr/local/lib/python2.7/site-packages/googleapiclient/http.py", line 729, in execute raise HttpError(resp, content, uri=self.uri)
Error: <HttpError 500 when requesting https://www.googleapis.com/drive/v2/files/0B2oFXY6ToU6fNE04aUY wclZFMTg?alt=json&setModifiedDate=true&newRevision=true returned "Internal Error"> sent 307341525 bytes received 0 bytes 60409.28 bytes/sec
When I go in to the link, it tells me the following:
{ "error": { "errors": [ { "domain": "usageLimits", "reason": "dailyLimitExceededUnreg", "message": "Daily Limit for Unauthenticated Use Exceeded. Continued use requires signup.", "extendedHelp": "https://code.google.com/apis/console" } ], "code": 403, "message": "Daily Limit for Unauthenticated Use Exceeded. Continued use requires signup." } }
but the issue is that I already have a google API Secret with an account that has a 1TB limit.
Please can I have further assistance?