git-remote-dropbox
git-remote-dropbox copied to clipboard
anishathalye
error: failed to push some refs for large file
I'm pushing a very large file(7gigs, thousands of files) into my Dropbox, and my pushes fail partway through every time. I've tried pushing a couple different times to check where the progress stops, but it's inconsistent.
the error I normally get is: "error: failed to push some refs to Dropbox:///Git-Dropbox:Speeder"
I started using git-remote-dropbox in order to store an entire unity game project I'm working on, including assets in order to quickly be able to share between team members and also track updates to all the files.
A bit of research says that this looks similar to issue issues 15 , 36, and 57, but I couldn't find solutions in those issues.
git version 2.22 windows 10 v1903 python 3.8 git-remote-dropbox latest release
running the program with -v gives
Traceback (most recent call last): File "c:\users\0rson\appdata\local\programs\python\python38\lib\site-packages\urllib3\connectionpool.py", line 665, in urlopen httplib_response = self._make_request( File "c:\users\0rson\appdata\local\programs\python\python38\lib\site-packages\urllib3\connectionpool.py", line 387, in _make_request conn.request(method, url, **httplib_request_kw) File "c:\users\0rson\appdata\local\programs\python\python38\lib\http\client.py", line 1230, in request self._send_request(method, url, body, headers, encode_chunked) File "c:\users\0rson\appdata\local\programs\python\python38\lib\http\client.py", line 1276, in _send_request self.endheaders(body, encode_chunked=encode_chunked) File "c:\users\0rson\appdata\local\programs\python\python38\lib\http\client.py", line 1225, in endheaders self._send_output(message_body, encode_chunked=encode_chunked) File "c:\users\0rson\appdata\local\programs\python\python38\lib\http\client.py", line 1043, in _send_output self.send(chunk) File "c:\users\0rson\appdata\local\programs\python\python38\lib\http\client.py", line 965, in send self.sock.sendall(data) File "c:\users\0rson\appdata\local\programs\python\python38\lib\ssl.py", line 1204, in sendall debug: writing: /git-dropbox:speeder/objects/2c/5bf73c65039d992b554a6c683046fcae495417 v = self.send(byte_view[count:]) File "c:\users\0rson\appdata\local\programs\python\python38\lib\ssl.py", line 1173, in send return self._sslobj.write(data) socket.timeout: The write operation timed out
During handling of the above exception, another exception occurred:
Traceback (most recent call last): File "c:\users\0rson\appdata\local\programs\python\python38\lib\site-packages\requests\adapters.py", line 439, in send resp = conn.urlopen( File "c:\users\0rson\appdata\local\programs\python\python38\lib\site-packages\urllib3\connectionpool.py", line 719, in urlopen retries = retries.increment( File "c:\users\0rson\appdata\local\programs\python\python38\lib\site-packages\urllib3\util\retry.py", line 400, in increment raise six.reraise(type(error), error, _stacktrace) File "c:\users\0rson\appdata\local\programs\python\python38\lib\site-packages\urllib3\packages\six.py", line 734, in reraise raise value.with_traceback(tb) File "c:\users\0rson\appdata\local\programs\python\python38\lib\site-packages\urllib3\connectionpool.py", line 665, in urlopen httplib_response = self._make_request( File "c:\users\0rson\appdata\local\programs\python\python38\lib\site-packages\urllib3\connectionpool.py", line 387, in _make_request conn.request(method, url, **httplib_request_kw) File "c:\users\0rson\appdata\local\programs\python\python38\lib\http\client.py", line 1230, in request self._send_request(method, url, body, headers, encode_chunked) File "c:\users\0rson\appdata\local\programs\python\python38\lib\http\client.py", line 1276, in _send_request self.endheaders(body, encode_chunked=encode_chunked) File "c:\users\0rson\appdata\local\programs\python\python38\lib\http\client.py", line 1225, in endheaders self._send_output(message_body, encode_chunked=encode_chunked) File "c:\users\0rson\appdata\local\programs\python\python38\lib\http\client.py", line 1043, in _send_output self.send(chunk) File "c:\users\0rson\appdata\local\programs\python\python38\lib\http\client.py", line 965, in send self.sock.sendall(data) File "c:\users\0rson\appdata\local\programs\python\python38\lib\ssl.py", line 1204, in sendall v = self.send(byte_view[count:]) File "c:\users\0rson\appdata\local\programs\python\python38\lib\ssl.py", line 1173, in send return self._sslobj.write(data) urllib3.exceptions.ProtocolError: ('Connection aborted.', timeout('The write operation timed out'))
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "c:\users\0rson\appdata\local\programs\python\python38\lib\runpy.py", line 193, in _run_module_as_main
return _run_code(code, main_globals, None,
File "c:\users\0rson\appdata\local\programs\python\python38\lib\runpy.py", line 86, in run_code
exec(code, run_globals)
File "C:\Users\0rson\AppData\Local\Programs\Python\Python38\Scripts\git-remote-Dropbox.exe_main.py", line 9, in
Yeah, I agree, there seems to be some issue with pushing repos with lots of large files. Or even a small number of large files (though I see such issues randomly, it's not consistent).
I didn't have any repos quite as large as yours, so I made a synthetic one with 1000 1M files, 100 10M files, 10 100M files, and 1 1G file with the following script:
for i in $(seq 1000); do
dd if=/dev/urandom of=file-1M-$i.bin bs=1M count=1
done
for i in $(seq 100); do
dd if=/dev/urandom of=file-10M-$i.bin bs=1M count=10
done
for i in $(seq 10); do
dd if=/dev/urandom of=file-100M-$i.bin bs=1M count=100
done
for i in $(seq 1); do
dd if=/dev/urandom of=file-1000M-$i.bin bs=1M count=1000
done
After this, I created a git repo, added all the files, and pushed it to dropbox. I got a slightly different error than yours:
...
debug: writing: /big-files/objects/69/7aaf196a5642650502f0d40f19c2d8f7c25b7e (size 10 MB)
debug: writing: /big-files/objects/eb/e2cc3f1c2b556d204fa26925eba9fb96e054d0 (size 10 MB)
debug: writing: /big-files/objects/c6/dd8888eba473c57aaf1ab7d9b39ac9fe5ed1b2 (size 10 MB)
debug: writing: /big-files/objects/69/60bc14dcaacf4b153d4eb527f3f8d9484fa1ca (size 10 MB)
debug: writing: /big-files/objects/7d/d6344e6cf0dc3cbe1f843854d4466b6a08a88e (size 10 MB)
debug: writing: /big-files/objects/30/7897d0c185a84e43498d474879b61133d5fc14 (size 10 MB)
Traceback (most recent call last):
File "/home/anish/.local/bin/git-remote-dropbox", line 11, in <module>
load_entry_point('git-remote-dropbox', 'console_scripts', 'git-remote-dropbox')()
File "/home/anish/src/git-remote-dropbox/git_remote_dropbox/cli/helper.py", line 23, in main
helper.run()
File "/home/anish/src/git-remote-dropbox/git_remote_dropbox/helper.py", line 108, in run
self._do_push(line)
File "/home/anish/src/git-remote-dropbox/git_remote_dropbox/helper.py", line 152, in _do_push
self._push(src, dst)
File "/home/anish/src/git-remote-dropbox/git_remote_dropbox/helper.py", line 215, in _push
for done, _ in enumerate(res, 1):
File "/usr/lib/python2.7/multiprocessing/pool.py", line 673, in next
raise value
requests.exceptions.ConnectionError: ('Connection aborted.', error("(32, 'EPIPE')",))
Exception in thread Thread-20 (most likely raised during interpreter shutdown):
Traceback (most recent call last):
File "/usr/lib/python2.7/threading.py", line 801, in __bootstrap_inner
File "/usr/lib/python2.7/threading.py", line 754, in run
File "/usr/lib/python2.7/multiprocessing/pool.py", line 113, in worker
File "/home/anish/src/git-remote-dropbox/git_remote_dropbox/util.py", line 103, in __call__
File "/home/anish/src/git-remote-dropbox/git_remote_dropbox/helper.py", line 278, in _put_object
File "/home/anish/src/git-remote-dropbox/git_remote_dropbox/git.py", line 95, in encode_object
File "/home/anish/src/git-remote-dropbox/git_remote_dropbox/git.py", line 80, in object_data
File "/home/anish/src/git-remote-dropbox/git_remote_dropbox/git.py", line 12, in command_output
File "/usr/lib/python2.7/subprocess.py", line 217, in check_output
File "/usr/lib/python2.7/subprocess.py", line 480, in communicate
File "/usr/lib/python2.7/subprocess.py", line 1099, in wait
<type 'exceptions.AttributeError'>: 'NoneType' object has no attribute 'waitpid'
error: failed to push some refs to 'dropbox:///big-files'
The failures are also nondeterministic. I tried a similar experiment as above, but with only 10 100M files and 1 1G file, and sometimes the push worked, and sometimes it failed. Sometimes, it failed with a different error:
debug: writing: /big-files/objects/ac/aa333ca0f93885f2cf872a3d04f6f5a65e6bd7 (size 1 MB)
Traceback (most recent call last):
File "/home/anish/src/git-remote-dropbox/git_remote_dropbox/helper.py", line 316, in _put_object
conn.files_upload_session_finish(chunk, cursor, commit_info)
File "/home/anish/.local/lib/python2.7/site-packages/dropbox/base.py", line 2306, in files_upload_session_finish
debug: writing: /big-files/objects/b1/84940e6c51735c7933af51818f1821429be37d (size 1 MB)
f,
File "/home/anish/.local/lib/python2.7/site-packages/dropbox/dropbox.py", line 296, in request
user_message_locale)
ApiError: ApiError('71252a8652b3eb530ddc9f3f3512b613', UploadSessionFinishError(u'lookup_failed', UploadSessionLookupError(u'incorrect_offset', UploadSessionOffsetError(correct_offset=1048895882))))
Traceback (most recent call last):
File "/home/anish/.local/bin/git-remote-dropbox", line 11, in <module>
load_entry_point('git-remote-dropbox', 'console_scripts', 'git-remote-dropbox')()
File "/home/anish/src/git-remote-dropbox/git_remote_dropbox/cli/helper.py", line 23, in main
helper.run()
File "/home/anish/src/git-remote-dropbox/git_remote_dropbox/helper.py", line 108, in run
self._do_push(line)
File "/home/anish/src/git-remote-dropbox/git_remote_dropbox/helper.py", line 152, in _do_push
self._push(src, dst)
File "/home/anish/src/git-remote-dropbox/git_remote_dropbox/helper.py", line 215, in _push
for done, _ in enumerate(res, 1):
File "/usr/lib/python2.7/multiprocessing/pool.py", line 673, in next
raise value
dropbox.exceptions.ApiError: ApiError('71252a8652b3eb530ddc9f3f3512b613', UploadSessionFinishError(u'lookup_failed', UploadSessionLookupError(u'incorrect_offset', UploadSessionOffsetError(correct_offset=1048895882))))
error: failed to push some refs to 'dropbox:///big-files'
As a temporary workaround, here's a suggestion you can try. If you have many commits, rather than trying to push the big repository at once, you can do it in pieces, so you can "checkpoint" progress. You can push older commits, e.g. with git push origin <commit hash of some ancestor of master>:master. And you can keep doing this with newer and newer ancestors, so you'll push partial data, and finally you can git push -u origin master to sync up Dropbox.
And actually, even the above example, with the huge number of files (1000 x 1M, ..., 1 x 1G) worked for me just now (pushing the entire repo at once).
coming back to this because I'm still having issues. I was able to make it work before just by continuously trying, but I've run into the same problem again.
For reference I'm pushing 14000 objects, so that could very well be the issue.
I have programming experience in C# but not in python, and I don't fully know what this code is doing, but here's what I've done and where I'm stuck.
so as per the advice of this, I've changed socket timeout to none.
gone into helper.py and changed
90 return dropbox.Dropbox(self._token)
to
90 return dropbox.Dropbox(self._token, timeout=None))
This manages to finish writing all the objects for my project, except then I get issues in UploadSessionOffsetError.
tracking down this error leads me to: link1 link2
I can see that understanding and solving this issue will take some time for me, but I think I'm going to keep working on it because it would be really elegant if I could get this program working for my use case.
I'd be grateful for help or advice.
I have no idea if it will help, but look at constants.py. Try reducing processes to 10 or fewer, increasing max retries, reducing (or increasing) chunk size.
Also try setting a larger finite timeout as suggested here (instead of None, which may just cause problems, conceivably):
https://github.com/anishathalye/git-remote-dropbox/issues/36#issuecomment-261399141
I was able to get the entire project pushed after trying several times, changing the constants.py settings between pushes. I also added a GitIgnore to the unity project in order to stop syncing non important files.
I solved my problem, but the issue is one change to constants worked for one project, but not for another, and having to go through that makes me uncomfortable using this with collaborators.
I've looked into UploadSessionOffsetError and found this. I think line endings could be the issue, might explain discrepancies in file size and I have been getting alerts that my line endings have been getting changed. I'll check this out later.
Hello, i have the same issues with a project in Unreal Engine where i have to push 400 files for my first push. I installed Git LFS for letting me to push heavy files as fbx or psd files. i got the following errors : "SSH: Could Not Resolve Hostname dropbox: no such host is known status 255" and at last line the "error: failed to push some refs for large file"
the link of my repo is "dropbox:///Git" which is neither a https nor a ssh in my opinion that's probably why it's bugging. I uninstalled LFS and it's working again.
is it possible to get a solution for keeping your dropbox module at same time using LFS ? Thank you.
ps: i tried again with another commit having only few files by letting LFS and it worked. i don't know why i had that error tho, but removing LFS let me to push my 400 files though.
git-remote-dropbox does not currently have any integration with Git LFS.