jottalib icon indicating copy to clipboard operation
jottalib copied to clipboard

Growing files without re-uploading

Open havardgulldahl opened this issue 10 years ago • 10 comments

JottaCloud supports partial uploads using the HTTP Content-Range header.

jottalib needs to add support for this to fix #10 and hopefully #8 as well.

havardgulldahl avatar Dec 07 '14 20:12 havardgulldahl

Some clarification:

We can upload files in a stream ( #18 ), and hopefully resume uploads ( #22 ), so this bug is about uploading arbitrary chunks of data to an existing file.

If someone is able to do this with the official client, please drop the HTTP log (headers only) here.

havardgulldahl avatar May 27 '15 22:05 havardgulldahl

@havardgulldahl I'm looking into this now, any suggestions on how to simulate this? I tried shutting down the client while uploading but that restarts the entire upload from scratch (when starting the client again)

EDIT: https://gist.github.com/jkaberg/f387a9de553f374a202c (let me know if this is wrong)

EDIT2: One more, this one is a lot smaller aswell https://gist.github.com/jkaberg/a20b80291eb041801540 . This time I paused the upload, exited the client, started up the client and unpaused the client which should have the effect you desire

jkaberg avatar May 28 '15 07:05 jkaberg

UPDATE: I see somebody already did this...

I've been writing my own client in java (lost the code, but still). I was thinking of splitting up the file client side and then uploading it chunks. Of course it wouldn't help with re uploading only the changed chunk later (maybe somebody can think of something clever here), but it would help in terms of md5ing large files (and I believe memory usage) and I suppose you could even upload in parallel.

In addition you won't have to upload the whole file again, if corrupted data is transferred.

Downside is this client would have to be used to automatically merge it back together. Still worth some thoughts.

nuth avatar May 28 '15 12:05 nuth

Don't you have a pause button? Or in the context menu from the tray?

Sendt fra min mobiltelefon

Den 28. mai 2015 kl. 09.19 skrev Joel Kåberg [email protected]:

@havardgulldahl I'm looking into this now, any suggestions on how to simulate this? I tried shutting down the client while uploading but that restarts the entire upload from scratch

— Reply to this email directly or view it on GitHub.

havardgulldahl avatar May 28 '15 15:05 havardgulldahl

@havardgulldahl See my post above, I revisited it with 2 logs

jkaberg avatar May 28 '15 16:05 jkaberg

Thanks, @jkaberg , these logs clearly show that resuming upload is doable, so good news for #22

However, for this bug, which is about growing an existing file without re-uploading (i.e. appending data at the end of the file), this doesn't get any further. I'm not sure the Jottacloud API needs to support it, so it may not be possible.

havardgulldahl avatar May 28 '15 17:05 havardgulldahl

@havardgulldahl Not sure on how I can emulate this? The 2nd logs in my previous post is when I use pause/resume in the Jotta client

jkaberg avatar May 29 '15 07:05 jkaberg

@jkaberg Yeah, I don't know if their official client would ever need to do this. As long as they transfer file by file, and reupload the whole file if it's changed, it seems we're out of luck. What we're looking for here, is some support for patching a file by pushing diffs.

Could you capture a log from the websocket chatter, maybe there is something interesting there?

havardgulldahl avatar May 29 '15 08:05 havardgulldahl

@havardgulldahl The websocket only ever gets upgraded, nothing more is happening on that part (atleast of what Fiddler can decipher, and afaik it supports WS fully)

Seems we're out of luck

jkaberg avatar May 29 '15 08:05 jkaberg

Removing milestone. I no longer think this is possible to solve, because this is my current understanding of the protocol:

The md5 checksum is how the JottaCloud servers differentiates between file revisions. So you can't keep growing a file, because with each iteration the file would have a new checksum, and thus JottaCloud would see a new file every time.

havardgulldahl avatar Sep 12 '15 21:09 havardgulldahl