mirror icon indicating copy to clipboard operation
mirror copied to clipboard

Support very large files

Open stephenh opened this issue 4 years ago • 1 comments

Files over ~500mb / ~1gb give mirror trouble b/c the RPC framework (grpc-java) we use for "really snappy bi-directional streaming" only supports in-memory / on-Java-heap byte[]s for its RPC messages, and doesn't support passing around zero-copy files / memory regions like netty's FileRegion.

So, for now, mirror just can't really send those sort of files unless/maybe you give it a huge heap.

We should probably detect the current size of heap, guess that we can do files ~half that size, and just ignore any file larger than that, which is basically #37.

stephenh avatar Jan 26 '20 03:01 stephenh

The best solution will be to change the transfer algorithm for such large files. For example, use rsync for delta-transfers.

Thanks

rleon avatar Jan 26 '20 09:01 rleon