More supported archive protocols

[chadmccune]

rsync does not really work well with 10gbe, It would be nice to support some additional transfer methods, ie. ftp

With rsync, I seem to cap out @ about 1 gbps, but with ftp I can easily saturate the sata drive write speed of 250 MB/s

[altendky]

We are working on roughly arbitrary configuration of the archive transfer command.

[syadnom]

Even just a move command for a mounted volume would be handle. CIFS or NFS for example.

[chadmccune]

We are working on roughly arbitrary configuration of the archive transfer command.

Any progress on this one? I'm currently plotting faster than rsync can handle :)

[syadnom]

faster than rsync can handle or faster than your network can handle or your storage drives can handle? For any individual transfer, rsync is certainly fast enough.

My issue with the current method is really that it only does one transfer at a time. I wish it was configurable like temp directories where you can say how many concurrent per and that new rsyncs round robin the destination.

This is all a balancing act. If you are on 1G ethernet, a single rsync process is really all that's reasonable considering that 1G network and the write speed of the target are probably similar enough. If you are on 10G, then a single transfer to a single drive could be the bottleneck so spreading that load out is optimal.

I'm half way through a transition to doing archiving as a separate process targetting a round-robin aufs mountpoint over 10Gbe... just waiting for a new switch to come in to finish the job.

I'll likely stick to rsync for resumability and graceful handling of destination tmp files and renames that chia farming is happy with. simple. easy. no reliance on network mount points. and I'll knock them out in parallel so I don't care if rsync can't use the whole 10Gbps in a single process.

[chadmccune]

faster than rsync can handle or faster than your network can handle or your storage drives can handle? For any individual transfer, rsync is certainly fast enough.

My issue with the current method is really that it only does one transfer at a time. I wish it was configurable like temp directories where you can say how many concurrent per and that new rsyncs round robin the destination.

This is all a balancing act. If you are on 1G ethernet, a single rsync process is really all that's reasonable considering that 1G network and the write speed of the target are probably similar enough. If you are on 10G, then a single transfer to a single drive could be the bottleneck so spreading that load out is optimal.

I'm half way through a transition to doing archiving as a separate process targetting a round-robin aufs mountpoint over 10Gbe... just waiting for a new switch to come in to finish the job.

I'll likely stick to rsync for resumability and graceful handling of destination tmp files and renames that chia farming is happy with. simple. easy. no reliance on network mount points. and I'll knock them out in parallel so I don't care if rsync can't use the whole 10Gbps in a single process.

faster than rsync, rsync apparently doesn't work well past 1Gbps. I'm plotting at about 3TB per round (which takes about 7 hours)

I'm not bottle necked anywhere except rsync. The drive's I'm rsynching to are mechanical, but when not using rsync will do sustained writes of 250 MB/s (about 3Gbps)

When I move batches manually to multiple destination drives I can saturate the 10Gbps link easily

[nqzhang]

I use the local mounted glusterfs as the dst, but the write performance of the gluster cluster is slow, so I want to write plots to a local faster disk, and then move it to gluster. Like nfs, only a simple mv command is required.

[nqzhang]

I found this pr and its really useful.Follow this progress

https://github.com/ericaltendorf/plotman/pull/88