scp.py

scp.py copied to clipboard

Hello! Thank you for this module. I noticed the transfer time can be quite slow (compared to command line scp). An order of magnitude or more slower. Is this an issue of the scp module or perhaps of the underlying paramiko/ssh settings?

Are you talking about a single file? Multiple small files?

Have this ever been benchmarked?

single file around 30 MBs.

Probably a paramiko issue 😢

Have you tried specifying a different buff_size?

def get_ssh_connection():
    host = data_server
    ssh = SSHClient()
    ssh.set_missing_host_key_policy(paramiko.AutoAddPolicy())
    ssh.load_system_host_keys()
    try:
        ssh.connect(data_server, username="root")
    except:
        password = getpass.getpass("SSH password for {}@{} : ".format("root", data_server))
        ssh.connect(data_server, username="root", password=password)
    return ssh


def scp_to_server(local_filenames, remote_directory, skip_if_exists=True):
    ssh_cli = get_ssh_connection()
    scp_cli = scp.SCPClient(ssh_cli.get_transport())

Ok, where would I add the bufsize? Do I need a channel object and if so, how do I give to the SCPClient? (just in case it's convenient for you to answer, I will look at it later myself also)

Last line:

scp_cli = scp.SCPClient(ssh_cli.get_transport(), buff_size=2*1024*1024)

Thank you! I will report back when I benchmark it, because I think more people may find it useful.

Bad news: I didn't perceive a difference one way or the other.

@carolinux are you comparing transfer speed of scp without compression to your code? I made the mistake of not setting client.connect(compress=True) so my transfers were considerably slower compared to my normal usage of scp -C

any update on this, am having a similar issue

(Oops)

I can't reproduce the issue, so it's pretty hard. It shouldn't be much slower, and if it is, that's probably an issue with Paramiko. But we can run more tests.

I ran some tests and I really don't see much of a slowdown.

NYC MacBook to GCE: 14.605 / 14.248, French server to GCE: 13.114, 13.294

Data

MacBook (New York) -> Google Cloud (us-central1-c): 100MB

/usr/bin/scp	scp.py
14.563	15.142392
14.459	14.081794
14.84	14.152773
14.362	14.174169
15.175	14.07147
14.465	14.245316
14.84	14.053202
14.464	14.059526
14.497	13.943941
14.453	14.419479
14.662	14.142728
14.505	13.978535
14.544	14.187512
14.383	14.549694
14.865	14.518703

Online.net server (France) -> Google Cloud (us-central1-c): 15MB

/usr/bin/scp	scp.py
4.356	11.241386
10.042	23.640888
8.805	15.00032
5.784	4.558009
6.611	18.062356
23.206	32.331319
30.934	28.441495
16.557	23.172128
28.782	22.457937
37.943	18.350743
3.14	2.257147
3.04	2.339132
3.136	2.088671
3.041	2.080863
7.649	8.392694
10.077	12.570131
12.095	11.449674
11.217	9.9099
14.234	10.832424
18.752	14.649188
17.175	9.148867
13.618	9.986314
10.291	15.070032
14.25	11.0324

If you see a really significant difference, do you mind sharing a bit more on your setup? Which OS? Is the machine CPU bound? High latency? Are you running on a very fast link?