Ansible copy module (remote target) extreme performance decrease

[el-pako]

I am using the Ansible copy module to copy a file (3gb) to a remote target in my LAN. This takes roughly 0:30 without and 8:30 with Mitogen enabled.

- name: Test copy
  copy:
    src: /src/a.file
    dest: /dest

Mitogen version: Git master 03 march 2023 Remote Machine: RHEL 8.5 / Python 3.6.8 Ansible Controller: RHEL 8.6 / Python 3.6.8

ansible [core 2.11.12] 
  config file = None
  configured module search path = ['/home/ehtest/.ansible/plugins/modules', '/usr/share/ansible/plugins/modules']
  ansible python module location = /home/ehtest/.local/lib/python3.6/site-packages/ansible
  ansible collection location = /home/ehtest/.ansible/collections:/usr/share/ansible/collections
  executable location = /home/ehtest/.local/bin/ansible
  python version = 3.6.8 (default, Jan 14 2022, 11:04:20) [GCC 8.5.0 20210514 (Red Hat 8.5.0-7)]
  jinja version = 3.0.3
  libyaml = True

[moreati]

I can't reproduce such an extreme slowdown, merely a 2.5 x. Copying macOS 13/arm64/python 3.6.15 -> Ubuntu 22.04/amd64/python 3.6.15, 1 GB of /dev/random, over WiFi.

Some clarifying questions

1. Can you confirm the OS/Python combination for controller and target? Context: I'm surprised at the combination RHEL 8.x, Python 3.6.8. Going by distrowatch RHEL 8.x ships Python 3.9. It was RHEL 7.x that shipped with 3.6.
2. What is the raw network speed? (e.g. 1 Gbit ethernet)
3. How much does the file you are compying compress? (e.g. 40% original size with gzip)
4. Does your result change with current master (rev ec212a10d8490db582b217ecac1f86344c56347e)

For posterity, this was my reproduction attempt.

copy_1_GiB.yml

- name: Test copy
  hosts: bertha.local
  gather_facts: false
  tasks:
    - file: {path: ~/1_GiB, state: absent}
    - copy: {src: 1_GiB, dest: ~/1_GiB, mode: u=rw}

tox.ini

[tox]
envlist =
    py{27,36}-tgt{27,36}-{ansible,mitogen}
requires =
    virtualenv<20.22.0

[testenv]
basepython =
    py27: python2.7
    py36: python3.6
deps =
    py27: ansible~=4.0
    py36: ansible~=4.0
install_command =
    python -m pip --no-python-version-warning --disable-pip-version-check install {opts} {packages}
commands =
    tgt27: ansible-playbook -i inv.yml -e ansible_python_interpreter=python2.7 copy_1_GiB.yml
    tgt36: ansible-playbook -i inv.yml -e ansible_python_interpreter=python3.6 copy_1_GiB.yml
setenv =
    ANSIBLE_PIPELINING = true
    ANSIBLE_STRATEGY_PLUGINS = ../ansible_mitogen/plugins/strategy
    ansible: ANSIBLE_STRATEGY = linear
    mitogen: ANSIBLE_STRATEGY = mitogen_linear

$ dd if=/dev/random of=1_GiB bs=1m count=1024
$ tox
...
  py27-tgt27-ansible: OK (45.26=setup[0.03]+cmd[45.23] seconds)
  py27-tgt27-mitogen: OK (40.15=setup[0.01]+cmd[40.15] seconds)
  py27-tgt36-ansible: OK (38.80=setup[0.01]+cmd[38.79] seconds)
  py27-tgt36-mitogen: OK (36.27=setup[0.01]+cmd[36.27] seconds)
  py36-tgt27-ansible: OK (44.14=setup[0.01]+cmd[44.14] seconds)
  py36-tgt27-mitogen: OK (101.77=setup[0.01]+cmd[101.77] seconds)
  py36-tgt36-ansible: OK (39.63=setup[0.01]+cmd[39.63] seconds)
  py36-tgt36-mitogen: OK (97.18=setup[0.00]+cmd[97.18] seconds)
  congratulations :) (443.25 seconds)

$ time (ssh bertha.local rm 1_GiB; scp 1_GiB bertha.local:)
1_GiB                                                                           100% 1024MB  29.1MB/s   00:35    
( ssh bertha.local rm 1_GiB; scp 1_GiB bertha.local:; )  3.42s user 1.12s system 12% cpu 36.373 total

[moreati]

@el-pako as a short term measure using https://github.com/moreati/mitogen/commit/2d1fff23e31b7c77532147260c6e48d9e66ef468 should improve your 3.x -> 3.x copy speed. It brought vanilla Ansible and Ansible+Mitogen to a similar speed in my test playbook.

$ tox -e py36-tgt36-ansible,py36-tgt36-mitogen
...
  py36-tgt36-ansible: OK (37.64=setup[0.02]+cmd[37.62] seconds)
  py36-tgt36-mitogen: OK (32.96=setup[0.00]+cmd[32.95] seconds)

The longer term saga is in #485.