Synchronize module cannot use SSH multiplexing if ControlPath is autogenerated

[eszense]

SUMMARY

• If ControlMaster and ControlPersist is specified in ssh_args of ansible.cfg, the omitted ControlPath will be autogenerated by ansible with ~/.ansible/cp/<digest of user-host-port> as per the ssh connection plugin
• This autogenerated ControlPath is not picked up by the synchronize when ssh_connection_multiplexing and use_ssh_args is enabled, leading to failure in multiplexing
• This is not only a performance issue. In my case I use Yubikey for SSH authentication, and relies on multiplexing on a single SSH connection to remains authenticated (Can't use ssh-agent). Failure in multiplexing will lead to me having to touch the Yubikey for each invocation of synchronize task.

ISSUE TYPE

• Bug Report

COMPONENT NAME

synchronize

ANSIBLE VERSION

  python version = 3.9.7 (v3.9.7:1016ef3790, Aug 30 2021, 16:39:15) [Clang 6.0 (clang-600.0.57)]
  jinja version = 3.0.3
  libyaml = True

COLLECTION VERSION

Collection                    Version
----------------------------- -------
ansible.posix                 1.3.0  

OS / ENVIRONMENT

Controller: MacOS Big Sur Target: Ubuntu Impish

STEPS TO REPRODUCE

• Disable your ssh-agent
• Add ssh_args in ansible.cfg

ssh_args = -o ControlMaster=auto -o ControlPersist=1200

• Run this playbook

---
- hosts: MY_HOST
  tasks:
    - command: echo This task used multiplexing and no reauthentication needed
    - synchronize: # This failed multiplexing and need reauthentication
        src: ~/tmp
        dest: ~/tmp

• Connect to a host

EXPECTED RESULTS

When ssh_connection_multiplexing and use_ssh_args are enabled, synchronize module is expected to reuse all ssh_args, especially the implicit ControlPath

ACTUAL RESULTS

Implicit ControlPath not reused

[saito-hideki]

@eszense thank you for reporting the issue. I have confirmed this behavior in my test environment using the following ansible.cfg and test playbook:

• Ansible Core version: 2.13.0.dev0
• ansible.cfg

[defaults]
host_key_checking = False
collections_paths = ./collections

[ssh_connection]
ssh_args = -o ControlMaster=auto -o ControlPersist=1200

• playbook

---
- hosts: test
  gather_facts: false

  tasks:
    - name: TASK_A
      ansible.builtin.ping:

    - name: TASK_B
      ansible.builtin.file:
        path: /tmp/test_data.txt
        state: absent

    - name: TASK_C
      ansible.posix.synchronize:
        src: test_data.txt
        dest: /tmp/test_data.txt
        ssh_connection_multiplexing: yes
        use_ssh_args: yes

Looking at the following output from the ansible-playbook command, the ControlPath="/Users/hsaito/.ansible/cp/5450f158dd" was reused in TASK_A and TASK_B. However, it didn't use in TASK_C:

(venv) devel/ansible » ansible-playbook -i inventory/issue_317 playbook/issue_317/main.yml -vvv

...snip...

TASK [TASK_A] ****************************************************************************
task path: /Users/hsaito/work/devel/ansible/playbook/issue_317/main.yml:9
...snip...
<server10> SSH: EXEC ssh -o ControlMaster=auto -o ControlPersist=1200 -o StrictHostKeyChecking=no -o KbdInteractiveAuthentication=no -o PreferredAuthentications=gssapi-with-mic,gssapi-keyex,hostbased,publickey -o PasswordAuthentication=no -o 'User="redhat"' -o ConnectTimeout=10 -o 'ControlPath="/Users/hsaito/.ansible/cp/5450f158dd"' server10 '/bin/sh -c '"'"'echo ~redhat && sleep 0'"'"''
...snip...

TASK [TASK_B] ****************************************************************************
...snip...
<server10> SSH: EXEC ssh -o ControlMaster=auto -o ControlPersist=1200 -o StrictHostKeyChecking=no -o KbdInteractiveAuthentication=no -o PreferredAuthentications=gssapi-with-mic,gssapi-keyex,hostbased,publickey -o PasswordAuthentication=no -o 'User="redhat"' -o ConnectTimeout=10 -o 'ControlPath="/Users/hsaito/.ansible/cp/5450f158dd"' server10 '/bin/sh -c '"'"'( umask 77 && mkdir -p "` echo /home/redhat/.ansible/tmp `"&& mkdir "` echo /home/redhat/.ansible/tmp/ansible-tmp-1646398060.406553-18289-171556998362201 `" && echo ansible-tmp-1646398060.406553-18289-171556998362201="` echo /home/redhat/.ansible/tmp/ansible-tmp-1646398060.406553-18289-171556998362201 `" ) && sleep 0'"'"''
...snip...

TASK [TASK_C] ****************************************************************************
...snip...
changed: [server10] => {
    "changed": true,
    "cmd": "/usr/bin/rsync --delay-updates -F --compress --archive --rsh='/usr/bin/ssh -o StrictHostKeyChecking=no -o UserKnownHostsFile=/dev/null -o ControlMaster=auto -o ControlPersist=1200' --out-format='<<CHANGED>>%i %n%L' /Users/hsaito/work/devel/ansible/playbook/issue_317/files/test_data.txt redhat@server10:/tmp/test_data.txt",
...snip...
}

I will triage this issue as a bug. Please let me know if I have misunderstood your situation.

[toadjaune]

Hi, I've run into this myself, and attempted a fix. Here are my findings :

here we're extracting the existing ssh args, but not the control_persist one.

I therefore attempted to recover this value from the connection object, but the corresponding logic is really non-trivial (depending on whether control_path is defined or not, whether control_path_dir is defined or not, etc).

If I understand this code correctly, there's currently no way to extract this specific argument from the connection object (it's only calculated as intermediate values inside _build_command). Duplicating this entire logic sounds like a very bad idea, so I'm thinking we should probably do something along the lines of :

• refactoring the ssh action plugin to extract this calculation in a function callable from synchronize
• refactoring the ssh action plugin to calculate and store this calculation in an attribute readable from synchronize

Which would also require coordination between the ansible base repo and this collection for releasing.

As you can see, this issue calls for someone with more knowledge of the general ansible architecture. I hope my debugging turns up useful !

As a side note, this issue also affects situations where control_path is manually defined in ansible.cfg, not just situations where it is to its default value.