pyinfra-dev
Detected changed across mulitple hosts are wrong
Describe the bug
Making an inventory containing 2 "hosts", each with its own list of items to use, makes pyinfra mix up stuff in the "Detected changes" output.
Everything does deploy correctly, even though the "Detected changes" are showing something different than what is actually being done.
In my case, I have 1 server with a 2 incus containers on it, that i can ssh into individually. That is why the inventory hosts are named as "server01/incus01" and "server01/incus02". ssh_hostname holds the actual host-name used for logging in.
I have configured a list of users, user-1 through user-5. user 1-4 are configured for incus01, and user-5 is on incus02. Yet "Detected changes" shows:
--> Detected changes:
Operation Change Conditional Change
Create user-5 user on server01/incus02 2 (server01/incus01, server01/incus02) -
Create user-2 user on server01/incus01 1 (server01/incus01) -
Create user-3 user on server01/incus01 1 (server01/incus01) -
Create user-4 user on server01/incus01 1 (server01/incus01) -
It shows "Create user-5 user on server01/incus02" for both hosts, and "user-1" is completely absent.
To Reproduce
inventory.py
hosts = [
('server01/incus01',
{'users': [
'user-1',
'user-2',
'user-3',
'user-4'
],
'ssh_hostname': 'server01.example.com',
'ssh_port': 2222,
'ssh_user': 'pyinfra+incus01'}),
('server01/incus02',
{'users': [
'user-5'
],
'ssh_hostname': 'server01.example.com',
'ssh_port': 2222,
'ssh_user': 'pyinfra+incus02'})
]
deploy_test.py
from pyinfra.operations import server
from pyinfra import host
users = host.data.get("users")
for user in users:
server.shell(
name=f"Create {user} user on {host.name}",
commands=[
f"echo {user}",
],
)
Expected behavior
Because the text is unique, due to it containing eg. username, I expected it to look like this:
--> Detected changes:
Operation Change Conditional Change
Create user-1 user on server01/incus01 1 (server01/incus01) -
Create user-2 user on server01/incus01 1 (server01/incus01) -
Create user-3 user on server01/incus01 1 (server01/incus01) -
Create user-4 user on server01/incus01 1 (server01/incus01) -
Create user-5 user on server01/incus02 1 (server01/incus02) -
Meta
pyinfra --support:
System: Linux
Platform: Linux-6.11.0-26-generic-x86_64-with-glibc2.39
Release: 6.11.0-26-generic
Machine: x86_64
pyinfra: v3.3.1
black: v25.1.0
black: v25.1.0
click: v8.1.8
distro: v1.9.0
gevent: v24.11.1
jinja2: v3.1.5
packaging: v24.2
paramiko: v3.5.0
pytest: v8.3.5
pytest: v8.3.5
python-dateutil: v2.9.0.post0
pywinrm: v0.5.0
pyyaml: v6.0.2
pyyaml: v6.0.2
setuptools: v75.8.0
typeguard: v4.4.1
typing-extensions: v4.12.2
Executable: /home/dan/development/pyinfra/venv/bin/pyinfra
Python: 3.12.3 (CPython, GCC 13.3.0)
Pyinfra was installed via pip.
debug output:
--> Loading config...
--> Loading inventory...
[pyinfra_cli.inventory] Creating fake inventory...
[pyinfra_cli.inventory] Checking possible group_data at: /home/dan/development/pyinfra/test/group_data
--> Connecting to hosts...
[pyinfra.connectors.ssh] Connecting to: server01.example.com ({'allow_agent': True, 'look_for_keys': True, '_pyinfra_ssh_forward_agent': False, '_pyinfra_ssh_config_file': None, '_pyinfra_ssh_known_hosts_file': None, '_pyinfra_ssh_strict_host_key_checking': 'accept-new', '_pyinfra_ssh_paramiko_connect_kwargs': None, 'username': 'pyinfra+incus01', 'port': 2222, 'timeout': 10})
[pyinfra.connectors.sshuserclient.client] Loading SSH config: None
[pyinfra.connectors.ssh] Connecting to: server01.example.com ({'allow_agent': True, 'look_for_keys': True, '_pyinfra_ssh_forward_agent': False, '_pyinfra_ssh_config_file': None, '_pyinfra_ssh_known_hosts_file': None, '_pyinfra_ssh_strict_host_key_checking': 'accept-new', '_pyinfra_ssh_paramiko_connect_kwargs': None, 'username': 'pyinfra+incus02', 'port': 2222, 'timeout': 10})
[server01/incus02] Connected
[server01/incus01] Connected
[pyinfra.api.state] Activating host: server01/incus01
[pyinfra.api.state] Activating host: server01/incus02
--> Preparing operation files...
Loading: deploy_test.py
[pyinfra.api.operation] Adding operation, {'Create user-5 user on server01/incus02'}, opOrder=(0, 8), opHash=e7b37f7edef2ef843cea2aa2075da5d7f922fee3
[server01/incus02] Ready: deploy_test.py
[pyinfra.api.operation] Adding operation, {'Create user-1 user on server01/incus01'}, opOrder=(0, 8), opHash=e7b37f7edef2ef843cea2aa2075da5d7f922fee3
[pyinfra.api.operation] Duplicate hash (e7b37f7edef2ef843cea2aa2075da5d7f922fee3) detected!
[pyinfra.api.operation] Adding operation, {'Create user-2 user on server01/incus01'}, opOrder=(0, 8, 1), opHash=e7b37f7edef2ef843cea2aa2075da5d7f922fee3-0
[pyinfra.api.operation] Duplicate hash (e7b37f7edef2ef843cea2aa2075da5d7f922fee3) detected!
[pyinfra.api.operation] Duplicate hash (e7b37f7edef2ef843cea2aa2075da5d7f922fee3-0) detected!
[pyinfra.api.operation] Adding operation, {'Create user-3 user on server01/incus01'}, opOrder=(0, 8, 2), opHash=e7b37f7edef2ef843cea2aa2075da5d7f922fee3-0-1
[pyinfra.api.operation] Duplicate hash (e7b37f7edef2ef843cea2aa2075da5d7f922fee3) detected!
[pyinfra.api.operation] Duplicate hash (e7b37f7edef2ef843cea2aa2075da5d7f922fee3-0) detected!
[pyinfra.api.operation] Duplicate hash (e7b37f7edef2ef843cea2aa2075da5d7f922fee3-0-1) detected!
[pyinfra.api.operation] Adding operation, {'Create user-4 user on server01/incus01'}, opOrder=(0, 8, 3), opHash=e7b37f7edef2ef843cea2aa2075da5d7f922fee3-0-1-2
[server01/incus01] Ready: deploy_test.py
--> Detected changes:
Operation Change Conditional Change
Create user-1 user on server01/incus01 2 (server01/incus01, server01/incus02) -
Create user-2 user on server01/incus01 1 (server01/incus01) -
Create user-3 user on server01/incus01 1 (server01/incus01) -
Create user-4 user on server01/incus01 1 (server01/incus01) -
Detected changes may not include every change pyinfra will execute.
Hidden side effects of operations may alter behaviour of future operations,
this will be shown in the results. The remote state will always be updated
to reflect the state defined by the input operations.
--> Disconnecting from hosts...
Thank you for the detailed issue, @dfaerch, definitely not quite right. The debug output seems to be different again - 1,2,3,4 all present but no 5.
I'm fairly sure the issue is with the loop detection to operation hashing (see the "Duplicate hash" logs), and somehow we're incorrectly displaying the changes as a result. Your reproduction is great so this shouldn't be too hard to track down!
@Fizzadar I've also noticed this when adding operations in loops. The deploy works correctly but the dry run and detected changes output can be quite incorrect.
As far as I can tell, the operation gets the same hash on both hosts despite the args being different. The duplicates within a single host work fine. Throwing in a print("OP IN HOST:", op_hash, host) after https://github.com/pyinfra-dev/pyinfra/blob/b14c5751350c1c24dafa592daffc64791d5f6de4/src/pyinfra_cli/prints.py#L247 illuminates the issue quite clearly. Using the example above (with server01/incus01 -> @docker/pyinfra1 and server01/incus02 -> @docker/pyinfra2)
--> Preparing operation files...
Loading: deploy.py
[pyinfra.api.operation] Adding operation, {'Create user-1 user on @docker/pyinfra1'}, opOrder=(0, 7), opHash=815551104ae010bf29698ef5da6d2d29fd39115c
[pyinfra.api.operation] Duplicate hash (815551104ae010bf29698ef5da6d2d29fd39115c) detected!
[pyinfra.api.operation] Adding operation, {'Create user-2 user on @docker/pyinfra1'}, opOrder=(0, 7, 1), opHash=815551104ae010bf29698ef5da6d2d29fd39115c-0
[pyinfra.api.operation] Duplicate hash (815551104ae010bf29698ef5da6d2d29fd39115c) detected!
[pyinfra.api.operation] Duplicate hash (815551104ae010bf29698ef5da6d2d29fd39115c-0) detected!
[pyinfra.api.operation] Adding operation, {'Create user-3 user on @docker/pyinfra1'}, opOrder=(0, 7, 2), opHash=815551104ae010bf29698ef5da6d2d29fd39115c-0-1
[pyinfra.api.operation] Duplicate hash (815551104ae010bf29698ef5da6d2d29fd39115c) detected!
[pyinfra.api.operation] Duplicate hash (815551104ae010bf29698ef5da6d2d29fd39115c-0) detected!
[pyinfra.api.operation] Duplicate hash (815551104ae010bf29698ef5da6d2d29fd39115c-0-1) detected!
[pyinfra.api.operation] Adding operation, {'Create user-4 user on @docker/pyinfra1'}, opOrder=(0, 7, 3), opHash=815551104ae010bf29698ef5da6d2d29fd39115c-0-1-2
[@docker/pyinfra1] Ready: deploy.py
[pyinfra.api.operation] Adding operation, {'Create user-5 user on @docker/pyinfra2'}, opOrder=(0, 7), opHash=815551104ae010bf29698ef5da6d2d29fd39115c
[@docker/pyinfra2] Ready: deploy.py
--> Detected changes:
OP IN HOST: 815551104ae010bf29698ef5da6d2d29fd39115c @docker/pyinfra1
OP IN HOST: 815551104ae010bf29698ef5da6d2d29fd39115c @docker/pyinfra2
OP IN HOST: 815551104ae010bf29698ef5da6d2d29fd39115c-0 @docker/pyinfra1
OP IN HOST: 815551104ae010bf29698ef5da6d2d29fd39115c-0-1 @docker/pyinfra1
OP IN HOST: 815551104ae010bf29698ef5da6d2d29fd39115c-0-1-2 @docker/pyinfra1
Operation Change Conditional Change
Create user-5 user on @docker/pyinfra2 2 (@docker/pyinfra1, @docker/pyinfra2) -
Create user-2 user on @docker/pyinfra1 1 (@docker/pyinfra1) -
Create user-3 user on @docker/pyinfra1 1 (@docker/pyinfra1) -
Create user-4 user on @docker/pyinfra1 1 (@docker/pyinfra1) -
The first instance of the operation for each host gets the same op hash of 815551104ae010bf29698ef5da6d2d29fd39115c and so shows up as present in state.ops[host] in both cases.
I'm not 100% clear on the implications of this suggestion, but perhaps the op hash should take the operation args into account?
