pyinfra-dev
files.replace doesn't edit my file (support extended regex)
Describe the bug
In files.replace, my regex doesn't match , so sed is never run. Also, my flag isn't passed to grep.
To Reproduce
What I ran:
files.replace(
path=path,
text=r'127.0.0.1:[0-9]+',
replace=f'127.0.0.1:{serve_port}',
flags='-E',
)
server.shell(commands=f"grep -e 127.0.0.1 {path} 1>&2")
server.shell(commands=f"grep -e 127.0.0.1 {path} | od -ah 1>&2")
Output:
--> Starting operation: files.replace (path=/home/gamer/.local/state/syncthing/config.xml, text=127.0.0.1:[0-9]+, replace=127.0.0.1:8385, flags=-E)
[plus] >>> sh -c 'grep -e '"'"'127.0.0.1:[0-9]+'"'"' /home/gamer/.local/state/syncthing/config.xml 2> /dev/null || ( find /home/gamer/.local/state/syncthing/config.xml -type f > /dev/null && echo __pyinfra_exists_/home/gamer/.local/state/syncthing/config.xml || true )'
[plus] __pyinfra_exists_/home/gamer/.local/state/syncthing/config.xml
[plus] Loaded fact files.FindInFile (interpolate_variables=False, path=/home/gamer/.local/state/syncthing/config.xml, pattern=127.0.0.1:[0-9]+)
[plus] noop: string "127.0.0.1:[0-9]+" does not exist in /home/gamer/.local/state/syncthing/config.xml
[plus] No changes
--> Starting operation: server.shell (commands=grep -e 127.0.0.1 /home/gamer/.local/state/syncthing/config.xml 1>&2)
[plus] >>> sh -c 'grep -e 127.0.0.1 /home/gamer/.local/state/syncthing/config.xml 1>&2'
[plus] <address>127.0.0.1:40161</address>
[plus] Success
--> Starting operation: server.shell (commands=grep -e 127.0.0.1 /home/gamer/.local/state/syncthing/config.xml | od -ah 1>&2)
[plus] >>> sh -c 'grep -e 127.0.0.1 /home/gamer/.local/state/syncthing/config.xml | od -ah 1>&2'
[plus] 0000000 sp sp sp sp sp sp sp sp < a d d r e s s
[plus] 2020 2020 2020 2020 613c 6464 6572 7373
[plus] 0000020 > 1 2 7 . 0 . 0 . 1 : 4 0 1 6 1
[plus] 313e 3732 302e 302e 312e 343a 3130 3136
[plus] 0000040 < / a d d r e s s > nl
[plus] 2f3c 6461 7264 7365 3e73 000a
[plus] 0000053
[plus] Success
Expected behavior
The "does not exist" is wrong; the regex 127.0.0.1:[0-9]+ does exist in that file, at that time, as shown by the two followup grep commands. Also, why isn't my -E getting passed to grep?
Meta
- Include output of
pyinfra --support.
System: Linux
Platform: Linux-6.14.0-23-generic-x86_64-with-glibc2.41
Release: 6.14.0-23-generic
Machine: x86_64
pyinfra: v3.4.1
click: v8.2.1
click: v8.2.1
click: v8.2.1
distro: v1.9.0
gevent: v25.5.1
jinja2: v3.1.6
packaging: v25.0
paramiko: v3.5.1
python-dateutil: v2.9.0.post0
pywinrm: v0.5.0
typeguard: v4.4.3
typing-extensions: v4.14.0
Executable: /home/drewp/.local/share/pdm/venvs/infra-BD9LDDRE-3.12/bin/pyinfra
Python: 3.13.3 (CPython, GCC 14.2.0)
- How was pyinfra installed (source/pip)? pdm
- Include pyinfra-debug.log (if one was created)
The workaround I used, FYI:
server.shell([f"perl -pi -e 's/127.0.0.1:[0-9]+/127.0.0.1:{serve_port}/g' {path}"])
Aha, so the extra flags only seem to be applied to the sed part, not passed to the grep fact here:
https://github.com/pyinfra-dev/pyinfra/blob/6fa182dfde25ba64e65bc895b31c2ef40bc43201/pyinfra/operations/files.py#L524-L529
Which makes this complicated because the flags argument is specific to sed not grep, so we might need a second grep_flags argument. Perhaps a better solution is to add a extended_regex=False argument that adds the -E flag to both grep and sed, since they have the same meaning.
EDIT: So in your case @drewp I think your workaround is the only current solution, (extended) regex in files.replace is not currently supported.
I agree extended_regex is the better option. What about s/foo/bar/g (the global flag)? It needs no support on the grep pass, but it's useful and hard to workaround for the replacement pass.