pyinfra
                                
                                
                                
                                    pyinfra copied to clipboard
                            
                            
                            
                        Exception raised trying to collect facts for file
Describe the bug
When attempting to execute the files.file or files.directory operations defined here an exception is raised while attempting to execute the file fact.
This error is new as of v1.4.5 and v1.4.6. Previous versions work without error.
To Reproduce
Clone the repository at github.com/mitodl/ol-infrastructure and create a deploy file that contains:
from bilder.components.hashicorp.steps import install_hashicorp_products
from bilder.components.hashicorp.vault.models import Vault
install_hashicorp_products(Vault())
Expected behavior
The expected behavior is to download and unpack the Vault binary and then set the permissions and create a configuration directory. Instead it generates a traceback.
Meta
- Include output of 
pyinfra --support. 
--> Support information:
    If you are having issues with pyinfra or wish to make feature requests, please
    check out the GitHub issues at https://github.com/Fizzadar/pyinfra/issues .
    When adding an issue, be sure to include the following:
    System: Linux
      Platform: Linux-5.12.13-1-MANJARO-x86_64-with-glibc2.32
      Release: 5.12.13-1-MANJARO
      Machine: x86_64
    pyinfra: v1.4.6
    Executable: /home/tmacey/code/mit/ops/infra/ol-infrastructure/.venv/bin/pyinfra
    Python: 3.8.6 (CPython, GCC 10.2.0)
- How was pyinfra installed (source/pip)? pip
 - Include pyinfra-debug.log (if one was created)
 
  File "/home/tmacey/code/mit/ops/infra/ol-infrastructure/.venv/lib/python3.8/site-packages/pyinfra_cli/main.py", line 218, in cli
    _main(*args, **kwargs)
  File "/home/tmacey/code/mit/ops/infra/ol-infrastructure/.venv/lib/python3.8/site-packages/pyinfra_cli/main.py", line 604, in _main
    run_ops(state, serial=serial, no_wait=no_wait)
  File "/home/tmacey/code/mit/ops/infra/ol-infrastructure/.venv/lib/python3.8/site-packages/pyinfra/api/operations.py", line 375, in run_ops
    _run_single_op(state, op_hash)
  File "/home/tmacey/code/mit/ops/infra/ol-infrastructure/.venv/lib/python3.8/site-packages/pyinfra/api/operations.py", line 339, in _run_single_op
    if not greenlet.get():
  File "src/gevent/greenlet.py", line 803, in gevent._gevent_cgreenlet.Greenlet.get
  File "src/gevent/greenlet.py", line 371, in gevent._gevent_cgreenlet.Greenlet._raise_exception
  File "/home/tmacey/code/mit/ops/infra/ol-infrastructure/.venv/lib/python3.8/site-packages/gevent/_compat.py", line 65, in reraise
    raise value.with_traceback(tb)
  File "src/gevent/greenlet.py", line 906, in gevent._gevent_cgreenlet.Greenlet.run
  File "/home/tmacey/code/mit/ops/infra/ol-infrastructure/.venv/lib/python3.8/site-packages/pyinfra/api/operations.py", line 134, in _run_server_op
    status = _run_shell_command(state, host, command, global_kwargs, executor_kwargs)
  File "/home/tmacey/code/mit/ops/infra/ol-infrastructure/.venv/lib/python3.8/site-packages/pyinfra/api/operations.py", line 48, in _run_shell_command
    status, combined_output_lines = command.execute(state, host, executor_kwargs)
  File "/home/tmacey/code/mit/ops/infra/ol-infrastructure/.venv/lib/python3.8/site-packages/pyinfra/api/command.py", line 104, in execute
    return host.run_shell_command(
  File "/home/tmacey/code/mit/ops/infra/ol-infrastructure/.venv/lib/python3.8/site-packages/pyinfra/api/host.py", line 200, in run_shell_command
    return self.executor.run_shell_command(self.state, self, *args, **kwargs)
  File "/home/tmacey/code/mit/ops/infra/ol-infrastructure/.venv/lib/python3.8/site-packages/pyinfra/api/connectors/ssh.py", line 277, in run_shell_command
    actual_command = command.get_raw_value()
  File "/home/tmacey/code/mit/ops/infra/ol-infrastructure/.venv/lib/python3.8/site-packages/pyinfra/api/command.py", line 91, in get_raw_value
    return self.separator.join(self._get_all_bits(
  File "/home/tmacey/code/mit/ops/infra/ol-infrastructure/.venv/lib/python3.8/site-packages/pyinfra/api/command.py", line 81, in _get_all_bits
    bit = bit_accessor(bit)
  File "/home/tmacey/code/mit/ops/infra/ol-infrastructure/.venv/lib/python3.8/site-packages/pyinfra/api/command.py", line 92, in <lambda>
    lambda bit: bit.get_raw_value(),
  File "/home/tmacey/code/mit/ops/infra/ol-infrastructure/.venv/lib/python3.8/site-packages/pyinfra/api/command.py", line 91, in get_raw_value
    return self.separator.join(self._get_all_bits(
  File "/home/tmacey/code/mit/ops/infra/ol-infrastructure/.venv/lib/python3.8/site-packages/pyinfra/api/command.py", line 84, in _get_all_bits
    bit = shlex_quote(bit)
  File "/home/tmacey/.pyenv/versions/3.8.6/lib/python3.8/shlex.py", line 325, in quote
    if _find_unsafe(s) is None:
TypeError: expected string or bytes-like object
- Consider including output with 
-vvand--debug. 
    Loaded fact sha256_file (path=/tmp/vault.zip)
    [3.88.169.89] noop: file /tmp/vault.zip has already been downloaded
    [pyinfra.api.operation] Adding operation, {'Install Hashicorp Products | Unzip vault'}, opOrder=(124, 65), opHash=7b609db36deb56f0c3e4ce53d1a2453fdaa7b6de
    [pyinfra.api.operation] Adding operation, {'Install Hashicorp Products | Ensure vault binary is executable'}, opOrder=(124, 71), opHash=3293f92ef79521f2b72f9af8c9dffa6116146892
    [pyinfra.api.facts] Getting fact: file (path=/usr/local/bin/vault) (ensure_hosts: (Host(3.88.169.89),))
Traceback (most recent call last):
  File "src/gevent/greenlet.py", line 906, in gevent._gevent_cgreenlet.Greenlet.run
  File "/home/tmacey/code/mit/ops/infra/ol-infrastructure/.venv/lib/python3.8/site-packages/pyinfra/api/host.py", line 200, in run_shell_command
    return self.executor.run_shell_command(self.state, self, *args, **kwargs)
  File "/home/tmacey/code/mit/ops/infra/ol-infrastructure/.venv/lib/python3.8/site-packages/pyinfra/api/connectors/ssh.py", line 277, in run_shell_command
    actual_command = command.get_raw_value()
  File "/home/tmacey/code/mit/ops/infra/ol-infrastructure/.venv/lib/python3.8/site-packages/pyinfra/api/command.py", line 102, in get_raw_value
    return self.separator.join(self._get_all_bits(
  File "/home/tmacey/code/mit/ops/infra/ol-infrastructure/.venv/lib/python3.8/site-packages/pyinfra/api/command.py", line 92, in _get_all_bits
    bit = bit_accessor(bit)
  File "/home/tmacey/code/mit/ops/infra/ol-infrastructure/.venv/lib/python3.8/site-packages/pyinfra/api/command.py", line 103, in <lambda>
    lambda bit: bit.get_raw_value(),
  File "/home/tmacey/code/mit/ops/infra/ol-infrastructure/.venv/lib/python3.8/site-packages/pyinfra/api/command.py", line 102, in get_raw_value
    return self.separator.join(self._get_all_bits(
  File "/home/tmacey/code/mit/ops/infra/ol-infrastructure/.venv/lib/python3.8/site-packages/pyinfra/api/command.py", line 95, in _get_all_bits
    bit = shlex_quote(bit)
  File "/home/tmacey/.pyenv/versions/3.8.6/lib/python3.8/shlex.py", line 325, in quote
    if _find_unsafe(s) is None:
TypeError: expected string or bytes-like object
2021-06-28T21:04:59Z Traceback (most recent call last):
  File "src/gevent/greenlet.py", line 906, in gevent._gevent_cgreenlet.Greenlet.run
  File "/home/tmacey/code/mit/ops/infra/ol-infrastructure/.venv/lib/python3.8/site-packages/pyinfra/api/host.py", line 200, in run_shell_command
    return self.executor.run_shell_command(self.state, self, *args, **kwargs)
  File "/home/tmacey/code/mit/ops/infra/ol-infrastructure/.venv/lib/python3.8/site-packages/pyinfra/api/connectors/ssh.py", line 277, in run_shell_command
    actual_command = command.get_raw_value()
  File "/home/tmacey/code/mit/ops/infra/ol-infrastructure/.venv/lib/python3.8/site-packages/pyinfra/api/command.py", line 102, in get_raw_value
    return self.separator.join(self._get_all_bits(
  File "/home/tmacey/code/mit/ops/infra/ol-infrastructure/.venv/lib/python3.8/site-packages/pyinfra/api/command.py", line 92, in _get_all_bits
    bit = bit_accessor(bit)
  File "/home/tmacey/code/mit/ops/infra/ol-infrastructure/.venv/lib/python3.8/site-packages/pyinfra/api/command.py", line 103, in <lambda>
    lambda bit: bit.get_raw_value(),
  File "/home/tmacey/code/mit/ops/infra/ol-infrastructure/.venv/lib/python3.8/site-packages/pyinfra/api/command.py", line 102, in get_raw_value
    return self.separator.join(self._get_all_bits(
  File "/home/tmacey/code/mit/ops/infra/ol-infrastructure/.venv/lib/python3.8/site-packages/pyinfra/api/command.py", line 95, in _get_all_bits
    bit = shlex_quote(bit)
  File "/home/tmacey/.pyenv/versions/3.8.6/lib/python3.8/shlex.py", line 325, in quote
    if _find_unsafe(s) is None:
TypeError: expected string or bytes-like object
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
  File "/home/tmacey/code/mit/ops/infra/ol-infrastructure/.venv/lib/python3.8/site-packages/gevent/hub.py", line 624, in print_exception
    context = self.format_context(context)
  File "/home/tmacey/.pyenv/versions/3.8.6/lib/python3.8/pprint.py", line 67, in saferepr
    return _safe_repr(object, {}, None, 0, True)[0]
  File "/home/tmacey/.pyenv/versions/3.8.6/lib/python3.8/pprint.py", line 569, in _safe_repr
    rep = repr(object)
  File "src/gevent/greenlet.py", line 533, in gevent._gevent_cgreenlet.Greenlet.__repr__
  File "src/gevent/greenlet.py", line 539, in gevent._gevent_cgreenlet.Greenlet._formatinfo
  File "src/gevent/greenlet.py", line 539, in gevent._gevent_cgreenlet.Greenlet._formatinfo
  File "src/gevent/greenlet.py", line 558, in gevent._gevent_cgreenlet.Greenlet._formatinfo
  File "/home/tmacey/code/mit/ops/infra/ol-infrastructure/.venv/lib/python3.8/site-packages/pyinfra/api/command.py", line 80, in __repr__
    return 'StringCommand({0})'.format(self.get_masked_value())
  File "/home/tmacey/code/mit/ops/infra/ol-infrastructure/.venv/lib/python3.8/site-packages/pyinfra/api/command.py", line 109, in get_masked_value
    for bit in self._get_all_bits(lambda bit: bit.get_masked_value())
  File "/home/tmacey/code/mit/ops/infra/ol-infrastructure/.venv/lib/python3.8/site-packages/pyinfra/api/command.py", line 95, in _get_all_bits
    bit = shlex_quote(bit)
  File "/home/tmacey/.pyenv/versions/3.8.6/lib/python3.8/shlex.py", line 325, in quote
    if _find_unsafe(s) is None:
TypeError: expected string or bytes-like object
Traceback (most recent call last):
  File "src/gevent/greenlet.py", line 906, in gevent._gevent_cgreenlet.Greenlet.run
  File "/home/tmacey/code/mit/ops/infra/ol-infrastructure/.venv/lib/python3.8/site-packages/pyinfra/api/host.py", line 200, in run_shell_command
    return self.executor.run_shell_command(self.state, self, *args, **kwargs)
  File "/home/tmacey/code/mit/ops/infra/ol-infrastructure/.venv/lib/python3.8/site-packages/pyinfra/api/connectors/ssh.py", line 277, in run_shell_command
    actual_command = command.get_raw_value()
  File "/home/tmacey/code/mit/ops/infra/ol-infrastructure/.venv/lib/python3.8/site-packages/pyinfra/api/command.py", line 102, in get_raw_value
    return self.separator.join(self._get_all_bits(
  File "/home/tmacey/code/mit/ops/infra/ol-infrastructure/.venv/lib/python3.8/site-packages/pyinfra/api/command.py", line 92, in _get_all_bits
    bit = bit_accessor(bit)
  File "/home/tmacey/code/mit/ops/infra/ol-infrastructure/.venv/lib/python3.8/site-packages/pyinfra/api/command.py", line 103, in <lambda>
    lambda bit: bit.get_raw_value(),
  File "/home/tmacey/code/mit/ops/infra/ol-infrastructure/.venv/lib/python3.8/site-packages/pyinfra/api/command.py", line 102, in get_raw_value
    return self.separator.join(self._get_all_bits(
  File "/home/tmacey/code/mit/ops/infra/ol-infrastructure/.venv/lib/python3.8/site-packages/pyinfra/api/command.py", line 95, in _get_all_bits
    bit = shlex_quote(bit)
  File "/home/tmacey/.pyenv/versions/3.8.6/lib/python3.8/shlex.py", line 325, in quote
    if _find_unsafe(s) is None:
TypeError: expected string or bytes-like object
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
  File "/home/tmacey/code/mit/ops/infra/ol-infrastructure/.venv/lib/python3.8/site-packages/gevent/hub.py", line 624, in print_exception
    context = self.format_context(context)
  File "/home/tmacey/.pyenv/versions/3.8.6/lib/python3.8/pprint.py", line 67, in saferepr
    return _safe_repr(object, {}, None, 0, True)[0]
  File "/home/tmacey/.pyenv/versions/3.8.6/lib/python3.8/pprint.py", line 569, in _safe_repr
    rep = repr(object)
  File "src/gevent/greenlet.py", line 533, in gevent._gevent_cgreenlet.Greenlet.__repr__
  File "src/gevent/greenlet.py", line 539, in gevent._gevent_cgreenlet.Greenlet._formatinfo
  File "src/gevent/greenlet.py", line 539, in gevent._gevent_cgreenlet.Greenlet._formatinfo
  File "src/gevent/greenlet.py", line 558, in gevent._gevent_cgreenlet.Greenlet._formatinfo
  File "/home/tmacey/code/mit/ops/infra/ol-infrastructure/.venv/lib/python3.8/site-packages/pyinfra/api/command.py", line 80, in __repr__
    return 'StringCommand({0})'.format(self.get_masked_value())
  File "/home/tmacey/code/mit/ops/infra/ol-infrastructure/.venv/lib/python3.8/site-packages/pyinfra/api/command.py", line 109, in get_masked_value
    for bit in self._get_all_bits(lambda bit: bit.get_masked_value())
  File "/home/tmacey/code/mit/ops/infra/ol-infrastructure/.venv/lib/python3.8/site-packages/pyinfra/api/command.py", line 95, in _get_all_bits
    bit = shlex_quote(bit)
  File "/home/tmacey/.pyenv/versions/3.8.6/lib/python3.8/shlex.py", line 325, in quote
    if _find_unsafe(s) is None:
TypeError: expected string or bytes-like object
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
  File "src/gevent/greenlet.py", line 908, in gevent._gevent_cgreenlet.Greenlet.run
  File "src/gevent/greenlet.py", line 896, in gevent._gevent_cgreenlet.Greenlet._Greenlet__report_error
  File "/home/tmacey/code/mit/ops/infra/ol-infrastructure/.venv/lib/python3.8/site-packages/gevent/hub.py", line 541, in handle_error
    self.print_exception(context, type, value, tb)
  File "/home/tmacey/code/mit/ops/infra/ol-infrastructure/.venv/lib/python3.8/site-packages/gevent/hub.py", line 627, in print_exception
    context = repr(context)
  File "src/gevent/greenlet.py", line 533, in gevent._gevent_cgreenlet.Greenlet.__repr__
  File "src/gevent/greenlet.py", line 539, in gevent._gevent_cgreenlet.Greenlet._formatinfo
  File "src/gevent/greenlet.py", line 539, in gevent._gevent_cgreenlet.Greenlet._formatinfo
  File "src/gevent/greenlet.py", line 558, in gevent._gevent_cgreenlet.Greenlet._formatinfo
  File "/home/tmacey/code/mit/ops/infra/ol-infrastructure/.venv/lib/python3.8/site-packages/pyinfra/api/command.py", line 80, in __repr__
    return 'StringCommand({0})'.format(self.get_masked_value())
  File "/home/tmacey/code/mit/ops/infra/ol-infrastructure/.venv/lib/python3.8/site-packages/pyinfra/api/command.py", line 109, in get_masked_value
    for bit in self._get_all_bits(lambda bit: bit.get_masked_value())
  File "/home/tmacey/code/mit/ops/infra/ol-infrastructure/.venv/lib/python3.8/site-packages/pyinfra/api/command.py", line 95, in _get_all_bits
    bit = shlex_quote(bit)
  File "/home/tmacey/.pyenv/versions/3.8.6/lib/python3.8/shlex.py", line 325, in quote
    if _find_unsafe(s) is None:
TypeError: expected string or bytes-like object
2021-06-28T21:04:59Z <callback at 0x7fcfd9ddd5c0 stopped> failed with TypeError
--> An unexpected exception occurred in: src/bilder/images/edxapp/deploy.py:
  File "/home/tmacey/code/mit/ops/infra/ol-infrastructure/.venv/lib/python3.8/site-packages/pyinfra_cli/util.py", line 79, in exec_file
    exec(PYTHON_CODES[filename], data)
  File "src/bilder/images/edxapp/deploy.py", line 124, in <module>
    install_hashicorp_products(hashicorp_products)
  File "/home/tmacey/code/mit/ops/infra/ol-infrastructure/.venv/lib/python3.8/site-packages/pyinfra/api/deploy.py", line 142, in decorated_func
    func(*args, **kwargs)
  File "/home/tmacey/code/mit/ops/infra/ol-infrastructure/src/bilder/components/hashicorp/steps.py", line 71, in install_hashicorp_products
    files.file(
  File "/home/tmacey/code/mit/ops/infra/ol-infrastructure/.venv/lib/python3.8/site-packages/pyinfra/api/operation.py", line 365, in decorated_func
    commands = unroll_generators(func(*actual_args, **actual_kwargs))
  File "/home/tmacey/code/mit/ops/infra/ol-infrastructure/.venv/lib/python3.8/site-packages/pyinfra/api/util.py", line 192, in unroll_generators
    for item in generator:
  File "/home/tmacey/code/mit/ops/infra/ol-infrastructure/.venv/lib/python3.8/site-packages/pyinfra/operations/files.py", line 1075, in file
    info = host.get_fact(File, path=path)
  File "/home/tmacey/code/mit/ops/infra/ol-infrastructure/.venv/lib/python3.8/site-packages/pyinfra/api/host.py", line 138, in get_fact
    return get_host_fact(self.state, self, name_or_cls, args=args, kwargs=kwargs)
  File "/home/tmacey/code/mit/ops/infra/ol-infrastructure/.venv/lib/python3.8/site-packages/pyinfra/api/facts.py", line 336, in get_host_fact
    fact_data = get_facts(state, name, args=args, kwargs=kwargs, ensure_hosts=(host,))
  File "/home/tmacey/code/mit/ops/infra/ol-infrastructure/.venv/lib/python3.8/site-packages/pyinfra/api/facts.py", line 276, in get_facts
    status, combined_output_lines = greenlet.get()
  File "src/gevent/greenlet.py", line 803, in gevent._gevent_cgreenlet.Greenlet.get
  File "src/gevent/greenlet.py", line 371, in gevent._gevent_cgreenlet.Greenlet._raise_exception
  File "/home/tmacey/code/mit/ops/infra/ol-infrastructure/.venv/lib/python3.8/site-packages/gevent/_compat.py", line 65, in reraise
    raise value.with_traceback(tb)
  File "src/gevent/greenlet.py", line 906, in gevent._gevent_cgreenlet.Greenlet.run
  File "/home/tmacey/code/mit/ops/infra/ol-infrastructure/.venv/lib/python3.8/site-packages/pyinfra/api/host.py", line 200, in run_shell_command
    return self.executor.run_shell_command(self.state, self, *args, **kwargs)
  File "/home/tmacey/code/mit/ops/infra/ol-infrastructure/.venv/lib/python3.8/site-packages/pyinfra/api/connectors/ssh.py", line 277, in run_shell_command
    actual_command = command.get_raw_value()
  File "/home/tmacey/code/mit/ops/infra/ol-infrastructure/.venv/lib/python3.8/site-packages/pyinfra/api/command.py", line 102, in get_raw_value
    return self.separator.join(self._get_all_bits(
  File "/home/tmacey/code/mit/ops/infra/ol-infrastructure/.venv/lib/python3.8/site-packages/pyinfra/api/command.py", line 92, in _get_all_bits
    bit = bit_accessor(bit)
  File "/home/tmacey/code/mit/ops/infra/ol-infrastructure/.venv/lib/python3.8/site-packages/pyinfra/api/command.py", line 103, in <lambda>
    lambda bit: bit.get_raw_value(),
  File "/home/tmacey/code/mit/ops/infra/ol-infrastructure/.venv/lib/python3.8/site-packages/pyinfra/api/command.py", line 102, in get_raw_value
    return self.separator.join(self._get_all_bits(
  File "/home/tmacey/code/mit/ops/infra/ol-infrastructure/.venv/lib/python3.8/site-packages/pyinfra/api/command.py", line 95, in _get_all_bits
    bit = shlex_quote(bit)
  File "/home/tmacey/.pyenv/versions/3.8.6/lib/python3.8/shlex.py", line 325, in quote
    if _find_unsafe(s) is None:
TypeError: expected string or bytes-like object
                                    
                                    
                                    
                                
🤦 had not tested the operations with Path objects. I've added tests for non-string StringCommand bits in https://github.com/Fizzadar/pyinfra/commit/2e4053f28f278e873499f6b716f20e4900dbcd1c and handled them in https://github.com/Fizzadar/pyinfra/commit/b91281e9e00106e4c1860b0209aa3bd8abf0ee19. I have just released this in v1.4.7 which should fix this issue!
Thanks for the quick turnaround!
After testing it is unfortunately not fixed yet. For now I can coerce the Path objects to str in the function call. Longer term it's a design question of whether you want to support Path objects as inputs to the function. If that is the case then it might be beneficial to add a helper that does that coercion early in the function calls for operations that take paths as input to make the internal representation uniform rather than having to do the coalescing at multiple locations throughout the code.
I had tried out moving the change that you made in line 99 of https://github.com/Fizzadar/pyinfra/commit/b91281e9e00106e4c1860b0209aa3bd8abf0ee19 above line 94 where the current error is happening. That resolved the traceback noted here, but resulted in a different traceback at a separate code location because of the parameter still being a Path object. That is why I think this merits a more thorough design consideration as opposed to attempting to patch the immediate issue.
This should now be handled properly in https://github.com/Fizzadar/pyinfra/commit/48d6ff7e445856fdc20e971536872dbfcdcf5f4c - Path objects should absolutely be supported IMO but the string conversion wasn't in the right place. Have confirmed this is now working properly for both facts & operations.
I've also added https://github.com/Fizzadar/pyinfra/issues/622 to track adding some proper tests for this!
So, unfortunately I'm still running into bugs with Path arguments. I think the problem is that the type conversion is happening too far down in the stack. What would probably be more robust and easier to maintain is to do the conversion at the perimeter of the system as opposed to doing a whack-a-mole bug hunt.
What I am envisioning is to have a decorator for all operations that accept a Path object that does the conversion before it even makes it into the body of the operation function. Alternatively, if there is a common layer within the operations mechanism that can serve as this choke point then that would allow for file (and other operations that would benefit from Path objects) to be agnostic to the argument type and do the conversion automatically.
I'm happy to have a real-time conversation to discuss the design/architecture aspects.
@blarghmatey agreed - the commit fixes unrelated issues but I definitely jumped the gun a bit here! Would be great to chat real-time about this and understand the issue better before identifying a proper fix (I've emailed you!).