site-package search path doesn't support single file Python modules

[luizribeiro]

One of my projects that uses pyre depends on pypng, which is a Python module that consists of a single .py file (i.e. its path is <venv>/lib/python3.9/site-packages/png.py).

I tried adding it to the search path by doing this:

{
  ...
  "search_path": [
    { "site-package": "png" },
    ...
  ]
}

But then pyre rage shows me that that path was just ignored since it's non-existent:

DEBUG Filtering out nonexistent search path: /home/luiz/.cache/pypoetry/virtualenvs/mariner-Ytm4xNoM-py3.9/lib/python3.9/site-packages/png

This makes sense, since the real path is .../png.py, so I tried { "site-package": "png.py" } instead, but then pyre just fatals with this message:

(mariner-Ytm4xNoM-py3.9) luiz@nuc mariner % pyre
ƛ Traceback (most recent call last):
  File "/home/luiz/.cache/pypoetry/virtualenvs/mariner-Ytm4xNoM-py3.9/lib/python3.9/site-packages/pyre_check/client/socket_connection.py", line 45, in perform_handshake
    json_rpc.perform_handshake(self.input, self.output, version_hash)
  File "/home/luiz/.cache/pypoetry/virtualenvs/mariner-Ytm4xNoM-py3.9/lib/python3.9/site-packages/pyre_check/client/json_rpc.py", line 190, in perform_handshake
    raise ValueError("Handshake from server was malformed.")
ValueError: Handshake from server was malformed.

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/home/luiz/.cache/pypoetry/virtualenvs/mariner-Ytm4xNoM-py3.9/lib/python3.9/site-packages/pyre_check/client/pyre.py", line 115, in run_pyre_command
    exit_code = command.run().exit_code()
  File "/home/luiz/.cache/pypoetry/virtualenvs/mariner-Ytm4xNoM-py3.9/lib/python3.9/site-packages/pyre_check/client/commands/command.py", line 207, in run
    self._run()
  File "/home/luiz/.cache/pypoetry/virtualenvs/mariner-Ytm4xNoM-py3.9/lib/python3.9/site-packages/pyre_check/client/commands/incremental.py", line 112, in _run
    self._ensure_server_and_monitors_are_initialized()
  File "/home/luiz/.cache/pypoetry/virtualenvs/mariner-Ytm4xNoM-py3.9/lib/python3.9/site-packages/pyre_check/client/commands/incremental.py", line 76, in _ensure_server_and_monitors_are_initialized
    Start(
  File "/home/luiz/.cache/pypoetry/virtualenvs/mariner-Ytm4xNoM-py3.9/lib/python3.9/site-packages/pyre_check/client/commands/command.py", line 207, in run
    self._run()
  File "/home/luiz/.cache/pypoetry/virtualenvs/mariner-Ytm4xNoM-py3.9/lib/python3.9/site-packages/pyre_check/client/commands/start.py", line 93, in _run
    file_monitor = project_files_monitor.ProjectFilesMonitor(
  File "/home/luiz/.cache/pypoetry/virtualenvs/mariner-Ytm4xNoM-py3.9/lib/python3.9/site-packages/pyre_check/client/project_files_monitor.py", line 68, in __init__
    self.socket_connection.perform_handshake(
  File "/home/luiz/.cache/pypoetry/virtualenvs/mariner-Ytm4xNoM-py3.9/lib/python3.9/site-packages/pyre_check/client/socket_connection.py", line 47, in perform_handshake
    raise SocketException(
pyre_check.client.socket_connection.SocketException: Exception encountered during handshake: `Handshake from server was malformed.`

It seemed to me like the problem was on the OCaml side, so I found this:

https://github.com/facebook/pyre-check/blob/c1d0995b111f352c6abc69725b51c906b0199393/source/service/infer.ml#L68-L73

Do we really need to make the assumption that all search paths are directories?

[dkgi]

Ironically, this is hitting us as well when running Pyre on Pyre. cc @grievejia

[grievejia]

Thanks for the report! Yeah I think the issue here is actually more general: currently Pyre interprets elements in search_path to be all directories. This fundamentally limits its capability to handle single-file Python modules. Both the OCaml side and the Python side have to be updated to address the issue.

As a temporary workaround, @dkgi recently added a feature where if search_path is not specified in the configuration and if a venv is detected, Pyre would automatically include every site package in the venv. And that would work with single-file packages. The feature is currently only available in pyre-check-nightly but let me cut a new pyre-check release this afternoon to make it more accessible.