Improve site package pathing

[WangGithubUser]

Pre-submission checklist

• [x] I've ran the linters locally and fixed lint errors related to the files I modified in this PR. You can install the linters by running pip install -r requirements-dev.txt && pre-commit install
• [x] pre-commit run

Summary

A top-level package could implement by many ways, like .so, .pyd file, etc. However, pyre only treat a .py file as a top-level package: https://github.com/facebook/pyre-check/blob/689a8bed35d3ab2a134a9bfb0c49845764760361/client/configuration/search_path.py#L76-L77 This PR improved the pathing by seeing the RECORD file in the package'sdist-info e.g.This is the RECORD file in the dist-info of the ujson(ujson-5.8.0.dist-info):

ujson-5.8.0.dist-info/INSTALLER,sha256=zuuue4knoyJ-UwPPXg8fezS7VCrXJQrAP7zeNuwvFQg,4
ujson-5.8.0.dist-info/LICENSE.txt,sha256=NoZjRPCkhlQb2YAPGkB6BWwMIUqCxljwWWlNrz3DlNI,5975
ujson-5.8.0.dist-info/METADATA,sha256=QjK2Swt-E5vB5zyhRdsqHQ3iNODpRnbhQBXI-MPpTIM,8691
ujson-5.8.0.dist-info/RECORD,,
ujson-5.8.0.dist-info/REQUESTED,sha256=47DEQpj8HBSa-_TImW-5JCeuQeRkm5NMpJWZG3hSuFU,0
ujson-5.8.0.dist-info/WHEEL,sha256=iZaXX0Td62Nww8bojl0E84uJHjT41csHPKZmbUBbJPs,152
ujson-5.8.0.dist-info/top_level.txt,sha256=IBMYuVICHWi9xnmMN22UNMEFYf14TccC7o9uYPhycg0,6
ujson.cpython-310-x86_64-linux-gnu.so,sha256=Zk7sJGRXyPb_nJdf2ArotWK0zAFBWrcRwGJMfMXMHZ8,106000

As you see, We only need ujson.cpython-310-x86_64-linux-gnu.so but there are a lot of other unnessarry files in the RECORD.For this, I filterd them out by matching if the file is in XXX-XXX.dist-info or __pycache__.

Also see https://github.com/facebook/pyre-check/issues/773#issuecomment-1672481776.

Issue fixed: #773

Test Plan

Before this PR:

After this PR:

$ ls -A
.pyre  .pyre_configuration  pyre-check

$ cat .pyre_configuration
{
  "site_package_search_strategy": "pep561",
  "source_directories": [
    "."
  ],
  "strict": true,
  "search_path": [
    {
      "site-package": "ujson",
      "is_toplevel_module": true
    }
  ],
  "exclude": [
    "pyre-check"
  ]
}

$ python -m pyre-check.client.pyre --noninteractive check # PR 781
2023-09-17 10:37:08,345 [PID 1496] INFO No binary specified, looking for `pyre.bin` in PATH
2023-09-17 10:37:08,347 [PID 1496] INFO Pyre binary is located at `/home/cbw/.local/bin/pyre.bin`
2023-09-17 10:37:08,351 [PID 1496] INFO Could not determine the number of Pyre workers from configuration. Auto-set the value to 7.
2023-09-17 10:37:08,355 [PID 1496] INFO No typeshed specified, looking for it...
2023-09-17 10:37:08,357 [PID 1496] INFO Found: `/usr/lib/pyre_check/typeshed`
2023-09-17 10:37:08,360 [PID 1496] INFO Writing arguments into /tmp/pyre_arguments_ym16vc1b.json...
2023-09-17 10:37:08,362 [PID 1496] DEBUG Arguments:
{
  "source_paths": {
    "kind": "simple",
    "paths": [
      "/mnt/d/Downloads/TestPyre"
    ]
  },
  "search_paths": [
    "/home/cbw/.local/lib/python3.10/site-packages$ujson.cpython-310-x86_64-linux-gnu.so",
    ...<I omited other search_paths>
  ],
  "excludes": [
    "pyre-check"
  ],
  "checked_directory_allowlist": [
    true
  ],
  "checked_directory_blocklist": [],
  "extensions": [],
  "log_path": "/mnt/d/Downloads/TestPyre/.pyre",
  "global_root": "/mnt/d/Downloads/TestPyre",
  "debug": false,
  "python_version": {
    "major": 3,
    "minor": 10,
    "micro": 12
  },
  "shared_memory": {},
  "parallel": true,
  "number_of_workers": 7,
  "additional_logging_sections": [
    "-progress"
  ],
  "show_error_traces": false,
  "strict": true
}
2023-09-17 10:37:08,407 [PID 1496] ERROR  Malformed server specification JSON. Expected string, got bool
2023-09-17 10:37:08,411 [PID 1496] ERROR Check command exited with non-zero return code: 1.

[facebook-github-bot]

@inseokhwang has imported this pull request. If you are a Meta employee, you can view this diff on Phabricator.

[kinto0]

Hi! It seems like your issue is site packages are not being detected? Is this on windows? Would you be able to try on WSL since we currently do not support windows?

[WangGithubUser]

Would you be able to try on WSL since we currently do not support windows?

(on WSL)

https://github.com/facebook/pyre-check/blob/baad875701eed38747cc30cad95eafb7754f991c/client/configuration/search_path.py#L76-L77 Pyre only treat a .py file as a site-package, and this PR improved the pathing of site-package.

[kinto0]

I'm not convinced yet that this is the right approach. I left some comments but I'll also bring it up with the team.

General feedback:

• could you put more details in the summary?
• could you do a before / after end to end test of this behavior? It's important to understand how this search_path changes the ocaml daemon functionality

[WangGithubUser]

Here is a tasklist for this PR:

1. [ ] Add a cache system for files in site_root or the check command will be very slow.
2. [ ] Fix error Malformed server specification JSON. Expected string, got bool.
3. [ ] Factor out the logic of path func of SitePackageElement.

[WangGithubUser]

@kinto0 For the second task, I guess this may because a .so can't detect it's annotations.But I'm not sure, could you give me more info about this?