do not inspect folders/files Excluded by pycharm

[yarikoptic]

in my case

[978310] 2020-10-27 09:06:30,089 - watchgod.watcher - WARNING - error walking file system: FileNotFoundError [Errno 2] No such file or directory: '/home/yoh/proj/datalad/datalad-master/venvs/dev3.5/lib/python3.5/os.py'
[978310] 2020-10-27 09:06:30,488 - watchgod.watcher - WARNING - error walking file system: FileNotFoundError [Errno 2] No such file or directory: '/home/yoh/proj/datalad/datalad-master/venvs/dev3.5/lib/python3.5/os.py'
[978310] 2020-10-27 09:06:30,897 - watchgod.watcher - WARNING - error walking file system: FileNotFoundError [Errno 2] No such file or directory: '/home/yoh/proj/datalad/datalad-master/venvs/dev3.5/lib/python3.5/os.py'
[978310] 2020-10-27 09:06:31,296 - watchgod.watcher - WARNING - error walking file system: FileNotFoundError [Errno 2] No such file or directory: '/home/yoh/proj/datalad/datalad-master/venvs/dev3.5/lib/python3.5/os.py'
[978310] 2020-10-27 09:06:31,694 - watchgod.watcher - WARNING - error walking file system: FileNotFoundError [Errno 2] No such file or directory: '/home/yoh/proj/datalad/datalad-master/venvs/dev3.5/lib/python3.5/os.py'
[978310] 2020-10-27 09:06:32,099 - watchgod.watcher - WARNING - error walking file system: FileNotFoundError [Errno 2] No such file or directory: '/home/yoh/proj/datalad/datalad-master/venvs/dev3.5/lib/python3.5/os.py'

keeps flooding the screen. venvs is completely excluded in pycharm

[gleb-sevruk]

Yeah, this currently needs to be in excluded in .pycrunch-config.yaml

discovery:
  exclusions:
  # do not search tests inside venv 
  - venvs
engine:
  runtime: pytest

I haven't researched if it is possible to load excluded folders via Intelij SDK

[yarikoptic]

THANKS! Is there inclusions, which is pretty much defaults to . and are considered before exclusions - i.e. they define what to consider and then exclusions exclude some "considered"? ;) Then for 99% of projects it would be just include: [package_name, tests] (tests just in case if they aren't embedded within package_name) and disregard all the exterior stuff which is not expected to be tested anyways.

(BTW: exclude and include IMHO would be a bit cleaner to use here)

[yarikoptic]

also, as example in the codebase shows:

  exclusions:
  - front/
  - front\
  - build/
  - test_discovery_specs_demo.py

there is difficulty of handling directories across platforms. So may be exclusions and paths should be first unified to be POSIX for uniform specification?

[yarikoptic]

Exclusions aren't enough. I have now

$> cat .pycrunch-config.yaml                                                                                
discovery:
  exclusions:
  - build
  - venvs
  - dist
  - doc
  - .
engine:
  cpu-cores: 4
  load-pytest-plugins: false
  multiprocessing-threshold: 4
  runtime: nose
  timeout: 0
env:
  DATALAD_TESTS_SSH: "1"

and I think it did not pick up tests from venvs, but then I still have lots of

[1072016] 2020-10-27 12:44:34,068 - watchgod.watcher - WARNING - error walking file system: FileNotFoundError [Errno 2] No such file or directory: '/home/yoh/proj/datalad/datalad-master/venvs/dev3.5/lib/python3.5/os.py'
[1072016] 2020-10-27 12:44:34,467 - watchgod.watcher - WARNING - error walking file system: FileNotFoundError [Errno 2] No such file or directory: '/home/yoh/proj/datalad/datalad-master/venvs/dev3.5/lib/python3.5/os.py'
[1072016] 2020-10-27 12:44:34,869 - watchgod.watcher - WARNING - error walking file system: FileNotFoundError [Errno 2] No such file or directory: '/home/yoh/proj/datalad/datalad-master/venvs/dev3.5/lib/python3.5/os.py'

I think it is largely because there is no inclusions and the whole tree is still traversed while applying exclusions later on

        py_files = folder_path.glob('**/*.py')
        ...
        with ModuleCleanup() as cleanup:

            for py_file in py_files:
...

I guess "inclusions" could be easily implemented by chaining all the folder_path.glob(f'{inclusion}**/*.py') when multiple provided, thus not even going into places were it shouldn't go.

edit 3: unfortunately it is not sufficient. I guess watcher also manages to look elsewhere ... I did hardcode in above glob datalad/ and still was getting those WARNINGS from watcher

[yarikoptic]

For fun of it -- here is how my venvs look like ;):

$> ls venvs         
build-v3.7.0a4-2497-g2f6fae6e510/  build-v3.8.0a4-388-g396e0a8d9dc/         dev-dbg/
build-v3.7.0a4-2498-ga8474d025ca/  build-v3.8.0b1/                          dev-isolated/
build-v3.8.0/                      build-v3.8.0b1-109-ga7072ff56e2/         dev-nosystem/
build-v3.8.0a1/                    build-v3.8.0b1-159-g63429c839b1/         dev-sregistry/
build-v3.8.0a1-110-g06babb24225/   build-v3.8.0b1-171-g515a026eb06/         dev-trusty/
build-v3.8.0a1-14-g2848d9d2991/    build-v3.8.0b1-178-g3e133c401a5/         dev3/
build-v3.8.0a1-27-g181835d5a9b/    build-v3.8.0b1-179-gffcc161c753+cb083f/  dev3+containers/
build-v3.8.0a1-4-gca7d2933a38/     build-v3.8.0b1-180-gbf8cb318035/         dev3-install/
build-v3.8.0a1-55-g903567e4f54/    build-v3.8.0b1-180-gbf8cb318035+cb083f/  dev3-installfrompip/
build-v3.8.0a1-7-g85e102a2b09/     build-v3.8.0b1-181-g4137514b949/         dev3-isolated/
build-v3.8.0a2-97-g495da292255/    build-v3.8.0b1-183-gf22c4cf11d1/         dev3.5/
build-v3.8.0a3-76-gb4bcefe5fe6/    build-v3.8.0b1-184-gcb083f7cdf6/         dev3.7/
build-v3.8.0a4/                    build-v3.8.0b2/
build-v3.8.0a4-297-g3880f263d29/   dev/

[gleb-britecore]

It looks like some symlink (lib/python3.5/os.py).

But I do not understand why is it watching this folder. Maybe there is some coverage collected on the file?

Can you try to put this as exclusion in: https://github.com/gleb-sevruk/pycrunch-engine/blob/27bbacc348cd073f2bef239ea5f19980d311b062/pycrunch/child_runtime/exclusions.py#L9

[gleb-britecore]

Maybe your tests hit lib/python3.5/os.py and then scheduled to be watched for changes inside ./pycrunch/watcher/fs_watcher.py FSWatcher.watch method

[yarikoptic]

I bet before I removed that venvs/dev3 it could have been hitting system wide python3.5 os . But the point was - why to watch files outside of the project at all? Sure they could change with upgrade, but I wouldn't expect them to -- should at least to be an option I guess (or configurable via inclusions thus limiting only to local project tree)

[gleb-britecore]

This is sometimes convenient to dive into libraries in pycharm and see what is covered inside of libraries: /Users/gleb/venv/lib/python3.6/site-packages/django/test/client.py:

But I agree this should not default behavior.

[yarikoptic]

Yeah, indeed very nice. We never dared to dive that deep in the coverage (and very rarely troubleshooted bugs in python itself), sand since upload only coverage for our package to codecov, never collected coverage outside.