BIDS validation routine should not go under `.dotdirs`

[yarikoptic]

Running tests for bidsschematools locally fails for me and floods the screen with WARNINGS like

$> python -m pytest -s -v tools/schemacode/bidsschematools/tests/ 2>&1 | nl | tail
 12496	WARNING  bids-schema:validator.py:688 The `/home/yoh/.tmp/tmp9gjr965o/.git/shallow` file was not matched by any regex schema entry.
 12497	WARNING  bids-schema:validator.py:688 The `/home/yoh/.tmp/tmp9gjr965o/.git/packed-refs` file was not matched by any regex schema entry.
 12498	WARNING  bids-schema:validator.py:688 The `/home/yoh/.tmp/tmp9gjr965o/.git/HEAD` file was not matched by any regex schema entry.
 12499	WARNING  bids-schema:validator.py:688 The `/home/yoh/.tmp/tmp9gjr965o/.git/config` file was not matched by any regex schema entry.
 12500	WARNING  bids-schema:validator.py:688 The `/home/yoh/.tmp/tmp9gjr965o/.git/config.worktree` file was not matched by any regex schema entry.
 12501	WARNING  bids-schema:validator.py:688 The `/home/yoh/.tmp/tmp9gjr965o/.git/index` file was not matched by any regex schema entry.
 12502	INFO     bids-schema:validator.py:527 BIDS validation log written to /home/yoh/.tmp/pytest-of-yoh/pytest-113/test_bids_datasets0/test_bids.log
 12503	=========================== short test summary info ============================
 12504	FAILED tools/schemacode/bidsschematools/tests/test_validator.py::test_bids_datasets
 12505	======================== 1 failed, 19 passed in 19.55s =========================
python -m pytest -s -v tools/schemacode/bidsschematools/tests/ 2>&1  4.79s user 0.37s system 24% cpu 21.408 total
nl  0.02s user 0.00s system 0% cpu 21.407 total
tail  0.01s user 0.00s system 0% cpu 21.407 total

suggesting that content of .git/ folders is considered. I believe that bids-validator (after a number of issues filed for .heudiconv/ folders etc) just added logic to exclude all directories with . prefix. I could be proven wrong. I think in the case of validator here .dotdirs also should not be navigated/tested for validity.

[TheChymera]

@yarikoptic not sure what version/branch you were running this on, but I canot reproduce this on current master or any of the other PRs I'm working on --- it's also the sort of thing which would fail in the test suite.

chymera@darkhost ~/src/bids-specification $ python -m pytest -s -v tools/schemacode/bidsschematools/tests/ 2>&1 | nl | tail
  9237		* /tmp/tmpx1798o55/invalid_pet001/sub-01/ses-01/anat/sub-01_ses-01_T1w.nii, with pattern: .*?/sub-(?P<subject>[0-9a-zA-Z]+)/(|ses-(?P<session>[0-9a-zA-Z]+)/)anat/sub-(?P=subject)(|_ses-(?P=session))(|_acq-(?P<acquisition>[0-9a-zA-Z]+))(|_ce-(?P<ceagent>[0-9a-zA-Z]+))(|_rec-(?P<reconstruction>[0-9a-zA-Z]+))(|_run-(?P<run>[0-9]*[1-9]+[0-9]*))(|_part-(?P<part>mag|phase|real|imag))_(T1w|T2w|PDw|T2starw|FLAIR|inplaneT1|inplaneT2|PDT2|angio|T2star|FLASH|PD)(\.nii\.gz|\.nii|\.json)$
  9238	Match identified.
       
  9239	-------------------------------- live log call ---------------------------------
  9240	WARNING  bids-schema:validator.py:686 The `/tmp/tmpx1798o55/invalid_pet001/sub-01/ses-01/anat/sub-02_ses-01_T1w.json` file was not matched by any regex schema entry.
  9241	2022-08-01 19:13:21,884 [    INFO] BIDS validation log written to /var/tmp/bids-validator/report_20220801231321Z-702050.log
  9242	INFO     bids-schema:validator.py:525 BIDS validation log written to /var/tmp/bids-validator/report_20220801231321Z-702050.log
  9243	PASSED
       
  9244	============================== 29 passed in 6.09s ==============================
chymera@darkhost ~/src/bids-specification $ git describe
v1.7.0-443-g54e7990

As for excluding all dotdirs, currently we have a list of dotdirs which we exclude, including ofc .git. Clearly if BIDS would ever put stuff under .git (or to a lesser extent .dandi or .datalad) BIDS is broken, and we do not need to tailor the validator to BIDS becoming broken (i.e. we can safely hard-code those exclusions).

I am not sure however, if we can say that BIDS will never cover any dotdirs and that the validator should assume all dotdirs have a sensible purpose so the user should never be notified. I think if some user has a dangling .cache inside with random logs or god knows what else there should be warnings about that.

[sappelhoff]

As for excluding all dotdirs, currently we have a list of dotdirs which we exclude, including ofc .git.

that sounds sensible to me

I think if some user has a dangling .cache inside with random logs or god knows what else there should be warnings about that.

agreed.

Okay to close @yarikoptic?

[sappelhoff]

closed in #1305