`no-import-test-files` is slow

[fisker]

TIMING=all xo

Rule                                             | Time (ms) | Relative
:------------------------------------------------|----------:|--------:
ava/no-import-test-files                         |  1133.224 |    81.5%
indent                                           |    34.157 |     2.5%
ava/assertion-arguments                          |     6.868 |     0.5%
unicorn/prevent-abbreviations                    |     4.737 |     0.3%
n/no-unsupported-features/es-syntax              |     4.222 |     0.3%
ava/no-nested-tests                              |     3.447 |     0.2%
capitalized-comments                             |     3.253 |     0.2%