performance: speed up ignore comment detection

[sk-]

Summary

Tokenizing a file is quite slow, as the module is implemented in pure python. So, instead of always tokenizing the files when use_noqa is set, we will first use a fast regex.

The following command on the LibCST repo went from 26 seconds to 17 seconds. fixit run_rules --rules RewriteToLiteralRule UseAssertInRule UseFstringRule NoRedundantFStringRule

Test Plan

Unit test for the new code and maybe some manual testing to verify the speed improvement claims

[codecov-io]

Codecov Report

Merging #165 (b37ca58) into master (ee0c2c9) will increase coverage by 0.03%. The diff coverage is 100.00%.

@@            Coverage Diff             @@
##           master     #165      +/-   ##
==========================================
+ Coverage   85.04%   85.07%   +0.03%     
==========================================
  Files          89       89              
  Lines        3623     3632       +9     
==========================================
+ Hits         3081     3090       +9     
  Misses        542      542              

Impacted Files	Coverage Δ
fixit/common/config.py	87.50% <100.00%> (+0.32%)	:arrow_up:
fixit/common/ignores.py	97.84% <100.00%> (+0.04%)	:arrow_up:
fixit/common/tests/test_ignores.py	100.00% <100.00%> (ø)
fixit/rule_lint_engine.py	95.29% <100.00%> (ø)

---

Continue to review full report at Codecov.

Legend - Click here to learn more Δ = absolute <relative> (impact), ø = not affected, ? = missing data Powered by Codecov. Last update ee0c2c9...b37ca58. Read the comment docs.

[sk-]

Fixed the PR.

In order to make it work I had to change one of the existing regexes, but the change should be safe.

I also had to add a helper method to remove capturing groups from the existing regex.

Let me know what you think:

ps: some tests are having issues installing dependencies.

On Tue, Jan 12, 2021 at 10:35 AM Jimmy Lai [email protected] wrote:

@jimmylai commented on this pull request.

In fixit/common/config.py https://github.com/Instagram/Fixit/pull/165#discussion_r555772940:

+HAS_LINT_IGNORE_REGEXP: Pattern[bytes] = re.compile(rb"# lint-(ignore|fixme):") +HAS_LINT_IGNORE_OR_NOQA_REGEXP: Pattern[bytes] = re.compile(

• rb"# (lint-(ignore|fixme):|noqa|flake8)" +)

The suggestion code has an undefined name config and requires a fix.

— You are receiving this because you authored the thread. Reply to this email directly, view it on GitHub https://github.com/Instagram/Fixit/pull/165#discussion_r555772940, or unsubscribe https://github.com/notifications/unsubscribe-auth/AAG6TGCIPDUZDI4EMY22MOTSZRFZPANCNFSM4VL7KV2Q .

-- Sebastian Kreft

[amyreese]

Hey there! We appreciate your contributions, but we're in the process of making some large changes to the core of Fixit. We will try to have more info about the direction we're heading soon, but in the mean time, we are closing all outstanding PR's from before we started this work. Thank you for your understanding.