sqlparse icon indicating copy to clipboard operation
sqlparse copied to clipboard

Compute TokenList.value dynamically

Open living180 opened this issue 3 years ago • 1 comments

Partial fix for #621:

To avoid quadratic behavior, make TokenList.value a dynamically-computed property rather than an attribute so that it does not need to be recomputed each time TokenList.group_tokens() is called with extend=True.

The first three commits in this PR are supporting work: I found that making TokenList.value a property caused test failures due to line endings not being correctly preserved when stripping comments. This is due to the fact that before making TokenList.value a property, the StripCommentsFilter was changing the underlying tokens without updating the value attribute of the parent TokenList. After making TokenList.value a property, those changes did get reflected in the parent TokenList's value and as a result the desired line endings were being lost.

The most straightforward way I found to address this was to make comment stripping happen before grouping is performed, which required a small amount of hackery to make grouping happen via a filter. I am open to suggestions to better ways to handle this.

I am also a little concerned about the possibility for slowdowns if a particular TokenList's value is accessed, and thus computed, multiple times, but I didn't actually observe any. This might still be an issue via a codepath I didn't look at. I have some ideas on how to address it if a performance problem comes up, but it would clutter up the code somewhat so I didn't want to implement it unless a need could be demonstrated.

living180 avatar May 19 '21 12:05 living180

Codecov Report

Merging #623 (0e187cd) into master (23d2993) will increase coverage by 0.03%. The diff coverage is 100.00%.

Impacted file tree graph

@@            Coverage Diff             @@
##           master     #623      +/-   ##
==========================================
+ Coverage   96.82%   96.86%   +0.03%     
==========================================
  Files          20       20              
  Lines        1514     1532      +18     
==========================================
+ Hits         1466     1484      +18     
  Misses         48       48              
Impacted Files Coverage Δ
sqlparse/__init__.py 100.00% <100.00%> (ø)
sqlparse/engine/filter_stack.py 100.00% <100.00%> (ø)
sqlparse/filters/__init__.py 100.00% <100.00%> (ø)
sqlparse/filters/others.py 98.90% <100.00%> (+0.10%) :arrow_up:
sqlparse/formatter.py 95.12% <100.00%> (ø)
sqlparse/sql.py 97.67% <100.00%> (+0.06%) :arrow_up:

Continue to review full report at Codecov.

Legend - Click here to learn more Δ = absolute <relative> (impact), ø = not affected, ? = missing data Powered by Codecov. Last update 23d2993...0e187cd. Read the comment docs.

codecov[bot] avatar May 19 '21 12:05 codecov[bot]

This PR will potentially fix jazzband/django-debug-toolbar#1402. I would really appreciate if it can be reviewed and considered for being merged!

adamantike avatar Sep 26 '22 00:09 adamantike

Closing in favor of #710.

living180 avatar Mar 28 '23 13:03 living180