sqlparse
sqlparse copied to clipboard
Compute TokenList.value dynamically
Partial fix for #621:
To avoid quadratic behavior, make TokenList.value
a dynamically-computed property rather than an attribute so that it does not need to be recomputed each time TokenList.group_tokens()
is called with extend=True
.
The first three commits in this PR are supporting work: I found that making TokenList.value
a property caused test failures due to line endings not being correctly preserved when stripping comments. This is due to the fact that before making TokenList.value
a property, the StripCommentsFilter
was changing the underlying tokens without updating the value
attribute of the parent TokenList
. After making TokenList.value
a property, those changes did get reflected in the parent TokenList
's value
and as a result the desired line endings were being lost.
The most straightforward way I found to address this was to make comment stripping happen before grouping is performed, which required a small amount of hackery to make grouping happen via a filter. I am open to suggestions to better ways to handle this.
I am also a little concerned about the possibility for slowdowns if a particular TokenList
's value is accessed, and thus computed, multiple times, but I didn't actually observe any. This might still be an issue via a codepath I didn't look at. I have some ideas on how to address it if a performance problem comes up, but it would clutter up the code somewhat so I didn't want to implement it unless a need could be demonstrated.
Codecov Report
Merging #623 (0e187cd) into master (23d2993) will increase coverage by
0.03%
. The diff coverage is100.00%
.
@@ Coverage Diff @@
## master #623 +/- ##
==========================================
+ Coverage 96.82% 96.86% +0.03%
==========================================
Files 20 20
Lines 1514 1532 +18
==========================================
+ Hits 1466 1484 +18
Misses 48 48
Impacted Files | Coverage Δ | |
---|---|---|
sqlparse/__init__.py | 100.00% <100.00%> (ø) |
|
sqlparse/engine/filter_stack.py | 100.00% <100.00%> (ø) |
|
sqlparse/filters/__init__.py | 100.00% <100.00%> (ø) |
|
sqlparse/filters/others.py | 98.90% <100.00%> (+0.10%) |
:arrow_up: |
sqlparse/formatter.py | 95.12% <100.00%> (ø) |
|
sqlparse/sql.py | 97.67% <100.00%> (+0.06%) |
:arrow_up: |
Continue to review full report at Codecov.
Legend - Click here to learn more
Δ = absolute <relative> (impact)
,ø = not affected
,? = missing data
Powered by Codecov. Last update 23d2993...0e187cd. Read the comment docs.
This PR will potentially fix jazzband/django-debug-toolbar#1402. I would really appreciate if it can be reviewed and considered for being merged!
Closing in favor of #710.