g-plane
Fix AST inconsistency between SASS/SCSS caused by indentation
Fixes #7 by
- Don't consume the source when
Dedenttoken should be returned. - Keep
Dedenttokens in the state, because they don't have anything to actually consume - Keep indentation size per level to know how much
Dedents should be generated.- make "inconsistent indentation" unrecoverable error
At 522d0ca1e6cd884d46e2047c747b4f025a088aa7 new ast's are in snapshots. Also added scss test to compare ASTs - seems fine, plz check :3
Not sure about 24e70c358d9f919ad54f2562ca6e37a3e992ba85 though. Looks premature..?
We are cloning state in many cases. Does changing the definition of TokenizerState affect performance?
It would be as much slower, as two heap allocations of (n) + 2 Vec triplets are slow, AFAIK. I'd say that N is (2+16+3)*indentation bytes (2 for u16 for level, 16 is Span, 3 is the "worst" case for the Token I've found)
We could use the SmallVec which seems to be faster for the case, but the potential indentation depth must be pre-estimated, which is a magic number, similar to the capacity of "8" 24e70c358d9f919ad54f2562ca6e37a3e992ba85
I did 3 tests before and one after the changes:
Here are two last of them, those come from criterion for parse_selector/Raffia
<before>
Lower Bound Estimate Upper Bound
Value: 1.28µs 1.28µs 1.28µs
Change in Value: -0.6323% -0.0569% +0.4337%
Change within noise threshold.
<after>
Lower Bound Estimate Upper Bound
Value: 1.26µs 1.26µs 1.26µs
Change in Value: -1.9907% -1.8563% -1.7194%
No change in performance detected.
i9-13980HX
You can run the "self" benchmark which only runs for Raffia itself and supports CSS, SCSS, Sass and Less. Other benchmarks are only for CSS and they will run on other parsers, which wastes of time.
Other benchmarks are only for CSS and they will run on other parsers
Oh, my bad, I completely missed that. Then It's ~(+17%) for bootstrap5/formcheck.scss. Definitely must be better.
Pardon me for the PR-noise then, I'll convert this to draft for now, if it's OK.