scientific-python
SPEC 12: Formatting mathematical expressions
This follows the proposal on the forum https://discuss.scientific-python.org/t/spec-12-formatting-mathematical-expressions
See other linked discussions as well.
cc @mdhaber @stefanv @jarrodmillman @j-bowhay and add yourself as co-authors of course 😉
Hi @charliermarsh @ambv 👋 I had pinged you both on Twitter some years back about a standard for mathematical equations. At the time you both said this could interest you for both Ruff and Black if we, the Scientific Python community, could come to an agreement.
To be clear, I am of course in no position to ask for any commitment from you and I just hope that you find the topic interesting enough.
This is getting into shape and we have a preliminary draft we would like to present you 😃
Any comments would greatly help and in the end this can also only work if both Ruff and Black would be able and willing to implement this specification 🙏
Thanks again both!
Thanks @charliermarsh. I think the main question right now is whether this is close to being precise enough to be implementable. For example, I don't have much background with ASTs, so perhaps you can suggest ideas that would replace the notions like "implicit subexpressions" I attempted to define. One we have a better understanding of how to write this standard so that it's implementable, we will want to get feedback from a wider audience about adjustments to the particular rules.
My initial reaction is that the definitions are sufficiently clear (minor some comments inline) to be implementable. I don't know that it would be easy but I don't see ambiguities.
By the way, if you want to look at the AST for any of these, we have a playground: https://play.ruff.rs/85bce11a-11c4-4b66-b7ef-bf13cfb2d518. We deviate from the CPython AST in some places, but for binary expressions it should be roughly the same IIRC.
I only skimmed through it, but it's great to see more people think about improving expression formatting. I don't recall Ruff's and Black's rules explicitly. Do you know where this style guide deviates from Black/Ruff (evaluating one should be enough)?
I'm especially interested in the formatting of very long expressions. I've written up some of my thoughts in https://github.com/psf/black/issues/4123
I don't think it goes against any core Ruff rules because it doesn't conflict with PEP8, but please correct me @charliermarsh.
Ruff implements a few E2XX and E3XX rules that we would have to double-check. E.g. missing-whitespace-around-operator
What's not entirely clear to me from the proposal is what your goal is:
- Is it an automatic formatter similar to Black that enforces consistent formatting?
- Is it a set of lint rules that enforce a specific style for some expressions?
There are differences between the two and integrating your proposal into ruff format or ruff check imposes different challenges depending on what you're looking for.
Formatter
The main goal of a formatter is to enforce consistent styling across a code base. The implication is that letting the user decide on the formatting generally isn't an option because it wouldn't lead to consistent formatting. The only exception to this in Ruff's (and Black's) formatter today is the use of magic-trailing-commas where the user can choose one style by adding a trailing comma.
I think there are a few places where the proposal would have to become more opinionated if it should be implemented as an auto formatter.
The other challenge when it comes to building an auto-formatter is that the style guide must be consistent in its entirety. For example, ruff (and Black) place the opening parentheses on their own line:
(
t
+ (
w
+ (x + (y + z))
)
)
whereas this document proposes
(t
+ (w + (x + (y + z)))))
(I prefer your formatting, which is also what Prettier does, but let's ignore this for now).
The main challenge with this formatting is that it is very different from how Black/Ruff places parentheses for all other syntaxes (list, parameters, arguments...). It makes binary expressions the "odd" ones, standing out from the rest of the code. I'm not concluding that this is a reason to rule out the style, but we would have to consider the style guide in the context of Ruff's entire style guide.
Linter
The alternative is that the style guide is enforced by lint rules, each of which enforces a specific aspect. Some lint rules may have auto-fixes, but not all of them.
From what I understand in the proposal, that's a better fit. The main integration challenge for Ruff is that lint rules enforcing a style different from the formatter create a compatibility problem: running ruff check --fix fixes the style in one direction, and ruff format undoes the changes. Style rules are also prone to being incompatible with other stylistic lint rules. E.g. are all proposed changes compatible with Ruff's E2XX and E3XX rules?
This aren't reasons why Ruff can't support the proposed styles. But these are challenges that we would have to overcome and find solutions for before they can be added to ruff.
Note
Another annoying thing that might be worth considering is if the specification wants to be explicit about where to place comments. How would expression be formatted if there's a leading comment? What about end of line comments?
Ruff implements a few E2XX and E3XX rules
Thanks! I'll take a look. But yes, this definitely conflicts with E225 (which actually seems to go against PEP8, TBH). (Update: also 226 sometimes, but consistent with E227 because shifts are lower priority than PEMDAS operators. I think that's it.).
Style rules are also prone to being incompatible with other stylistic lint rules.
Rules can be turned on/off, though, right? After we determine which rules these conflict with, perhaps the SPEC would recommend which other rules would need to be disabled (e.g. E225) to adopt the SPEC?
For example, ruff (and Black) place the opening parentheses on their own line
Hmm I intended for this to be explicitly agnostic about that specific point:
Although examples do not show the use of hanging indent, any of the indentation styles
allowed by [PEP 8 Indentation](https://peps.python.org/pep-0008/#indentation) are permitted by this SPEC.
As for where the line breaks go (Rules 7/8), I'm not sure they are explicit enough yet. So maybe we can leave those out initially, if that would help.
...where to place comments.
Hmm I didn't think about that. I don't have that in mind for the scope. Does this interact with the other rules besides those regarding line breaks? Maybe that's another reason to table the rules about line breaks.
What's not entirely clear to me from the proposal is what your goal is:
The SPEC itself wouldn't say how the rules are to be enforced, but I think you are right that they would be better as part of a linter with some auto-fixes available.
So would that help to table rules 7 and 8 (about line break placement) and focus on enforcing the rest as linting rules (disabling other rules as needed)?
Thanks so much for your help!
What's not entirely clear to me from the proposal is what your goal is:
Is it an automatic formatter similar to Black that enforces consistent formatting? Is it a set of lint rules that enforce a specific style for some expressions?
Speaking for myself, I would like to be able to enforce these rules via ruff format. Hopefully this along with https://github.com/astral-sh/ruff/discussions/8452 would mean we could start to consider an auto-formatter in projects like SciPy.
Yep that's the idea, wish. Relying on Ruff's team to do it though as I am not sure any of us would have the know how to help on the implementation part.
Rules can be turned on/off, though, right? After we determine which rules these conflict with, perhaps the SPEC would recommend which other rules would need to be disabled (e.g. E225) to adopt the SPEC?
They can, but everyone using select=ALL will see a warning unless they explicitly ignore the rule. That's why we tried to avoid adding any new rules that conflict with each other. We also want to categorize the rules long term and it's somewhat likely that we'll remove the ALL selector, so that conflicting rules are less of a problem, as long as they aren't enabled by default. The other solution has been to introduce a setting, e.g. similar to pydocstyle's convention setting
As for where the line breaks go (Rules 7/8), I'm not sure they are explicit enough yet. So maybe we can leave those out initially, if that would help.
I would have to re-read them but it's okay to be vague if this proposal should be implemented as lint rules because Ruff can then just not enforce the style or ask users to use noqa to opt out.
Yep that's the idea, wish. Relying on Ruff's team to do it though as I am not sure any of us would have the know how to help on the implementation part.
Happy to help you to learn more about how the formatter works.
Integrating this into the formatter certainly has a higher bar than into the linter. If possible, I would prefer to improve the existing style for the entire ecosystem over maintaining (and justifying) two styles. I think there's room to make expression formatting better (see https://github.com/psf/black/issues/4123). If that's not possible, we then have to decide if Ruff should support multiple styles and what the bar is to justify why Ruff supports one style vs. any other.
Thanks @MichaReiser 🙇 Assuming we get this in (which we should if everyone is happy as to not be blocking anything), what do we do next? Would you be willing to take a shot at implementing these rules? First in the linter if I read correctly.
If possible, I would prefer to improve the existing style for the entire ecosystem over maintaining (and justifying) two styles. I think there's room to make expression formatting better (see psf/black#4123). If that's not possible, we then have to decide if Ruff should support multiple styles and what the bar is to justify why Ruff supports one style vs. any other.
IIRC, the np.array formatting discussion was what sparked the idea of a separate "scientific python" style. But if it could all be incorporated into the one style, great!
The very first step that I see is that you find agreement if you want this to be part of an auto-formatter (ruff format) or linter (ruff check) because both have different requirements on how specific a specification must be. Depending on that it is then our turn to figure out how this fits into ruff: e.g. we've been very intentional so far to not support multiple code styles in ruff format.
Would you be willing to take a shot at implementing these rules? First in the linter if I read correctly.
We're fairly thin-stretched right now with our work on UV, Ruff maintenance, and adding multifile analysis support to Ruff. We also have to solve many fundamental efforts like rule categorization first. Short answer: I'm very interested in improving code formatting for the scientific Python community, but it likely takes a while before I can make it a priority.
The alternative is that the style guide is enforced by lint rules, each of which enforces a specific aspect. Some lint rules may have auto-fixes, but not all of them.
From what I understand in the proposal, that's a better fit. The main integration challenge for Ruff is that lint rules enforcing a style different from the formatter create a compatibility problem: running ruff check --fix fixes the style in one direction, and ruff format undoes the changes. Style rules are also prone to being incompatible with other stylistic lint rules. E.g. are all proposed changes compatible with Ruff's E2XX and E3XX rules?
This aren't reasons why Ruff can't support the proposed styles. But these are challenges that we would have to overcome and find solutions for before they can be added to ruff.
The very first step that I see is that you find agreement if you want this to be part of an auto-formatter (ruff format) or linter (ruff check) because both have different requirements on how specific a specification must be.
@MichaReiser would the following be a feasible path forward here?
- Implement lint rules which can detect everything this document seeks to disallow
- this would cause conflict with some existing rules like https://docs.astral.sh/ruff/rules/missing-whitespace-around-operator/, which btw @mdhaber and I agree is not justified in claiming to be "according to PEP 8"^1
- start working on auto-fixes where possible
- if the coverage of auto-fixes is good enough to result in a consistent style, potentially integrate them into
ruff format, either through a separate style or through the wider community agreeing to adopt these rules.- noting the difficulty above that this would create inconsistency with other code-blocks where mathematical expressions are not used
I understand that step 3 is a very high bar, and that step 1 is a lot of work in itself that is not necessarily going to happen any time soon, but is this a reasonable roadmap/wishlist? Or do you see problems that would make this path impossible?
I think a decision on whether to make this a set of lint rules or integrate it into the formatter is essential.
- The formatter and linter work very differently. Implementing the style guide as lints rule first and then as part of the formatter means duplicated work.
- We would also have to keep maintaining the style guide lint rules because some users started to rely on them and use the flexibility that they can enable rules individually (unless this becomes a single massive rule)
- We don't have a good framework for style-guide like lint rules.
This is in addition to the challenges I mentioned before:
- We decided not to implement rules that conflict with the formatter because formatted code should never raise new lint violations. This property is essential to integrate the formatter into
check. Some of the proposed style guide rules do conflict with the formatter. - The style guide conflicts with existing rules, and we can't just remove them. We have users who depend on them, even if the rules might go beyond what PEP-8 specifies. Arguably, we should never have added those rules,, but here we are.
- Implementing a formatting style guide is a multi-week, if not multi-month, project. It also increases the complexity to push any new formatting changes because they now need to fit into and be tested with multiple styles guides.
- I'm very hesitant about introducing a new style guide into the formatter because it would undo some of Black's accomplishments in establishing a widely agreed upon style guide.
These decisions above aren't set in stone, but they exist to provide a good user experience. We may be able to lift some of those restrictions once we're done with rule categorization or when Ruff supports plugins. But we have yet to start working on these projects. We could decide that this style guide is so essential that it is worth regressing the experience for some users. I'm not familiar enough with the scientific Python community to make that case, but you may have some data to demonstrate the importance.
That's why I recommend you consider a more incremental approach. Is there a limited set of rules that would significantly improve your experience and don't conflict with the formatter or existing lint rules? Are there changes we could make to Ruff's formatter style guide that would significantly improve your experience and that the majority of non-scientific users would also perceive as an improvement?
I'm sorry that I can't give you better news. I know Ruff's formatting for the scientific community isn't great, and I would love to improve your experience. Unfortunately, we have to figure out how this all fits into a unified toolchain without dividing the ecosystem, regressing the experience of existing users, and creating a too-big maintenance burden.
thanks for the explanation @MichaReiser ! I'm not really sure what I think right now. maybe @jarrodmillman @stefanv have thoughts?
I’m sorry I made you feel that way. I reread my message, and I can see that I should have phrased it differently. It doesn’t give you a path forward; it only outlines the problem. I did phrase it so defensively because I can't promise that Ruff will be able to support the specification as is, at least not in the short-term.
A possible next step is to split this proposal into smaller pieces. For example, the style guide specifies when expressions should be parenthesized but also touches on operator spacing. Maybe we can even go more granular than this.
I think we could probably implement a lint rule for parentheses handling without much problem because I believe it isn’t incompatible with the formatter or any existing lint rule. We could even explore if it’s worth implementing it as part of the formatter.
Getting the operator spacing into ruff will probably be more challenging because of the concerns that I outlined in my previous response. Maybe we can implement parts of it. Some parts may require more extended discussions and possibly compromises. There’s also the chance that we aren’t in a position to support some of it today.
To phrase it differently. The next step is to take this specification and make concrete proposals for how to extend (new lint rule, changed lint rule, changed formatting…) Ruff. I can try to help you with this.
Sounds like we need to get this in to move forward. @mdhaber @lucascolley are you ok with that? Note that the process for SPECs is changing. We can, and should, get this in now and it would be marked as a draft at minima.
This specification didn't leave me alone this night. My brain kept thinking about it :)
Following a few questions and additional knowledge that I learned from working on formatters. I hope you find them useful.
- How do you identify mathematical expressions? Does it solely depend on the use of an operator (compare, binary, ...)? Does that mean that the formatting rules also apply to nonnumerical types that overload any of those operators? For example, what about
path / other / "folder"wherepathis apathlib.Path? I advise against relying on type information for formatting because it would mean that changing a symbols type can lead to formatting changes to any file referencing that symbol. - One goal of Black's and Prettier's formatting style is to be predictable and easy to understand (to avoid cases where you're like: Uhhm, why did it do this?). That's why they try to avoid context-specific formatting where e.g. an expression gets formatted differently when embedded in a different context (exceptions apply). What I understand from what's proposed for the operator spacing formatting is that it depends on its enclosing context. There are exceptions to this and it may make sense to make an exception for expression formatting too
- Have you looked at how other languages like Rust, Dart, JS, format mathematical expressions?
- My biggest concern with Ruff's and Black's formatting is the nested expression formatting:
I think you're proposal addresses this to some extend by requiring more parentheses. I would have to do some digging to find the conversation again. But I do remember that there's one case where adding parentheses around a binary expression may change the runtime semantic if the dunder method does something unholy. That might impose a challenge for implementing the spec as part of the formatter.( aaaaaaaaa + bbbbbbbbbbbb * ccccccccccccccc * ddddddddddddddd * xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx + yyyyyyyyyyyy + zzzzzzzzzzzzzz )
How do you identify mathematical expressions? Does it solely depend on the use of an operator (compare, binary, ...)?
I think it would depend solely on the use of the operators. In that example of file paths, the formatting would be the same with these rules as what you've shown - spaces around the division operator since it is a "simple" (as opposed to compound) expression. There may be cases in which these rules would suggest funny things for overloaded operators, but my guess is that in most cases, it would just mean a # noqa here and there.
One goal of Black's and Prettier's formatting style is to be predictable and easy to understand (to avoid cases where you're like: Uhhm, why did it do this?). That's why they try to avoid context-specific formatting where e.g. an expression gets formatted differently when embedded in a different context (exceptions apply).
Sure. There are costs to the simplicity, though. PEP8 explicitly recommends "If operators with different priorities are used, consider adding whitespace around the operators with the lowest priority(ies)". For the sake of simplicity (presumably), Black/E225 just ignore this recommendation. The downside is that some find this makes the resulting expression more difficult to read. The goal of this SPEC is to capture the spirit of the PEP8 recommendation in rules. They will not be as simple as "add whitespace around all operators", but some may find that the benefits outweigh this cost.
Have you looked at how other languages like Rust, Dart, JS, format mathematical expressions?
- Rust rules are here. e.g. "Do include spaces around binary ops." "Use parentheses liberally; do not necessarily elide them due to precedence."
- Dart rules are here. e.g. "Spaces around binary and ternary operators." "Place binary operators on the preceding line in a multi-line expression." "No spaces around unary operators."
- JS rules are here. e.g. "a single internal ASCII space also appears in the following places only...4. On both sides of any binary or ternary operator."
So, for the whitespace question, they all have essentially the same recommendation as Black and E225: always surround operators with whitespace. Others may have different goals, but one of my motivations here was to avoid that blanket rule. I think one of the reasons $LaTeX$ is preferred by many for mathematical documents is it goes to some lengths to make the whitespace look (subjectively) "good". We aren't going going that far here (and can't, as we're assuming monospaced font), but it's a small step in this direction.
My biggest concern with Ruff's and Black's formatting is the nested expression...
Because of 8) "If line breaks must occur within a compound subexpression, the break should be placed before the operator with lowest priority." and "If there are multiple candidates, include the break at the first opportunity.", these (and some other variations) would satisfy the rules prescribed here:
(aaaaaaaaa
+ bbbbbbbbbbbb*ccccccccccccccc*ddddddddddddddd*xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx
+ yyyyyyyyyyyy + zzzzzzzzzzzzzz)
(
aaaaaaaaa
+ bbbbbbbbbbbb*ccccccccccccccc*ddddddddddddddd*xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx
+ yyyyyyyyyyyy
+ zzzzzzzzzzzzzz
)
We can, and should, get this in now and it would be marked as a draft at minima.
Is this ready to go in as draft @stefanv ?
@lucascolley Thanks for the ping. Yes, it's definitely in good enough shape to shop around for discussion.
Note that there is one unclosed LaTeX expression, which would be good to fix first.
Some personal remarks:
- It feels like there could be more clarity between (1) and (5) around when brackets may be / should be inserted to improve an expression.
a + x*y**3is rather hard to parse for my eyes, would probably have preferreda + x * y**3, but could be just me, and I suppose we'll get used to it. It's kind of funny to see how the pendulum swings. In the early days, everything was compressed:a+x*y**3, then spaces started creeping in, eventually expressions were fully spaced:a + x * y ** 3. Now the tide is rolling back again :)- The definition section is a bit daunting, with all the descriptions of expressions, subexpressions, etc.; may be worth thinking about how to simplify that introduction, or move it to the end (most of the language is fairly intuitive, where it is used, so should be OK to have a glossary for reference?)
- Intro: "This leads to individual interpretation and styles which may conflict with those of others."; that sentence can perhaps be clarified along the lines of: "This leads to varying, even conflicting, mathematical expression styles across the ecosystem."
Anyone on here should feel free to move the PR out of draft status and merge, when they're ready. Or give me the thumbs-up and I'll do so.
a + x*y**3 is rather hard to parse for my eyes
I shared this concern, but after speaking to Matt I think it's fine considering the specification in (5) that there must be just a single ** in the rightmost position if there is one. I think it's difficult to parse only because I'm used to expecting the operators to appear in any order, rather than having all expressions formatted in a standard way.
And I think the rules don't forbid developers to include extra parentheses in cases like this if they would like to be extra clear.
Thanks again @MichaReiser for the points you raised above. I've had a little more time to think about them and I think you did a great job of explaining the challenges of adopting this in Ruff.
- We decided not to implement rules that conflict with the formatter because formatted code should never raise new lint violations. This property is essential to integrate the formatter into check.
- Some of the proposed style guide rules do conflict with the formatter. The style guide conflicts with existing rules, and we can't just remove them. We have users who depend on them, even if the rules might go beyond what PEP-8 specifies. Arguably, we should never have added those rules,, but here we are.
One idea that I'm not sure has been considered yet would be a completely separate mode of the formatter + linter. The start point would be implementing this specification as lint rules and being able to format code to comply with it, but in isolation (just this spec). If that could be achieved, then any rules and formatting that do not conflict with this spec could then be added into this separate mode.
This separate mode would never be able to offer a superset of ruff's regular functionality. But it could still be enough for us.
- Implementing a formatting style guide is a multi-week, if not multi-month, project. It also increases the complexity to push any new formatting changes because they now need to fit into and be tested with multiple styles guides.
Of course, the main problem is always time. At least with this suggestion, the separate mode wouldn't block new formatting changes from integrating with "normal" ruff. That work would have to happen at some point if we wanted those changes in the separate mode, but in theory it could be done at a way later point.
- I'm very hesitant about introducing a new style guide into the formatter because it would undo some of Black's accomplishments in establishing a widely agreed upon style guide.
Makes sense, although the context for this discussion is that Black’s accomplishments have been largely unable to reach the Scientific Python community due to concerns like this and the formatting of arrays. So we would at least be extending past Black’s accomplishments in some ways :)
Now I can see a lot of arguments for why a completely separate mode like this wouldn’t belong in Ruff’s API but be a separate package which uses Ruff’s machinery under the hood. Maybe that is what we’re looking for.
Still, I think there would be value in translating the non-conflicting parts of this spec to additional lint rules. I’m just trying to be ambitious :)
Note that there is one unclosed LaTeX expression, which would be good to fix first.
Done.
It feels like there could be more clarity between (1) and (5) around when brackets may be / should be inserted to improve an expression.
Can you give an example expression?
a + x*y**3 is rather hard to parse for my eye
Perhaps you're happy with Black's rules for mathematical expressions, and that's fine. I got the sense that others weren't, though, so the rules here attempt to balance the conflicting desires:
- include whitespace, which makes it easier to distinguish multi-character variables from one another and from operators, and
- remove whitespace to visually reinforce the order of operations (as typeset math does and PEP8 calls for)
without requiring the addition of lots of parentheses in expressions involving the common PEMDAS operators.
When I type $a+x \cdot y^3$, it renders as $a+x \cdot y^3$, adding more space around the addition than the multiplication, and none for the exponentiation. (And of course, we usually omit the $\cdot$ operator unless there's a reason to add it for emphasis, so $xy$ usually renders without any space.) There might be something to be said for including different amounts of whitespace for different priority operators, e.g. x + y * x**3, and I'd be happy to suggest a set of rules based on that. But assuming that's not in the cards and we only get one whitespace around operators, I preferred to use it to show that the multiplication happens before the addition even though the addition operation appears first on the line.
That's the rationale, but other viewpoints (e.g. "whitespace is only omitted around the highest-priority operator of a compound expression") are perfectly valid, too, so I don't think we'll be able to reason out which is best. I think it may be a matter of either adopting a mostly-baked proposal like this one or putting forth equally complete alternatives and voting. Or maybe we accept something like this as a draft and vote on specific changes like that.
The definition section is a bit daunting...
Ah, right, Pamphile mentioned that before, and we discussed moving it to a reference section at the end. I'll do that.
Intro...
Yeah, I'll take another look at that.
And I think the rules don't forbid developers to include extra parentheses in cases like this if they would like to be extra clear.
The first rule is "do not add extraneous parentheses", but the last is "Any of the preceding rules may be broken if there is a clear reason to do so." So if the rules lead you to an expression that is particularly difficult to parse, break them. But a + (x * y**3) is not recommended because the rules would permit a + x*y**3.
(Another set of rules I considered would be based on "Whitespace is only omitted around the highest-priority operator of a compound expression" and something like "Expressions with more than two operator priority levels must make subexpressions with two operator priority levels explicit.* This would mean you would have to write a + (x * y**3). But I didn't think people would want all those parentheses.)
The first rule is "do not add extraneous parentheses", but the last is "Any of the preceding rules may be broken if there is a clear reason to do so."
I guess I was alluding to the possibility that a formatter wouldn't add parentheses by default, but also wouldn't take them away if they are already there.
- It feels like there could be more clarity between (1) and (5) around when brackets may be / should be inserted to improve an expression.
Sorry, that should have been (0) and (5).
It's not super intuitive to me when you can / should / must use brackets; but it may well be that the rules are comprehensive.
Wouldn't it be OK to always allow stylistic brackets, for aesthetic grouping, or grouping that is logical to the author?
[This discussion is not a blocker to merging the PR, btw.]