syntax: allow word iteration loops to keep all words in separate lines

[macau23]

Sometimes I have a very long for loop with lots of long items:

for i in aaaaaaaaaa bbbbbbbbb cccccccccccc dddddddddddddd eeeeeeeeeeee ; do

to keep it readable and sorted, I split them per-line:

for i in \
 aaaaaaaaaa \
 bbbbbbbbb \
 cccccccccccc \
 dddddddddddddd \
 eeeeeeeeeeee \
; do

shfmt reformats it to this:

for i in \
 aaaaaaaaaa \
 bbbbbbbbb \
 cccccccccccc \
 dddddddddddddd \
 eeeeeeeeeeee; do

which makes sorting it harder (the ";do" gets moved to the wrong place).

Could shfmt support not doing that for long for loops where the first line uses an "in $"?

[mvdan]

This is an interesting request, but would we force the do to be on a separate line in the case where in is at the end of the line, or would we simply relax the formatting to allow do to be in either of the two places?

We're in a bit of a tough spot because the first change would be a breaking change (new shfmt v3 versions would contradict previous v3 versions), but the second change makes the formatting less strict/consistent, which isn't a good thing either.

[mvdan]

Here's an alternative:

sorted=(
    aaa
    bbb
    ccc
)
for i in "${sorted[@]}"; do
    ...
}

There's an extra step, but in my opinion the syntax is far nicer - no backslashes, and no need to worry about the in and do tokens. I don't think Bash as a language allows doing anything like this directly in a for loop.

[macau23]

Fine by me. I would like it supported in shfmt but if it's a breaking change then I will use your alternative.

[dseynhae]

I almost entered this as a new issue. I left it below as initially typed it up, to illustrate that this would indeed be nice to resolve... Feedback to @mvdan 's question: I would put the ;do on a new line, because:

1. It helps to keep the lines short
2. It allows to process individual list items more easily (adding or removing at the end)

Feedback to @mvdan 's alternative: I think the point of a format tool is to be able to only use white-space to update the document. We should never change the functional content, to create the desired format. (So a format tool should not come up with an extra variable, to create the "nicer format")

Therefore, the solution should be a manipulation of content, as specified by the developer...

TL;DR

I think that this might not only be a specific rule for select and for (although it certainly would be nice!), but a more general rule for the developer:

If you think that you know better than the shfmt tool, then use the continuation character '' to indicate your line-splitting intent. It will be respected by shfmt ... (indentation can still be overwritten)...

If people can agree on that, the title of this request would change once again...

Background

Consider the following code:

#!/bin/bash
select o in \
	o1\
	o2\
	o3\
	; do
	echo $o
done

This illustrates one of many situations, where I take advantage of the continuation character \ to make my code more readable, and maintainable (it is much easier to add or remove elements to the list).

Similar situations are there for for loops, and other commands that process lists...

Problem

My careful formatting is undone by shfmt:

#!/bin/bash
select o in \
	o1 o2 o3; do
	echo $o
done

Request

Frankly, I think that my coding style is the better one:

• It avoids that my lines bump out of the reasonable 80 character range
• It makes the list more readable and maintanable

So maybe shfmt should impose this as the "formatted" select, for,... solution?

If that causes concern or objections, then I suggest that formatting respects the continuation line character \ (i.e: never remove it). Consider it a manual overwrite for the user to say "I know better"...?

[mvdan]

So maybe shfmt should impose this as the "formatted" select, for,... solution?

As I explained in my earlier comment, I don't think we can do this anytime soon. It would be a breaking change in v3, and I don't have plans for a v4 right now. I also don't think it would be a good idea in general; forcing backslashes would likely upset a good portion of the users.

then I suggest that formatting respects the continuation line character \ (i.e: never remove it). Consider it a manual overwrite for the user to say "I know better"...?

That's the second option I was considering, but I was a bit on the fence. I generally agree with you that having to use an extra variable with an array is not very good. I still think that the backslash solution is somewhat hacky, but I guess that's just the shell language in general :)

Does someone want to take a crack at this second option? That is:

1. allow "word iteration" words to be on separate lines
2. allow the "do" token to be on a separate line if the "in" token was similarly followed by a newline

[dseynhae]

I also don't think it would be a good idea in general; forcing backslashes would likely upset a good portion of the users.

I'm not sure about that: formatting per definition is playing with whitespaces, over the boundaries of EOL, to make the code more readable (which often includes to keep it under 80 characters), easier to maintain, and consistent.

If you look at formatters for C, Perl,..., they go as far as breaking up regular expressions, concatenations,... to achieve that goal (PerlTidy even breaks up expressions over multiple lines based on operator priorities...).

However, with Sh/Bash, we're in a situation where we format a language that is interpreted, and hence needs a continuation character to split a statement over lines. But that shouldn't stop us from splitting over lines, in order to achieve the primary goals!

I pose that people who don't like continuation characters, aren't looking for a decent formatter in the first place... But they can be accommodated with a non-default switch to the formatter, that orders shfmt "never to split lines"...

It would be a breaking change in v3.

My apologies, I'm just familiarizing myself with shfmt, so I'm not sure what the change means. I assume that it implies that the ;do should be kept on the same line as the statement at all cost. But couldn't that be changed to: keep it on the same line, IF there is only one item in the list (e.g. a glob)?

[mvdan]

But they can be accommodated with a non-default switch to the formatter, that orders shfmt "never to split lines"...

That also comes with its cost. The more formatting options the tool has, the harder it is to maintain, test, and use.

I assume that it implies that the ;do should be kept on the same line as the statement at all cost.

Not quite; see the two changes I explained at the end of my last comment. Both of those forms would be obeyed by the formatter:

# single-line form, what's done now
for x in foo bar baz; do
    echo $x
done

# many-line form; allowed as long as the iteration words are all separated from "in" and "do" by newlines
for x in \
    foo \
    bar \
    baz \
    ; do
    echo $x
done

[dseynhae]

The more formatting options the tool has, the harder it is to maintain, test, and use.

Very good point on minimizing the command-line options! I even have the controversial attitude that I won't touch ANY "options" to a formatter, so that I'm guaranteed consistent with anybody else in our organization (as long as they don't touch those options either). And I'm hoping that the default for any option is a default that will please "the majority". But as far as I am concerned, even "no options" is good for a decent formatter.

Both of those forms would be obeyed by the formatter:

"would", implying that the second option still needs implementing, right?

[mvdan]

"would", implying that the second option still needs implementing, right?

Correct, hence the two changes proposed earlier as a way to implement it.

[zackw]

N.B. the proposed alternative is no good for -p mode (i.e. for scripts that must remain free of bashisms).

[mvdan]

Indeed, it's far from a direct replacement, but more of an alternative depending on one's situation.

[macau23]

Close?

[dseynhae]

I don't mind formatting depending on the shell-type. But the more I think about it, the more I would object to "options" to steer the formatting in the desired direction.

When coding in Python, I'm using the Black formatter. No options. To the extreme. That way, no patch by other designers will change the formatting. And that in turn makes the review of the code so much easier.... Plus the style is now completely predictable. I'm assuming that a coding style only gets implemented if the majority of the designers agree. How would I pretend to know better than the majority, and insist on an option to get it my way... This is in the face of the original intention to make the code easier to read and maintain. By everybody...

For the proposal here, it seems that there is no objection that it would be an improvement, appreciated by the people who care. Therefore, I would love to hang on to the request (not close it).