proposal-pipeline-operator icon indicating copy to clipboard operation
proposal-pipeline-operator copied to clipboard

Deeply nested expressions A(B(C(D(E(…), …), …), …), …) vs. temp variables

Open WillAvudim opened this issue 8 months ago • 29 comments

Unrelated

@tabatkins promised to get on with it back in August 2024 - https://github.com/tc39/proposal-pipeline-operator/issues/306 Well, it's April 2025 already, and I'm churning a lot of typescript nowadays, littered with A(B(C(D(E(param))))). Having the pipeline operator would definitely help a lot, I'm going nearly blind just from counting the parenthesis alone.

Please, guys, let's make it happen.

WillAvudim avatar Apr 07 '25 21:04 WillAvudim

I propose we feed o1-pro with all the relevant information and let it decide as we humans can't even choose a topic token. I may be kidding or not.

gustavopch avatar Apr 07 '25 21:04 gustavopch

  • Hey, thanks for your interest in the pipe operator. I’m still hopeful for it, any pipe operator, too. Every day that I write JavaScript (or work in a language with a pipe operator), I wish for the pipe operator to be done.
  • Coincidentally, I’d been drafting an update post for the past week. I just posted it to https://github.com/tc39/proposal-pipeline-operator/issues/232#issuecomment-2784879225.
  • There’s been a lot of stasis, then a little recent movement in the past few months. For example, I had almost no volunteer time over the past four years to work on TC39 until recently.
  • The proposal isn’t dead, but if it succeeds, it won’t be done for another year, if not two or even more.
  • There’s movement in related proposals that might or might not free up # as the topic. But there may also be complications from proposed changes to the language in JS0/JSSugar. Read the update for more details.
  • I’m going to close this issue, since I don’t think there’s anything actionable here. I hope that makes sense to you.
  • If you read the update and have any questions, please feel free to open a new issue.

js-choi avatar Apr 07 '25 23:04 js-choi

I'm churning a lot of typescript nowadays, littered with A(B(C(D(E(param)))))

Actually, I’m interested in this part. One objection that some people in the Committee has given to any pipe operator is that “temporary variables might be good enough”.

  • I have continued to argue that the explainer’s real-world examples (and, indeed, any real-world codebase) show clear tendencies by developers to create poorly readable deeply nested expressions, instead of reaching for temporary variables.
  • Indeed, this has been the impetus for another potential usability study by TG5. See the general proposal update and #216 for more information.
  • So…why don’t you just assign to temporary variables in your code “littered with A(B(C(D(E(param)))))”?
    • I think I know why—I live through it myself—but I would love to hear experiences from other people that I would be able to take to the Committee.
  • Also, how often do you have n-ary calls in those deeply nested expressions?

js-choi avatar Apr 08 '25 00:04 js-choi

  • I have continued to argue that the [explainer’s real-world examples][] (and, indeed, any real-world codebase) show clear tendencies by developers to create poorly readable deeply nested expressions, instead of reaching for temporary variables.
  • Indeed, this has been the impetus for another potential usability study by TG5. See the general proposal update and Role of usability studies in the explainer #216 for more information.
  • So…why don’t you just assign to temporary variables in your code “littered with A(B(C(D(E(param)))))”?

Today I've been rewriting sqlite queries to work against in-memory representation of data, stored in arrays and objects (perf optimization), which means I've got to (repeatedly) do filter -> apply is_null verifications and merge missing data -> group-by -> filter -> apply computation -> iterate over subgroups -> sum up the totals. That is, a fairly simple SQL query translates into 6 (!) transitions, and the names, if you'd like to take the temporary variable route, would be disastrous. How do you imagine it? filtered_values, filtered_values_without_nulls, filtered_values_without_nulls_grouped_by_org, etc? And that's just a single query while I have to convert up to 7 per functionality module. Thing is, temp variables names are meaningless, because essentially you'll end up just repeating the name of the operation you just did, and it's already in the code. So, now I'm forced to duplicate the op names all over the place.

  • Also, how often do you have n-ary calls in those deeply nested expressions?

Oh, I have a function that, given an iterator, runs lambda functions against each of the subgroup. So, it would be a pipeline (multiple ops per subgroup) inside the pipeline over the entire dataset.

WillAvudim avatar Apr 08 '25 01:04 WillAvudim

Thanks for the response.

  • If any of this deeply nested multi-step code is open source, or if you can share simplified versions of the code that you’re allowed to share, that would be valuable.
  • We might even put it on a slide someday—or at least the explainer.
  • What matters most is:
    • that it contains deeply nested expressions,
    • that it’s from the real world,
    • and that you deliberately decided not to use temporary variables in it.

This invitation extends to anyone else too: your thoughts on “just use temp variables” and any real-world code examples.

The more real-world examples to show the Committee just how big of a problem this is, the better.

js-choi avatar Apr 08 '25 02:04 js-choi

I also don't like using temp variables. I'll try to explain how I think.

I like my code to be as close as possible to how the equivalent pseudocode would be. Pseudocode is usually what we intend to do and focuses on essential complexity. Actual code is what we have to do and includes accidental complexity. Temp variables are usually accidental complexity.

Code is for humans. The closest it is to natural language (without being ambiguous of course), the better. We don't name everything we talk about. We frequently use a whole sentence to describe something that doesn't have a specific name. Things that are frequently referred to naturally end up getting a name. I see it the same way in code. I don't think literally everything should be named, sometimes it's more convenient to describe it with a sentence/expression than name it. And, in those cases, naming it would probably produce a name that's basically a redundant description of the expression on the right side of the = as the examples given by @WillAvudim:

filtered_values, filtered_values_without_nulls, filtered_values_without_nulls_grouped_by_org

I see this kind of temp variable, whose names are redundant, just like this kind of comment:

// Increment counter
incrementCounter()

It's not adding any new information. We already know it's incrementing a counter because the statement says exactly that.

In some other cases, it's less about redundancy and more about "better show than tell":

const gradients = [
  from + step * 0,
  from + step * 1,
  from + step * 2,
  from + step * 3,
].map(
  from => `linear-gradient(${direction}, white ${from}%, transparent ${to}%)`,
)

I don't think naming this array would improve the readability. I'm not even sure how I'd name it... gradientFroms? I mean, IMO, not even worth wasting time trying to find a good name. The code is already readable as it is. If I had to make it even clearer, I'd just add some comments to show an example of how the values should flow, like this:

const gradients = [
  from + step * 0, // E.g.: 50 + 10 * 0
  from + step * 1, // E.g.: 50 + 10 * 1
  from + step * 2, // E.g.: 50 + 10 * 2
  from + step * 3, // E.g.: 50 + 10 * 3
].map(
  // E.g.: 50 => linear-gradient(to bottom, white 50%, transparent 100%)
  from => `linear-gradient(${direction}, white ${from}%, transparent ${to}%)`,
)

I try to write my code in a way that, just by reading the start of each statement, I can already understand what's happening. In this case, it's easy to see this statement is producing an array of gradients. If I need to understand how it's doing that, then I read the expression and see it starts with an array of 4 values and maps those values to a template literal.

I'm not saying temp variables are always bad, I'm just saying they're not useful (IMO, of course) when the name isn't meaningful and doesn't add new information that you wouldn't easily obtain by reading the expression — and sometimes comments with examples are better in improving the comprehension than names.

@js-choi BTW, thanks for the updates and explanations in the messages you recently posted here and in the other issues.

gustavopch avatar Apr 08 '25 13:04 gustavopch

I reviewed our project, which is about ~200 KLOC at this point, and I must admit I don't have truly great examples where the pipeline operator would be of substantial benefits. Normally, our code doesn't venture past A(B(C))).

E.g. let's take the error processing lines:

ForkAsync(dispatcher.TerminateDispatcher(new Error(`${dbg} exited with ${JSON.stringify(code_or_signal)}.`)))

It's hard to tell if rewriting them via the pipeline operator would yield more clarity:

`${dbg} exited with ${JSON.stringify(code_or_signal)}.` |> new Error(%) |> dispatcher.TerminateDispatcher |> ForkAsync

Unit test checks would definite become better:

await Match(await a.Read({ index: 0 }), 7)

because you can formulate the operation first, and pipe it into the check, thus clearly separating the op from its expected value:

await a.Read({ index: 0 }) |> await Match(%, 7)

I've made quite a bit of progress on rewriting my sqlite queries, and they are no longer at the point where they could be expressed via the pipeline operator because of micro-optimizations.

SELECT org_name, SUM(budget) AS total_budget
FROM Org
WHERE Org.last_updated < ?update_from
GROUP BY org_name

That roughly translates into:

orgs |> FilterBy(%, org => org.last_update < update_from) |> GroupBy(%, org => org.org_name) |> Map(%, group => group.values => SumBy(%, val => val.budget))

However, in practice |> would result in generation of intermediate arrays after every operation, which is inefficient. What I truly want instead is probably LINQ, where this would be natural - https://learn.microsoft.com/en-us/dotnet/standard/linq/

var sums = 
    from org in Orgs
    where org.last_update < update_from
    group org by org.org_name into g
    select new { org_name = g.Key, total_budget = g.ToList().Sum() }

Here the data is also pipelined from one line to the next, and there is a list of predefined item transformations that mimic SQL operators. C# compiler does a lot of optimizations in order to make it robust, and nearly as fast as hand-written code.

Google has also started doing the same recently: https://cloud.google.com/blog/products/data-analytics/simplify-your-sql-with-pipe-syntax-in-bigquery-and-cloud-logging

-- Pipe Syntax 
FROM `bigquery-public-data.chicago_taxi_trips.taxi_trips`
|> EXTEND EXTRACT(YEAR FROM trip_start_timestamp) AS year
|> AGGREGATE COUNT(*) AS num_trips 
   GROUP BY year, payment_type
|> AGGREGATE AVG(num_trips) AS avg_trips_per_year GROUP BY payment_type ASC;

As you can see, the stream of items is piped through a set of distinctly predefined operations, and the query optimization engine can produce high-perf code that is hard to match if you write everything manually. However, both LINQ and the SQL pipe syntax result in a code that doesn't necessarily flow through separate stages, and the result is compiled as a whole, without emitting intermediate arrays. I don't think we can achieve this with the current pipeline operator proposal. Maybe we are aiming too low?

WillAvudim avatar Apr 08 '25 18:04 WillAvudim

However, both LINQ and the SQL pipe syntax result in a code that doesn't necessarily flow through separate stages, and the result is compiled as a whole, without emitting intermediate arrays.

At least in c#, the reason why it doesn't create intermediate arrays is because IEnumerable is evaluated lazily. In JavaScript you can achieve similar behavior using generators. And for pipelining them through operators you can use libs like IxJS. If you want you could use |> to chain the operators (although with hack pipes I'd prefer the native lib way of chaining...).

Jopie64 avatar Apr 08 '25 20:04 Jopie64

However, both LINQ and the SQL pipe syntax result in a code that doesn't necessarily flow through separate stages, and the result is compiled as a whole, without emitting intermediate arrays.

At least in c#, the reason why it doesn't create intermediate arrays is because IEnumerable is evaluated lazily. In JavaScript you can achieve similar behavior using generators. And for pipelining them through operators you can use libs like IxJS. If you want you could use |> to chain the operators (although with hack pipes I'd prefer the native lib way of chaining...).

Linq can compile into expression trees, e.g.

Func<int, bool> comparer = num => num < 5;

is translated by the C# compiler into

Expression<Func<int, bool>> comparerExpression2 = 
  Expression.lambda<Func<int, bool>>(
    Expression.LessThan(
      numParam,
      Expression.Constant(5)),
    numParam);

And then the executor (e.g. Entity Framework, linq-to-sql) can optimize it, translate into SQL, and compile the statement. It's a whole new level of possibilities! Sources:

  1. https://learn.microsoft.com/en-us/dotnet/csharp/advanced-topics/expression-trees/
  2. https://youtu.be/r69ZxXgOIK4?t=552

It's unfair to compare it to straightforward passing of an object to a function. C# compiler applies custom optimizations, such as merging of Where and Select calls, preemptively stopping iteration for Any/All/First methods, etc. It's aware of the expression tree, and emits modified IL. It doesn't simply invoke IEnumerable methods lazily as one might assume. And they do a lot of smart optimizations underneath precisely because they have this internal representation as an expression tree, and the meaning of individual elements comprising it. While with the pipeline operator you're effectively piping lambdas without any possibility for a query analyzer or an external (non-javascript) optimizer to handle it.

WillAvudim avatar Apr 08 '25 21:04 WillAvudim

Right, for that sort of expression analysis you need a context with bounds so the operations can be lazily transformed. Nothing about that is incompatible with pipeline (or any other function-calling syntax), you'll just be loading up the expressions into a context and then executing it at the end, rather than calling eager transformations.

tabatkins avatar Apr 08 '25 21:04 tabatkins

Right, for that sort of expression analysis you need a context with bounds so the operations can be lazily transformed. Nothing about that is incompatible with pipeline (or any other function-calling syntax), you'll just be loading up the expressions into a context and then executing it at the end, rather than calling eager transformations.

That's true in case of Python:

import torch
a = torch.tensor([2.0], requires_grad=True)
b = torch.tensor([3.0], requires_grad=True)
c = a * b  # The '*' operator is overloaded to add a multiplication node to the graph

Javascript by itself doesn't have the full capacity for the execution graph extraction, in this particular case, we can't overload operator*() to pass that part of the execution graph into the subsequent transformation. You can only extract incomplete pieces without the full compiler support. And the full graph is required for the eager transformation, or you'll be left with partially unknown structure. Well, you can probably construct everything via c = Mul(a, b), but is it Javascript at that point? Lambdas will be opaque functions, we won't be able to optimize the whole construct end-to-end. We need a better support for AST extraction, just passing functions around is not ideal.

WillAvudim avatar Apr 08 '25 22:04 WillAvudim

@WillAvudim:

  • Thank you for sharing your real-world code.
  • Error processing code:
    • I do think that the original version is less clear than the linear, pipeline version.
    • Especially think about if any of these function calls needed more than one argument.
    • Please note that the last two lines of your pipe version would need to also have (%), like |> dispatcher.TerminateDispatcher(%).
  • Test assertions:
    • It’s a good point that test assertions commonly use deep nested calls.
    • This is part of why alternative, method-based assertion APIs like except are so popular: they’re more fluent, and they read much more nicely.
    • But, even in except-style assertions, the test-subject method arguments still often require deeply nested ad-hoc dataflows.
  • Query building and avoiding intermediate arrays:
    • Your SQLite query’s problem with intermediate arrays is present whether or not you use pipeline or nested function calls:
      Map(GroupBy(FilterBy(orgs, …), …), …)
      
    • Like @tabatkins said, an API could build queries with lazily evaluated queries, and these could be easily pipelined with nested function calls or |>.
    • Your pipelined SQLite query orgs |> … does not have to eagerly evaluate each step.
      • FilterBy could return a data structure containing orgs and the filter’s parameters.
      • GroupBy and Map similarly could return data structures containing previous.
      • You would add a final run, execute, or query function call that actually executes the query.
    • Likewise, the BigQuery SQL syntax could also use nested function calls / |> pipelines to build up a lazy query, then call a function that executes the lazy query at the very end.
    • Many query-builder APIs use patterns like this, although using a library-provided piping function designed for query data structures is also common.
    • Either way, there’s nothing intrinsic to |> that requires query pipelines that require eager evaluation of each step into intermediate arrays.
  • LINQ is more complicated:
    • LINQ queries can be turned into pipelines to some extent, but oftentimes they cannot.
    • This is because each step in a LINQ query binds a variable that can be used in later steps.
    • There is a deep relationship between |> pipelines and LINQ queries’ evolution in F#: query expressions and other computation expressions.
    • Specifically, both pipelines and query expressions can both be extended by monads (and functors, monoids, etc.).
    • No abstract syntax trees are needed for this.
    • Pipes could theoretically be extended to “context pipes” that create complex query scaffolding at each step of the pipeline.
      • Such “context pipes” would cover optional pipes (#159 and #198).
      • They would also support custom contexts like LINQ query building.
    • These are just half-formed ideas right now, but you may want to read my very old ES context blocks proposal.
    • This custom “context pipes” idea won’t be able to be proposed for many, many years.
    • As you said, “Maybe we are aiming too low?” but we are limited by the difficulty of adding even simple simple syntax like the current proposal, let alone complex new syntax.
    • This custom “context pipes” idea is off-topic from “nested expressions vs. temporary variables”.
    • I’ve opened a long-overdue new issue about this custom “context pipes” idea in #312.

Circling back to the “nested expressions vs. temporary variables”, everyone is still invited:

  • To (legally) share real-world code that:
    • contains deeply nested expressions,
    • is from the real world,
    • and that you deliberately decided not to use temporary variables in it.
  • …or to just share your thoughts on “just use temp variables” and any real-world code examples.

js-choi avatar Apr 09 '25 05:04 js-choi

@js-choi Bringing some more cases where I avoided using temp variables. These $-prefixed methods are custom methods I added to the prototypes (I use this prefix so it's easy to distinguish from built-in methods). I know modifying prototypes is not considered a good practice, but this is an app, not a lib, and I'm the only one working on it, so there's no risk of conflicts (especially because of the dollar prefix) and I like how fluent the code looks:

const users = await reservations
  .map(_ => _.userId) // I use `_` as a sort of topic token when I consider the name would be redundant
  .$unique()
  .$chunk(30)
  .$mapAsync(ids => getUsers(ids))
  .then(chunks => chunks.flat())

I like the indentation to reflect the level of detail (as in video games). When I'm reading the code from where this snippet was extracted, if I only want to pay attention to the overall logic (to look at the forest, not the trees), the only thing I'll read from this snippet is const users. If I want to analyze in more details (to look at the this specific tree from the forest), then I'll read the indented expression. How this would look like with temp variables:

const userIds = reservations.map(_ => _.userId)
const uniqueUserIds = unique(userIds)
const uniqueUserIdsChunks = chunk(uniqueUserIds, 30)
const userChunks = await Promise.all(
  uniqueUserIdsChunks.map(async ids => await getUsers(ids))
)
const users = userChunks.flat()

Just look at how much redundancy there is. It's no longer possible to distinguish the levels of detail. You have to read the whole code and allocate all those variables in your mind as you're reading this in the middle of the other code from where it was extracted. You don't know if these temp variables are just used in the next line or if they'll be used somewhere else in the lines that are coming. The first version is more efficient for my mental parser and mental garbage collector. It's just more scannable.

Note that I'm not saying people should modify prototypes — it's just a workaround I'm currently using as we don't have a pipe operator. I'd be happy to write it like this:

const users = await reservations
  .map(_ => _.userId)
  |> unique(#)
  |> chunk(#, 30)
  |> await Promise.all(#.map(ids => getUsers(ids)))
  |> #.flat()

gustavopch avatar Apr 09 '25 12:04 gustavopch

@WillAvudim

Linq can compile into expression trees ...

Sorry I wasn't aware you were talking about this C# feature 😅 I was specifically referring to the

... without emitting intermediate arrays

... part, which is what C# also does when you use Linq on IEnumerable. This part can be done in JavaScript as well e.g. with generators. JavaScript doesn't support (AFAIK) compiling mundane JavaScript expressions into some form of AST that you can then further reflect on in the JavaScript code itself. This is a C# feature and also I think this is unrelated to the pipe operator feature. (So a bit of topic I think...)

@gustavopch

... I know modifying prototypes is not considered a good practice, but ... I like how fluent the code looks:

const users = await reservations
  .map(_ => _.userId) // I use `_` as a sort of topic token when I consider the name would be redundant
  .$unique()
  .$chunk(30)
  .$mapAsync(ids => getUsers(ids))
  .then(chunks => chunks.flat())

... I'd be happy to write it like this:

const users = await reservations
  .map(_ => _.userId)
  |> unique(#)
  |> chunk(#, 30)
  |> await Promise.all(#.map(ids => getUsers(ids)))
  |> #.flat()

In JavaScript you can already write a ramda-style pipe function to make it fluent without resorting to new syntax or adding to prototype. Then it could become something like this:

const users = await pipeAsync( // your own pipe function or maybe from fp-ts or something
  reservations,
  R.map(_ => _.userId), // or R.map(R.prop('userId'))
  R.uniq,
  chunk(30), // your own function, or from some other lib
  R.map(getUsers),
  Promise.all,
  R.flatten);

Jopie64 avatar Apr 09 '25 14:04 Jopie64

@Jopie64 Yeah, indeed. I prefer to modify the prototype because it feels to me closer to natural/idiomatic JS.

Explanation

Putting this in a spoiler tag because it's kinda off-topic. I feel evolving like this:

- const result = foo(input)
+ const result = foo(input).bar()

is nicer than this:

- const result = foo(input)
+ const result = pipe(input, foo, bar)

And I don't like the idea of deviating completely from how JS is usually written. It feels like fighting the language, over-engineering. I also avoid analysis paralysis as much as possible, so I don't like to have multiple ways to do the same thing, which is why I prefer to just use Array.prototype.map than something like R.map — which either requires me to completely change how I write my code or mix two ways. But this all is really just personal preference. I only brought the prototype and _ opinions to keep the examples as close as possible to my real-world code. I'd be happy with this though:

- const result = foo(input)
+ const result = foo(input) |> bar

Or maybe I'd always try to start with a pipe in the simple case so the code evolves like this:

- const result = input |> foo
+ const result = input |> foo |> bar

gustavopch avatar Apr 09 '25 14:04 gustavopch

Another example I think could prove useful. The following snippet populates an object prototype with an accessor to a Float32Array embedded in a tensor, with dimensions specified externally:

Object.defineProperty(ro_prototype, stream_name, {
  configurable: false,
  numerable: true,
  get: function (this: any) {
    // (tick * width) is computed twice:
    return (tick: number) => this[tensor_property_name].subarray(tick * width, tick * width + width)
  },
})

With the pipeline operator I could do the computation once without declaring an extra variable, which would otherwise require me to declare a block with an extra variable:

get: function (this: any) {
  return (tick: number) => tick * width |> this[tensor_property_name].subarray(%, % + width)
},

VS w/o the pipeline operator:

get: function (this: any) {
  return (tick: number) => {
    const from: number = tick * width
    return this[tensor_property_name].subarray(from, from + width)
  }
},

WillAvudim avatar Apr 09 '25 23:04 WillAvudim

I like the indentation to reflect the level of detail (as in video games). When I'm reading the code from where this snippet was extracted, if I only want to pay attention to the overall logic (to look at the forest, not the trees), the only thing I'll read from this snippet is const users.

Playing the devil's advocate, this pattern can be met with (maybe future) do expressions and temp vars, too :


const users = do {
  const userIds = reservations.map(_ => _.userId)
  const uniqueUserIds = unique(userIds)
  const uniqueUserIdsChunks = chunk(uniqueUserIds, 30)
  const userChunks = await Promise.all(
    uniqueUserIdsChunks.map(async ids => await getUsers(ids))
  )
  give userChunks.flat()
}

fuunnx avatar Apr 10 '25 08:04 fuunnx

with (maybe future) do expressions

"Do expressions" proposal is stage 1 with no activity in the last 4 years. In other words, it's confirmed dead.

WillAvudim avatar Apr 10 '25 12:04 WillAvudim

@fuunnx True. In fact, I use IIFEs a lot too (as we unfortunately don't have do expressions). Benefits:

  • Indentation as an indicator of level-of-detail.
  • Simpler variable names because of the scope isolation. E.g.: when I start reading that scope, I already know all variables inside are temp variables related to "users", so I find it clear enough to write just ids there instead of userIds.

Usually, if I consider a certain logic is general enough to be added to... say... ESFuture, then I add the method to the prototype myself and pretend I have the privilege of using that future ES version today. Otherwise, if it's a very specific logic, not general enough, I use an IIFE to emulate a do expression.

IIFEs / do expressions still require significant redundancy in the temp variable names though.

Fun fact: another thing I'm not currently doing, but did in the past to emulate a pipe operator was adding a pipe-like method to Object.prototype, leveraging the fact that almost everything is an instance of Object in JavaScript, so I'd write it like this:

const users = await reservations
  .map(_ => _.userId)
  .$(_ => unique(_))
  .$(_ => chunk(_, 30))
  .$(_ => Promise.all(_.map(ids => getUsers(ids))))
  .then(_ => _.flat())

Note: I could call it $pipe, but $ is as concise as it gets, and I can more easily make my brain not pay attention to it. I mean, when we read this snippet, we mainly pay attention to the words and numbers (e.g.: map, userId, unique, chunk, 30, etc.), so naming it $ makes my brain parse it the same way as ., ( and ), in a kind of "different channel/frequency" that doesn't add noise to the "main channel/frequency".

Waiting for the JavaScript police at my front door now that I've confessed all these crimes. :P

gustavopch avatar Apr 10 '25 12:04 gustavopch

"Do expressions" proposal is stage 1 with no activity in the last 4 years. In other words, it's confirmed dead.

@WillAvudim

  • I would gently push back and say that this comment is inaccurate.
  • The do-expressions proposal is not dead, and https://github.com/tc39/proposal-do-expressions/issues/79#issuecomment-2676921351 did not confirm it to be dead.
  • That proposal is just dormant, just like how this pipe operator proposal and match expressions are dormant.
  • A key barrier is how chilly the Committee has been toward any new syntax due to the complexity it brings. See my update in https://github.com/tc39/proposal-pipeline-operator/issues/232#issuecomment-2784879225.
    • So we (and the do expression champions and the match expression champions) have mostly been focusing their free time on “lower-hanging fruit” proposals in TC39 or other standards venues—proposals that don’t involve new syntax.
    • This is especially complicated nowadays by a potential future JS0/JSSugar language split, which would separate new “syntax sugar” features from the core language.
    • Pipe operator, do expressions, and match expressions would all be affected by such a big change.
    • That’s one big reason why we’re focusing energy on other, less uncertain features like new standard function APIs.
  • do expressions (just like pipe operator and match expressions) are not dead. They’re just dormant.
  • Hopefully we’ll be able to progress one of those three proposals sometime, but we are volunteers and have a lot of other stuff to focus on too.
  • An example of an actually dead proposal is operator overloading, which was formally withdrawn in 2023 due to inexorable performance/optimization concerns from the engine implementors.

js-choi avatar Apr 10 '25 16:04 js-choi

Temporary variables break the 'flow'. When I'm solving a data transformation problem with code, suddenly having to extract code and having to tackle an impromptu "name the variable"-problem in the middle risks breaking my concentration with the context switch.

Pokute avatar Apr 11 '25 22:04 Pokute

Because Javascript currently doesn't have the pipe operator, we (Javascript developers) do not structure our code to exemplify how it could be useful. However, if you take a look at a real F# codebase, which does support the pipe operator, you'll see that it's used universally throughout the codebase all the time:

AccountOperations.fs AccountRepository.fs

That could be a much better example. I used to write in F#, and it indeed felt very natural to use |>. I took it for granted, and I embraced pattern matching with algebraic data types. Then I switched to TypeScript, and my style has changed: I now have to use constructs available in JavaScript, and the resulting code is written in a very different manner.

WillAvudim avatar Apr 13 '25 23:04 WillAvudim

Just had a new case here where I'd like to use a pipe. Basically, I have an array of topics and I want to sort them in a particular way.

Using temp variables (342 characters):

const topicsInDescLength = topics.toSorted((a, b) => b.length - a.length)
const topicPairs = []

while (topicsInDescLength.length > 0) {
  topicPairs.push([topicsInDescLength.shift(), topicsInDescLength.pop()].filter(Boolean))
}

const shuffledTopicPairs = shuffle(topicPairs).map(pair => shuffle(pair))
const sortedTopics = shuffledTopicPairs.flat()

Using an IIFE so I can write simpler, scoped names (326 characters):

const sortedTopics = (() => {
  const inDescLength = topics.toSorted((a, b) => b.length - a.length)
  const pairs = []

  while (inDescLength.length > 0) {
    pairs.push([inDescLength.shift(), inDescLength.pop()].filter(Boolean))
  }

  const shuffledPairs = shuffle(pairs).map(pair => shuffle(pair))
  return shuffledPairs.flat()
})()

Using my pipe-like Object.prototype.$ (292 characters):

const sortedTopics = topics
  .toSorted((a, b) => b.length - a.length)
  .$(topics => {
    const pairs = []

    while (topics.length > 0) {
      pairs.push([topics.shift(), topics.pop()].filter(Boolean))
    }

    return pairs
  })
  .$(pairs => shuffle(pairs))
  .map(pair => shuffle(pair))
  .flat()

Using the Hack style pipe operator (262 characters) — if I really understand how it would look like:

const sortedTopics = topics
  .toSorted((a, b) => b.length - a.length)
  |> (() => {
    const pairs = []

    while (#.length > 0) {
      pairs.push([#.shift(), #.pop()].filter(Boolean))
    }

    return pairs
  })()
  |> shuffle(#)
  .map(pair => shuffle(pair))
  .flat()

Well... is "smaller files and all the implications like faster internet and less pollution, etc." already a motivation of this proposal? 😄

gustavopch avatar Apr 17 '25 17:04 gustavopch

This article and a Hacker News thread that came from has a good discussion about variable shadowing / reassignment, in which temporary variable(s) with the same name is used as a pipeline:

Highlights:

Shadowing is the act of using the same name for two different bindings available in a scope hierarchy. Rust is an excellent example of how to do shadowing correctly. […] This lexical scoping in Rust prevents tons of boilerplate or having to come up with useless names. […] This is maybe a sign of the design choice (no move semantics), but all in all, I don’t like having to come up with either obscure names or just bad names for something that should be chained calls, or just reusing the same name. Sometimes, naming things is something we should refrain from.

When did shadowing become a feature? I was under the impression it's an anti-pattern. As per the example in the article

const foo = Foo.init(); const foo2 = try foo.addFeatureA(); const foo3 = try foo.addFeatureB();

It's a non issue to name vars in a descriptive way referring to the features initial_foo for example and then foo_feature_a. Or name them based on what they don't have and then name it foo. In the example he provided for Rust, vars in different scopes isn't really an example of shadowing imho and is a different concept with different utility and safety. Replacing the value of one variable constantly throughout the code could lead to unpredictable bugs.

Having variables with scopes that last longer than they're actually used and with names that are overly long and verbose leads to unpredictable bugs, too, when people misuse the variables in the wrong context later.

When I have initial_foo, foo_feature_a, and foo_feature_b, I have to read the entire code carefully to be sure that I'm using the right foo variant in subsequent code. If I later need to drop Feature B, I have to modify subsequent usages to point back to foo_feature_a. Worse, if I need to add another step to the process—a Feature C—I have to find every subsequent use and replace it with a new foo_feature_c. And every time I'm modifying the code later, I have to constantly sanity check that I'm not letting autocomplete give me the wrong foo!

Shadowing allows me to correctly communicate that there is only one foo worth thinking about, it just evolves over time. It simulates mutability while retaining all the most important benefits of immutability, and in many cases that's exactly what you're actually modeling—one object that changes from line to line.

When you have only one foo that is mutated throughout the code you are forced to organize the processes in your code (validation, business logic) based on the current state of that variable. If your variables have values which are logically assigned you're not bound by the current state of that variable. I think this a big pro. The only downside most people disagreeing with me are mentioning is related to ergonomics of it being more convenient.

It’s a trade-off. If you allow shadowing, then you rule out the possibility of the value being used later. This prevents accidental use (later on, in a location you didn't intend to use it) and helps readability by reducing the number of variables you must keep track of at once. If you ban shadowing, then you rule out the possibility of the same name referring to different things in the same scope. This prevents accidental use (of the wrong value, because you were confused about which one the name referred to) and helps readability by making it easier to immediately tell what names refer to.   Shadowing is a super important complement to Rust's immutability. Without it immutability would be less useful and therefore less used.

Shadowing always has been a feature […]. It is a promise to the reader (and compiler) that I will have no need of the old value again.

Shadowing is a feature. It's very common that given value transforms its shape and previous versions become irrelevant. Keeping old versions under different names would be just confusing. With type system there is no room for accidental misuse.

This is relevant because mutating a single let/var variable is yet another more-verbose but currently possible alternative to syntactic pipelines using |>. (Unlike in Rust, in JavaScript, lexical shadowing is not possible on the same const variable without creating a new lexical block.) In contrast, the pipe operator in this proposal would shadow the pipe topic with each new step in its pipeline, with similar benefits that lexical shadowing in Rust has.

js-choi avatar May 05 '25 20:05 js-choi

This is relevant because mutating a single let/var variable is yet another more-verbose but currently possible alternative to syntactic pipelines using |>.

When using TypeScript though, you can't change the type of the variable after it's declared, so this alternative is only really viable for vanilla JS.

gustavopch avatar May 05 '25 22:05 gustavopch

When using TypeScript though, you can't change the type of the variable after it's declared, so this alternative is only really viable for vanilla JS.

Why add new syntax to js, to avoid a limitation in ts? Isn't it ts that needs to change in such a case?

Jopie64 avatar May 06 '25 05:05 Jopie64

The reassignment pattern works fine with TypeScript.

const strLen = (str: string) => str.length
const isLongEnough = (len: number) => len > 3
const toArray = <T,>(val: T): T[] => [val]
const addValueToArray = <T, R>(arr: T[], val: R): Array<T | R> => [...arr, val]

function getValue() {
  let _
  _= 'foo'
  _= strLen(_)
  _= isLongEnough(_)
  _= toArray(_)
  _= addValueToArray(_, 'foo')

  _.toUpperCase() // This errors as expected, TS knows it's not a string here

  return _
}

const result: Array<string | boolean> = getValue() // Return type inference works fine
const wrongResult: Array<string> = getValue() // This errors as expected, TS understands what the return type is

TS Playground.

The trick is to

  • Declare the variable separately from its first assignment (i.e. don't do let _ = 'foo')
  • Don't give it an explicit type

That'll make TS infer the type separately between each assignment.
And no, it's not any, either. Autocomplete works fine and you'll get an error for misusing the value in some step of the "pipeline".

The problem with this pattern is not tooling. It's convincing people that this is fine actually, after years of being told that const good let bad.

noppa avatar May 06 '25 11:05 noppa

@Jopie64 A good programming language makes writing good code easy. If using pipelines is advisable for writing better code, it shouldn't feel like you're fighting the language; otherwise, people will just not write pipelines at all (which is the case currently).

There was another issue about using the comma operator as a pipe operator (the comma operator really works mostly like a pipe operator already AFAICT). The most interesting idea there IMO was the one proposed by @HansBrende in this comment: https://github.com/tc39/proposal-pipeline-operator/issues/308#issuecomment-2568695874 (NOTE: if you're gonna read that issue, consider that @HansBrende's idea was different from the original one in the issue).

The idea was: instead of adding new syntax, why don't we just use what already exists. In that issue, we thought new syntax would still be needed at least partially because a topic token would have to exist. Now, with @noppa's comment, I'm not sure about it. TIL, something like this is already possible and works well in TypeScript (FYI @VitorLuizC @snatvb):

let _; (
  _ = 'foo',
  _ = strLen(_),
  _ = isLongEnough(_),
  _ = toArray(_),
  _ = addValueToArray(_, 'foo')
)

It's still a workaround, not an officially supported pipe syntax (and Prettier won't format it this nicely unfortunately), so maybe it conflicts with what I wrote in the 1st paragraph. And as @noppa said:

The problem with this pattern is not tooling. It's convincing people that this is fine actually, after years of being told that const good let bad.

However, if the Committee is so reluctant about adding new syntax and can't decide on which topic token to use (as @js-choi wrote in https://github.com/tc39/proposal-pipeline-operator/issues/232#issuecomment-2784879225)... Shouldn't we be pragmatic and try to reduce how much syntax this proposal adds if we want it to be accepted some day?

The main problem I can see is this doesn't work in an expression. It would be solved if we could use let inside expressions, so this would be possible:

doSomething()
  .then(someString => (
    let _,
    _ = someString,
    _ = strLen(_),
    _ = isLongEnough(_),
    _ = toArray(_),
    _ = addValueToArray(_, 'foo')
  ))

It would be another win if let inside expressions could allow this:

if (let x = ...) {
  // `x` is only available inside this `if` block
}

Not my dream syntax, just throwing some food for thought here for the appreciation of the Committee members reading this issue based on the new information @noppa brought to the table.

gustavopch avatar May 06 '25 13:05 gustavopch

The trick is to

  • Declare the variable separately from its first assignment (i.e. don't do let _ = 'foo')
  • Don't give it an explicit type

That'll make TS infer the type separately between each assignment. And no, it's not any, either. Autocomplete works fine and you'll get an error for misusing the value in some step of the "pipeline".

@noppa: This indeed works in TypeScript. But I don’t think I’ve ever seen this let reassignment pattern in any JavaScript or TypeScript codebase. Maybe I’ve seen it once in my life…

I would be quite interested to see if there are any real-world open-source codebases that use this pattern. If they exist, I would like to study them.

The problem with this pattern is not tooling. It's convincing people that this is fine actually, after years of being told that const good let bad.

In my opinion, the bigger problems with this pattern are that it’s clunky/noisy and it must be done on the statement level.

  1. let _ requires an extra let declaration every time you want to perform a pipeline.
    • Many developers already expressed unhappiness with even merely 3 extra characters required for unary function calls x |> f(#) vs. x |> f (e.g., #238, https://github.com/tc39/proposal-pipeline-operator/issues/217#issuecomment-1177178880).
    • let _ reassignment would require an overhead of 6 extra characters per pipeline, which would usually be on its own line.
    • This boilerplate is pretty clunky to write and noisy to read.
  2. More importantly, it also is not possible to use let declarations within expressions.
    • So single-line pipelines, e.g., embedded in concise arrow functions like a.map(async x => a |> f(#, 0) |> await g(#) would not be possible.
    • Though @gustavopch proposed that this change.
  • I believe that those two problems—clunkiness/noisiness and requiring multiple statements—are the primary reasons why let reassignment is not already popular.
  • Similar arguments to these also apply to const assignment, although at least with let you don’t need to name every step’s variable.
  • It’s worth revisiting the real-world examples in the explainer.
    • There are reasons why the developers of jQuery, Node, Underscore, Ramda, Express, and React didn’t resort to const named variables or let _ reassignment in those real-world examples.
    • I don’t think that “const good let bad” being cultural wisdom was probably a major reason.

However, if the Committee is so reluctant about adding new syntax and can't decide on which topic token to use (as @js-choi wrote in #232 (comment))... Shouldn't we be pragmatic and try to reduce how much syntax this proposal adds if we want it to be accepted some day?

@gustavopch: The problem with this proposal isn’t its complexity.

  • The pipe operator is widely known within TC39 as the simplest active syntactic proposal. I’ve heard this repeatedly from other committee people. (Maybe the call-this operator and its ilk are simpler.)
  • This simplicity is a disadvantage in the opinion of some people at TC39!
  • The pipe operator is syntactic sugar for developer convenience that brings no truly new program capabilities that cannot already be done in userspace.
  • This makes it a prime target for the proposed new JSSugar layer, no matter its form.
  • What I had meant in https://github.com/tc39/proposal-pipeline-operator/issues/232#issuecomment-2784879225 is that the biggest barriers to the pipe operator’s advancement are:
    1. Increased chilliness towards all new syntax—especially simple syntax for developer convenience without new program capabilities.
    2. The uncertainty of all future syntactic proposals for JavaScript, especially given the JS0/JSSugar proposal.
      • The JS0/JSSugar proposal is still very much in flux and might never actually happen, but it is being actively discussed.
      • The future of all new syntax in JavaScript, even the let expression idea, depends on those discussions.
      • So it is worth seeing how things there settle down first.
    3. @TabAtkins’ and my limited volunteer time and attention, which we have been allocating to lower-hanging fruit (i.e., easier proposals in JavaScript, CSS, etc.).
  • (The choice of the pipe topic’s token was historically a major factor, but it has become a much smaller problem lately in my opinion, with the three frontline choices being #, ##, and ^^. We probably could decide which symbol to use once and for all in a single meeting, although that’s not relevant to this issue.)

The main problem I can see is [that a let declaration] doesn't work in an expression. It would be solved if we could use let inside expressions […]

It’s worth adding this let expression idea to the explainer, as an alternative that we have considered.

  • It gets rid of one of the key limitations of let _ reassignment today: inability to use within expressions.
  • But, like I also said to @noppa above, it’s really clunky…
  • I’m also uncertain how the lexical scoping of _ would work. What would the scope of _ be? This might be the biggest problem at all.
  • It might be worth creating an issue devoted to this idea.

Lastly, I think that the the debate from the Hacker News thread about Rust’s lexical shadowing should be revisited. There’s a lot of discussion there that is relevant, including these advantages of lexical shadowing:

  • Not only avoiding having to name a new variable each time.
  • But also ruling out the possibility of reusing values from earlier steps when they become irrelevant and preventing their accidental reuse.
  • And also complementing variable immutability.

I encourage people to read it. I plan to incorporate some of that discussion into the explainer sometime.

js-choi avatar May 06 '25 16:05 js-choi