Running usablity experiments with proposed syntaxes for Pipeline Operator

[codehag]

Hi everyone,

I am going to be assessing a couple of tools around language ergonomics, and I am looking for a proposal which is in an early enough stage that the assessment from the tools will be useful. The assessment will take the form of a structured survey with minimal bias that presents multiple options to a user and checks how quickly and easily they comprehended the meaning of the code. We are looking at the level of mental load that developers from different backgrounds will experience.

The pipeline operator looks like a good candidate for this. The experiment will be non-binding, it is just to assess whether or not such a tool will be useful for our process when considering new syntax. what do people think?

[codehag]

Ok, that looks like a positive response -- Who can I send the draft of the experiment to for review?

[littledan]

Would it work to post the draft here? Either way, I am happy to review.

[codehag]

For review: https://qsurvey.mozilla.com/s3/2019-TC39-Feature-Experiment

[nicolo-ribaudo]

I don't know if it is intended, but I think that the example contradicts the description: il the pipeline "passes promises to await" (assuming that it means "Do await promise automatically"), why should I use .then?

---

Here add is defined but not used, and wash, cut and punt are used but not defined. Is this intended, since you only need to know the definition of ferment to find the bug?

---

In inputs which expect code, maybe a monospaced font would be more familiar to developers?

[jhpratt]

That survey is very difficult to complete on mobile, and requires switching back and forth between pages. Overall, it looks pretty fair, though.

[jineshshah36]

Why is the # option such an afterthought at the end and only shown in an intentionally complex situation. Showing that and then asking if anyone would prefer the hash syntax seems like you're gonna get a much lower likelihood of anyone preferring that given you showed the most complicated possible example. The example should have been identical to the example previously like so:

const pickle = "cucumber"
  |> wash
  |> new Jar(#)
  |> add(["water"], #)
  |> salt
  |> ferment(#, 2)
  |> #.then(pickleJar => pickleJar.takeOne());

[codehag]

Jinesh I am guessing that you had the proposal 1 variant. You shouldn’t have seen the complex version of the # — that’s a bug. Thanks for pointing it out. There is another variant of this survey which tests both proposals in parallel, and actually has the code snippet you describe, but it does not have the cognitive load test as part of it.

The cognitive load version should only appear with the single proposal, showing the difficult version of that single proposal.

On Tue 28. May 2019 at 05:36, Jinesh Shah [email protected] wrote:

Why is the # option such an afterthought at the end and only shown in an intentionally complex situation. Showing that and then asking if anyone would prefer the hash syntax seems like you're gonna get a much lower likelihood of anyone preferring that given you showed the most complicated possible example. The example should have been identical to the example previously like so:

const pickle = "cucumber" |> wash |> new Jar(#) |> add(["water"], #) |> salt |> ferment(#, 2) |> #.then(pickleJar => pickleJar.takeOne());

— You are receiving this because you authored the thread. Reply to this email directly, view it on GitHub https://github.com/tc39/proposal-pipeline-operator/issues/150?email_source=notifications&email_token=AGNYEJ5HPOB7O7NU7OARPCLPXSSB7A5CNFSM4G4BPMVKYY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGODWK36WI#issuecomment-496353113, or mute the thread https://github.com/notifications/unsubscribe-auth/AGNYEJ22GD5B623LTMJXERTPXSSB7ANCNFSM4G4BPMVA .

[codehag]

@nicolo-ribaudo Thanks for pointing out the issue with the .then -- it looks like that is a left over from the iteration process.

As for the mono space font... its not possible with the survey framework. It doesn't make sense to make a custom survey framework since we are just testing this method right now. I know it isn't ideal.

[codehag]

@jineshshah36

I just took a look and it looks like everything is set up correctly so I am not sure how you saw the # proposal mixed in with a test about the other proposal. Do you remember any other features of the variant that you work working through?

[jineshshah36]

@codehag it had a really complicated code example that used "#" everywhere in it and then it asked if I preferred using "#" or using arrow functions with parenthesis. I went back to it and this is the screenshot:

I literally just went back to the survey link posted above and it got to the question that shows that image.

[codehag]

@jineshshah36 ok, I have that turned it off for now. We can use it later. I also addressed the other comments made.

There is a link promoting the survey here now: https://twitter.com/ioctaptceb/status/1133750888223465472

[mAAdhaTTah]

FWIW, I think the question RE "SyntaxError: # is no defined" throws something more specific in the @babel/parser implementation, specifically indicating that a # is expected in the particular pipeline step.

Couple of other pieces of feedback I've heard since sharing:

1. I wish the code wasn't images.
2. Asking people how long they have been “working as a programmer” vs. just “programming” seems fraught.
3. How many pages are there? ... I didn't realize I was going to have to do coding questions. Might be helpful to have a disclaimer.

Hope that helpful!

[reconbot]

Should this be salting the wateryCucumberJar so it's consistent with the pipeline example?

[mAAdhaTTah]

Got another issue:

At one point it asks the following, but there is no example above it related to "Pipeline with #".

Which of the above do you prefer?

   Current status quo    
   Pipeline    
   Pipeline with #    
   Other - Write In (Required)

[codehag]

Both are now fixed.

[codehag]

@mAAdhaTTah do you know the specific message?

[mAAdhaTTah]

I don't, that was the info I got, sorry.

[nicolo-ribaudo]

If you mean the error message thrown by Babel, it is "Topic reference was used in a lexical context without topic binding" repl

[mAAdhaTTah]

@nicolo-ribaudo whoops thanks for clarifying, I misunderstood the question.

---

Edit

Actually, I was thinking of this error:

/repl: Unexpected token (2:25)

  1 | async function test() {
> 2 |   return a |> b |> await |> c;
    |                          ^
  3 | }
  4 | 

which actually is a SyntaxError.

[misterdjules]

@codehag One of my coworkers was asking me about the pipeline operator proposal, and I mentioned to him that there was a usability survey that was available at https://qsurvey.mozilla.com/s3/2019-TC39-Feature-Experiment, but it seems it's not available anymore. Is that experiment done, and is there going to be another round in the future?

Unrelated to this proposal, do you have other experiments going on at this time? I think some of my coworkers would be interested in any experiment you may be doing in the future.

Thanks for all the work you're doing 🙏