Investigate UX improvements to the azd up provisioning flow

[savannahostrowski]

Background

As part of our usability studies and experience audit, we have identified that end-users sometimes struggle to understand what is happening behind the scenes when they run azd provision (or azd up during the provisioning stage). This contributes to the following negative outcomes:

• Decreased likelihood of using the tool again due to lack of understanding
• Inability to fully utilize capabilities of the azd CLI tool because they don't understand the various commands (init, deploy, provision, etc.)
• Ensure that the developer can have a consistent mental model between azd and the Portal
• Inability to troubleshoot error or failure states (we will focus specifically on error and failure states in X issue)

Hypothesis

We believe that we can reduce the amount of end-user journeys that end in failure, increase the likelihood a user will use azd again, and also increase a user's comprehension of the Azure Developer CLI by doing the following:

• Introducing layout, language and visual improvements to the provisioning flow
• Surfacing line-by-line feedback on each service as they're provisioned (aids in troubleshooting and comprehesion)
• Closing the feedback loop by improving our success state messaging

Initial task list

• [x] Investigate and capture existing experience and opportunities for improvement [@Austinauth]
• [x] Quick audit of competitor experiences [@Austinauth]
• [x] Create low-fidelity sketches exploring improvements to the provisioning flow (easy to implement & blue sky) [@Austinauth]
• [x] Review low-fidelity sketches with team, capture feedback [@Austinauth + Team]
• [x] Incorporate team feedback into designs and create final sketches for review [@Austinauth ]
• [ ] Set up time with the developer who will be implementing [@savannah]

Relevant links

• Initial experience audit
• Accelerators Research Report
• Provisioning flow improvements - Explorations (Figma)

[Austinauth]

Current behavior

Captured: 07/28/2022

Creating resource spinner

Creating resource spinner w/completion numbers

Installing dependancies

Deploying service APIs

(Very long delay at this step)

Deploying service web

Error messages

Error 1: Service Unavailable due to high region traffic

Error 2: Unclear what the error is

Error 3: NPM related issue

Deprovisioning (azd down)

(Very long delay at this step)

Deprovisioning complete (No success message)

[Austinauth]

The latest azd build (version: 0.1.0-beta.3) introduces a few quality-of-life improvements to the provisioning step; primarily a list-style view of your resources as they're being provisioned.

[Austinauth]

Update: 08/08/2022

• Completed initial explorations of an improved provisioning flow (Figma file)
• This includes explorations of: flow, layout, headers, provisioning items, color, loading states and more
• Setup time for us to discuss the explorations on 08/11/2022 (Feedback, critique, etc.)

Next steps:

• Review explorations with team during 08/11/2022 - capturing feedback and ideas
• Narrow down options we like best
• Incorporate feedback into designs

cc: @puicchan @savannahostrowski

[karolz-ms]

@Austinauth will the design support no-color environments? https://no-color.org/

[Austinauth]

@Austinauth will the design support no-color environments? https://no-color.org/

Designs should work without color if user chooses to go that route. Colors will enhance the experience, but we won't rely on color only for any affordances.

[Austinauth]

Update 08/16/2022

• @savannahostrowski to get me included on the bi-weekly dev demos
• Will showcase this work on the dev bi-weekly demo call
• Need to have discussion with Devs during that call on how we implement
• How do we measure success since we are not doing a UXR session?
• Don't need a/b testing since not comparing two options
• Low risk that this makes the experience worse
• Implement changes and next time we do a lab we'll focus more heavily on reactions to the CLI

[ellismg]

Overall, I love the direction. A few small notes:

I've noticed today that we use the prefix "Created" for resources in our status messages, even in cases where you run azd up a second time, without changing the infrastructure in any way. If we can determine an operation did a create vs an update for a resource, I think we should surface that in the tool. If we can't (I'm not 100% sure on the fidelity of the ARM API here) I think we need to come up with some other word other than "Created". It's very confusing on subsequent updates to see "Created" for a resource that already exists (and made me wonder if were creating multiple copies)

This may be out of scope, but I think we need to think a lot about if azd up --template is really a thing we want long term. I think the general concern here is that in steady state you never pass --template to azd up. It's something you may do once, when initializing you project, but never again.

One of the nice things about azd up --template was it provided a single command that got you from "zero to wow" quickly. It demos super well. But I think we should still remove this support.

In fact, in https://github.com/Azure/azure-dev/pull/235, we are removing the usage of azd up --template from our prose, precisely because it has caused user confusion.

I think we should remove the --template parameter from azd up. Instead, just support it on azd init (a command you would run only once anyway).

[ellismg]

A consequence of removing --template from up perhaps means that up now just means provision + deploy. init is no longer part of the up flow (since you only use up once you have a project initialized).

[Austinauth]

@ellismg

I've noticed today that we use the prefix "Created" for resources in our status messages, even in cases where you run azd up a second time, without changing the infrastructure in any way. If we can determine an operation did a create vs an update for a resource, I think we should surface that in the tool. If we can't (I'm not 100% sure on the fidelity of the ARM API here) I think we need to come up with some other word other than "Created". It's very confusing on subsequent updates to see "Created" for a resource that already exists (and made me wonder if were creating multiple copies)

This is a great idea. Love this. Could definitely help clarify that running azd up is not just something you can do once.

This may be out of scope, but I think we need to think a lot about if azd up --template is really a thing we want long term. I think the general concern here is that in steady state you never pass --template to azd up. It's something you may do once, when initializing you project, but never again.

One of the nice things about azd up --template was it provided a single command that got you from "zero to wow" quickly. It demos super well. But I think we should still remove this support.

In fact, in https://github.com/Azure/azure-dev/pull/235, we are removing the usage of azd up --template from our prose, precisely because it has caused user confusion.

I think we should remove the --template parameter from azd up. Instead, just support it on azd init (a command you would run only once anyway).

Yeah, probably out of scope for this. But I definitely see what you're talking about. I'm not sure I have a fully informed opinion on the matter yet, but certainly a discussion I'm interested in being a part of.

A consequence of removing --template from up perhaps means that up now just means provision + deploy. init is no longer part of the up flow (since you only use up once you have a project initialized).

Yeah, I would be very open to exploring this, but I don't feel confident enough to make a call on one approach over the other yet. Perhaps this is something we could prototype and a/b test? cc: @savannahostrowski , @puicchan

[jongio]

A consequence of removing --template from up perhaps means that up now just means provision + deploy. init is no longer part of the up flow (since you only use up once you have a project initialized).

I disagree about removing --template from up, so let's discuss before that happens. Having a single command that get a complete solution up on Azure is a big selling point and value prop - the soul of azd.

[savannahostrowski]

I've noticed today that we use the prefix "Created" for resources in our status messages, even in cases where you run azd up a second time, without changing the infrastructure in any way. If we can determine an operation did a create vs an update for a resource, I think we should surface that in the tool. If we can't (I'm not 100% sure on the fidelity of the ARM API here) I think we need to come up with some other word other than "Created". It's very confusing on subsequent updates to see "Created" for a resource that already exists (and made me wonder if were creating multiple copies)

This is a great point. We should definitely see where it's possible to differentiate between "created" and "updated" (or similar).

A consequence of removing --template from up perhaps means that up now just means provision + deploy. init is no longer part of the up flow (since you only use up once you have a project initialized).

I think we'd really want to understand developer workflows and preferences with azd before deciding to remove --template from up. We do have an issue/discussion tracking asking whether people think that init belongs in 'up` (see https://github.com/Azure/azure-dev/discussions/114) but as of yet, it has not gotten a lot of community attention or feedback.

[Austinauth]

Update 08/23/2022

• Expect discussion on how to best implement during our demo call today
• Follow up with Developers on how I have help provide them with guidance

[Austinauth]

Update 08/29/2022

• Work was demoed to the development team last week
• No major issues were identified, but I do need to work with whoever is implementing to provide guidance for edge cases (running init only for example).

---

@rajeshkamal5050 this work is ready for development. @savannahostrowski suggested I tag and assign you so we can get it prioritized for development. Here is a link to the prototype, I can also share a link to the Figma file directly over teams.

Once you have sized and determined who will be implementing, please reach out to me. I am happy to pair with the developer to answer any and all questions.

[charris-msft]

A consequence of removing --template from up perhaps means that up now just means provision + deploy. init is no longer part of the up flow (since you only use up once you have a project initialized).

I disagree about removing --template from up, so let's discuss before that happens. Having a single command that get a complete solution up on Azure is a big selling point and value prop - the soul of azd.

Absolutely agree. "One command to rule them all!".

Two things I would love to see here are:

1. Acknowledgement in the progress that the resources already exist (per the comment about differentiating between "created" and "updated")
2. "If it ain't broke, don't fix it" - figure out fast if the deployment will change a resource and if not, move on quickly. Perhaps azd is already doing this and the comparison is very slow, but ideally, if all I changed was adding a comment in a python file, azd up should run nearly as fast as azd deploy.

[charris-msft]

Seeing creating (3 of ~5 completed) change to (10 of ~11 completed), drives my brain nutz.

I love the idea of showing progress, but I would much rather you didn't show me the 'of ~##' part than keep moving the goalposts.

[Austinauth]

@charris-msft Our latest concepts remove this confusing piece of feedback in favor of showing more generalized progress!

Here's an example of how the loading state + feedback will look for individual items as they're provisioned / deployed (loading bar + success/failure state).

[charris-msft]

As for recovering SRE who spent lots of time watching the command line, I really appreciate it when I can estimate how much time is left, but if we can't do it well this is much better than today's experience.

Is it not possible to show all of the resources and then fill in the progress as azd is working on each one?

[Austinauth]

@charris-msft

As for recovering SRE who spent lots of time watching the command line, I really appreciate it when I can estimate how much time is left, but if we can't do it well this is much better than today's experience.

We had mocked up a few variations that explored showing the time remaining and the completion %, however, technically it doesn't sound like this is feasible at the moment without adding a significant amount of loading time. 😢

I'll continue to push on this in the future though.

Is it not possible to show all of the resources and then fill in the progress as azd is working on each one?

Hmm.. this is a good question, not something I had initially considered. I just want to make sure I'm understanding you correctly. You're suggesting that as soon as the provisioning starts you would like to see all the services being provisioned listed at once, and then the progress of each item individually?

I had just assumed items were provisioned in order and that this would not be possible, but I can definitely take it back to the team and create some explorations!

[charris-msft]

@Austinauth - regarding showing all the resources - perhaps something like this if there are just 4 resources being created:

Creating resources
(v Done): App Service Plan: plan_abcdefg
(v Done): Log Analytics Workspace: log_abcdefg
(==    ): Application Insights: appi_abcdefg
(pending): Portal Dashboard: appid_abcdefg

[Austinauth]

@charris-msft Gotcha! I'll check with the team to see if something like this is doable 👍🏼

[Austinauth]

@vhvb1989 You can find all the flow updates in this prototype under the "UX Flows" section. Feel free to reach out if you have any questions or want to pair on anything!

[rajeshkamal5050]

@Austinauth can we close this from a design perspective?

Implementation tracker - https://github.com/Azure/azure-dev/issues/557

[rajeshkamal5050]

Closing this as done from a UX design perspective.