Convert partner profile to step-wise form

[cielf]

Summary

We want to rework the partner profile into a step-wise form, with error checking at each step.

Reasoning

The current form is very large and intimidating enough that it is a barrier to entry for the partners -- or the banks have to fill it in for them. Breaking it up into bite-size chunks will make it easier for the partners to deal with.

Details

This applies to both the partner and bank edits of the form (partner: partners/profile/edit bank: profiles/[partner #]/edit)
See concept below. Expand it for each of the 'cards' in the current partner profile edit forms. At each "save and next", we would want to be able to check the 'local' fields and display any relevant errors. The partners will also need to be able to "Submit for approval", which will check the whole thing, and, if ready, set the status to indicate that it is ready for approval.

Please note: 1/ Several of the cards can be enabled/disabled by the bank in their organization edit (sign in as [email protected], "My Organization", click on "Edit" -- the field to look at is: "Partner Profile Sections"). If the card is not available, it should not appear in the "accordion".
2/ The user should be able to navigate to any of the available sections at will.
3/ Intermediate results (at save and next) will need to be saved, possibly over days

Criteria for completion

• [ ] ability to navigate the partner profile form for partners, as described above
• [ ] ability to navigate the partner profile form for banks, as described above
• [ ] replacement of current system/request tests with tests that support the above
• [ ] addition of automated tests for the new functionality above

[cielf]

Reserved for @danielabar -- who is fully briefed and interested! (let's say reserved until September 1)

[danielabar]

This is on my radar. Planning to complete partner request confirmation, then will turn my attention to this.

[danielabar]

Now that partner confirmation is done, I could start on this.

[cielf]

Please do!

[danielabar]

Started. Looking into Accordion component to handle expand/collapsible sections.

[cielf]

bringing @dorner into the loop for any early technical guidance.

[danielabar]

This might be simpler than a typical step-wise form in that the underlying model Partners::Profile has already been created by the time an invited partner is filling out the form. Also, very few of the fields are required, and those that are, have default values. This means we don't need to worry about storing form values temporarily in session, they could be saved as the user goes through each section.

Each collapsible section could be a form that submits to the same app/controllers/partners/profiles_controller.rb#update action that is already in use today. The difference being a smaller subset of the permitted params would be incoming.

A complexity could be that on any given "subset" update (i.e. where user clicked Save & Next), we wouldn't want to redirect them to the show view redirect_to partners_profile_path from a successful update, but rather, take them back to the edit view so they can continue. And in this case, it would be nice to know which collapsible section to have open. Although if they can go in any order, maybe this isn't determinable.

Possibly a different action button is needed for user to indicate something like "Save and I'm done with all this", or is that the "Submit for Approval" button?

[danielabar]

It would also be good to clarify how the sections from the current form should get split up? For example, is each existing "card" from the current form to become its own collapsible section?

There is an existing partial for Agency Information, but it also includes Program Delivery Address, which is not shown in the mockups. Should Program Delivery Address be another collapsible section?

[cielf]

Yes, each card should be a step. Program delivery address was moved to its current location after I made up the mockups. (Recall also that the presence of most of the cards is controlled at the bank organization level)

Submit for Approval, in addition to saving, changes the status so that it appears on the banks list of partners to approve.

We'll likely want to make more fields conditionally mandatory (definitely not for this issue) as we go on.
@dorner - What do you think of the approach @danielabar described above?

[danielabar]

Haven't had as much time to work on this but it's still on my todo list. Just wanted to post an update that it is possible to programmatically target the Bootstrap accordion sections whether they should render as expanded or collapsed.

Expanded

• button does not have collapsed class
• button has aria-expanded="true"
• content section has show class

<div class="accordion-item">
  <button class="accordion-button"... aria-expanded="true">
  <div class="accordion-collapse collapse show">
  ...
</div>

Collapsed

• button has collapsed class
• button has aria-expanded="false"
• content section does not have show class

<div class="accordion-item">
  <button class="accordion-button collapsed"... aria-expanded="false">
  <div class="accordion-collapse collapse">
  ...
</div>

[danielabar]

Tricky part may be on each given save of a section, determining which is the next section to render as expanded (and to ensure all the others are rendered as collapsed). It looks like there are 3 static (i.e. always shown) sections, with a number of dynamic sections in the middle (only to be shown if the partner has been configured as such).

Something like this:

n: loop over current_partner.partials_to_show k: last

1. STATIC, DEFAULT EXPANDED partners/profiles/step/agency_information_form
2. STATIC, DEFAULT COLLAPSED partners/profiles/step/program_delivery_address_form
n. DYNAMIC, DEFAULT COLLAPSED partners/profiles/step/{partial}_form
k. STATIC, DEFAULT COLLAPSED partners/profiles/step/partner_settings_form

[danielabar]

@cielf A question about the Submit for Approval button that shows up in each step/section from an earlier comment:

Submit for Approval, in addition to saving, changes the status so that it appears on the banks list of partners to approve.

I went through the existing workflow to understand how Submit for Approval works currently:

As org admin:

1. Create a new partner agency
2. Click Invite to send email to new partner

As new partner user:

1. Accept invite
2. Set password
3. Navigate to Edit my profile

However, I don't see any Submit for Approval action anywhere in the existing partner form. All I see is Update Information and Cancel Update:

Is the "Submit for Approval" a new action/logic to be developed as part of this? If yes, do you know the details of what this should do?

Or how does submit for approval logic work today?

For example, the partner_profiles table has a string column partner_status that defaults to pending. Should the Approve action change the value of this field? Are there any other table/fields that need to change?

Note that partner_status is not an enum (that I can find) so I'm unsure what are the valid values other than pending.

Update: I see that the partners/Partner table/model also has a status column and this one is an enum, so perhaps this is the value that the Submit for Approval action needs to change to awaiting_review?

# app/models/partner.rb
enum status: { uninvited: 0, invited: 1, awaiting_review: 2, approved: 3, error: 4, recertification_required: 5, deactivated: 6 }

[cielf]

I think that the submit for approval button is on the partner's view of the partner profile.

Yes, Submit for approval changes the status to awaiting_review. There are a very few fields that are mandatory for submitting if the appropriate sections are used by the bank. [Specifically, you have to enter something in the social media section if the bank uses it. ]

In the step-wise world, Submit for approval will do a full validation (in case there are mandatory fields on other steps) and, if it passes, change the status to awaiting review.

As background, there are a couple of other things we'd like to make conditionally mandatory (don't remember what they are atm), but we decided to put them off until the stepwise was done. Out of scope for this change.

Hrm! IIRC, submit for approval only appears at certain statuses (invited and recertification required?). We should maintain that -- because IIRC, the partner can't make requests when in "waiting for approval". I'm not sure about the status "error", though -- whether it's a relic or is still used..

[danielabar]

@cielf It's still a work-in-progress, but at this point, there's enough of the flow working that it might be useful if you have a chance to try it out and let me know if its on the right track.

Branch: https://github.com/danielabar/human-essentials/tree/4504-partner-profile-step-form

Feature so far:

1. Step-wise editing of partner form is behind a new flipper flag partner_step_form
2. New partner profile edit form displays each section collapsed except the current one being edited.
3. If organization has not restricted the partner_form_fields, then all sections are displayed and expanded one by one as user clicks "Save and Next"
4. If organization has restricted the partner_form_fields, then only those sections are shown, and the "Save and Next" logic knows to expand the next available section.
5. Submit for Approval makes use of existing app/controllers/partners/approval_requests_controller.rb

Still todo:

• Population served looks broken re: radio button ovals and no labels
• Submit for Approval is always displayed, still need to figure out this part of the logic if to display or not
• Test what happens if errors on saving any given partial
• Maintain system tests, how to do this wrt feature flag?
• Why does http://localhost:3000/partners/profile/edit sometimes browser tab spins and page not responsive
• Make accordion title styles stand out more
• Form field alignment? (eg: columns of fields rather than all one underneath the other)
• What's the process for adding a new feature toggle other than manually in /flipper? eg: migration?

How to use it:

1. Login as superadmin
2. Navigate to http://localhost:3000/flipper/features and enable partner_step_form
3. Login as org admin, create new partner agency and invite them
4. As the new partner, accept invite, set password and navigate to http://localhost:3000/partners/profile/edit

Variation: As org admin, restrict the partner form fields.

You should see the first section expanded and the rest collapsed like this:

[cielf]

Hey @danielabar FYI It'll probably be a couple of days before I get to it.

[cielf]

Hey @danielabar

This is definitely what we're driving at!

The following is (so far), long-winded musings rather than 'gospel'. We are coming up hard against my own ignorance in the area of UX.

I think the big question is -- if you pop around between sections, filling in a bit here and there, then click a save and next -- what should happen?

When I'm just playing around on the page, I feel that the "path of least surprise" is that the changes I made everywhere are kept. "Least surprise" would dictate we shouldn't lose any of that information -- which is what is happening at the moment. The extent to which we check things that are not in the current section is thornier. Is "save and next" a check of everything I've touched?

Do we force people to go through in order? I don't like that on the surface of it - but it might be the right thing to do. Do we do a save & check whenever someone goes to a different page? That's got a problem in a "what if they don't know the answer to a mandatory question?" way.

We could really use a UX person on this, so I've reached out to the rfg community to see if we have any atm. Otherwise, I think I'll need to do some research.

---

The following was my brain dump as I went through. I went through without reading through your list of todos above. I'm saving this here mostly for my future reference - a good deal of it is stuff you said above is still to do.

1/ I noticed something awry with the radio buttons under Population Served -- they don't have labels on them. Other radio buttons seem fine... and these seem fine on staging. (you already called that out) 2/ We're going to have to play around with the colours and font sizes on the accordion. I think we don't want black on white. Let me see if I can dig up a UX person to consult on this.
3/ I also think we will want to remove the inner title on each card -- so it's not repeating. That will probably mean making the fonts on the accordion headers bigger /heavier. 4/ I think Save and Next needs to be at least as prominent as Submit for Approval. Can we see what it looks like with the same colours?
5/ We might need to put the words "click to expand /collapse" somewhere. Not sure. 6/ "Attached documents" should be "Additional Documents" (probably my fault from the diagrams) 7/ And, of course, we want the same thing for the bank editing the partner profile -- I'll admit I thought it was the same form! 8/ It looks like the area served javascript is broken -- the total is no longer updating (Go to area served, change one of the values. the "The Total is" number should update. 9/ When we have an error, it would be good if the section the (first) error is in in expanded (I save and nexted the area served with the total not being 100%) 10/ I hit submit and approve with an error in play, and it didn't give an error. I don't think it's trying to save the profile first, and it should.
11/ Submit for Approval should only appear when the status is.. invited or recertification required. 12/ We should have a "Save" on the final section. (we just have submit for approval now) 13/ Hmm. I wonder if we should have the down arrow on the left instead of the right -- would that be a more obvious indicator (Again - hope I can dig up a UX person) 14/ I put errors in two different sections, went to a third section, and clicked "save and next". It "saved", but it didn't seem to touch the sections that were in error.

---

Sorry for the unconstrained brain dump !

[cielf]

@danielabar It is early days, but we have a monthly stakeholder meeting on Wednesday. We have some other things we need to talk about with them, but if there's time, I'd like to show them what you have so far. We first talked about doing it over a year ago, so it'd be nice to be able to show that there is progress.

[danielabar]

@cielf I'm thinking maybe I should hold off on further development (such as those TODOs) until hearing back on results of UX research?

[cielf]

I think we do need some more thought. But... can you tell me if it is plausible to have it check and save the previously open section whenever we move to a new one? If so, that could work as long as we don't have any conditions that span sections (and I really don't think we will).

[danielabar]

Currently every time the "Save and Next" button is clicked, it should be saving whatever is entered into that section, no matter what order the user goes on.

Did you find a sequence of steps where that's not the case?

[cielf]

No... but you can switch between sections... I'm pondering whether we can/should check the current section whenever someone changes to a new section, however they do it.

I have found someone to take a look. They aren't a UX by training, but have been doing UX in their day job -- so at least more UX-y than I. Hopefully they'll take a look later this week.

[danielabar]

Just to confirm the alternate interaction:

1. User has a current section open, for example "Agency stability"
2. They start filling it in, for example "Year Founded: 2014"
3. Then they realize they don't know what values to fill out for the remaining fields in that section (eg: Form 990 Filed, Program Name(s), etc)
4. User then scans down the list of remaining sections and sees another one, for example "Population served" which they think they know the answers to.
5. User clicks on that section "Population served", which also will close their current section "Agency stability"
6. In this case, user never clicked "Save and Next" on "Agency Stability" section, but the data they filled in so far (Year Founded: 2014) should be saved.

i.e. Anytime a section is closed, even if user didn't click Save and Next, save it.

This could be tricky because maybe what the user meant by navigating to a different section is they want to abandon their current edits, i.e. would a save action in this case be surprising?

If it is the intended interaction, I experimented with a Stimulus controller to listen for the bootstrap collapse event on the accordion sections, and then submit the form just as if the user had clicked "Save and Next". It "works" as in their data will be saved, but it messes up the flow.

Because the submit logic on this branch always navigates to the next section in sequence.

So we would need to know where they intend to go and modify the submit action accordingly.

There is also an expand/open event but it's not connected to the collapse/close event. This makes it difficult to know via the bootstrap events both at the same time, what's being closed (i.e. what to save) and what's being opened (i.e. where to navigate to next).

The current solution is going to the server for a round trip of saving and rendering the edit form with the next open section.

Possibly the entire thing could be modified to use ajax form submission with a stimulus controller on the accordion element. Then the url would never change and it would be relying entirely on bootstrap/js to maintain the state of which accordion is open. BUT, this might complicate things wrt validation, we'd lose a lot of niceties of form submit -> render with errors if any.

Just to capture what I attempted with a Stimulus controller listening to the accordion close event:

// app/javascript/controllers/accordion_controller.js
import { Controller } from "@hotwired/stimulus"

// Connects to data-controller="accordion"
// When an accordion section is collapsed (aka hidden), submit the form in that section as if user had clicked "Save and Next" button.
// Ref: https://stackoverflow.com/questions/67860141/listening-to-the-bootstrap-collapse-shown-hidden-events#67862044
export default class extends Controller {
  connect() {
    console.dir(this.element)

    this.element.addEventListener("hidden.bs.collapse", function (evt) {
      const form = evt.target.querySelector("form");
      // FIXME: 4504 doesn't work well with Partners::NextStepService because we lose where the user is trying to go
      form.requestSubmit()
    });
  }

  disconnect() {
    this.element.removeEventListener("hidden.bs.collapse")
  }
}

Modified the partner step wise edit view to attach the accordion stimulus at the top level accordion element. Then it receives the close event for every collapsible section:

# app/views/partners/profiles/step/edit.html.erb
<div class="accordion" id="accordionExample" data-controller="accordion">
  ...
</div>

[danielabar]

More on the above, it might be possible to capture both the section being closed and new section being opened if the close/open events always occur in a predictable sequence. In this case, it might be possible to modify the form before submitting from the stimulus controller to let the server know where the user wants to land after the current save operation. I'll investigate further.

[cielf]

Hmmm. I think the checking and saving the current section when you leave the section behaviour wouldn't be too surprising, or at least it would be less surprising than popping over to another section and having the first section's work disappear when you "save and next".

On the flow -- I think I'm missing something (and I admit I am nowhere near as deep in this as you) -- I guess what I'm asking would involve having the system 'know' the current section when you click a new one, and being able to check/save the current section, and that, then, clicking on "Save and Next" would be the equivalent of clicking on the next section

[danielabar]

Here's how it works currently (and this could be considered a first attempt and discarded if there's a better way):

Each section is built out as an individual form, they are all here.

They are laid out in the following order, with the dynamic ones following the rule if the org has restricted which sections they should see, otherwise all dynamic sections are shown:

Partial	Type	Default	Next
agency_information	static	expanded	program_delivery_address
program_delivery_address	static	collapsed	media_information
media_information	dynamic	collapsed	agency_stability
agency_stability	dynamic	collapsed	organizational_capacity
organizational_capacity	dynamic	collapsed	sources_of_funding
sources_of_funding	dynamic	collapsed	area_served
area_served	dynamic	collapsed	population_served
population_served	dynamic	collapsed	executive_director
executive_director	dynamic	collapsed	pick_up_person
pick_up_person	dynamic	collapsed	agency_distribution_information
agency_distribution_information	dynamic	collapsed	attached_documents
attached_documents	dynamic	collapsed	partner_settings
partner_settings	static	collapsed	NA

Each form has a "Save and Next" action button that will submit the form to save the data. So as long as user clicks on "Save and Next", they should never lose any data entered in that section.

Each section/form as part of submitting, also includes an extra piece of information to the server, a string name of the submitted_partial. For example when Agency Information is submitted:

url: partners_profile_path(submitted_partial: "agency_information")

The server, after successful save will use that submitted_partial string to determine what should be the next open section. This logic is in Partners::NextStepService, which essentially follows the order in that table above, considering the dynamic logic.

After the server has determined which section should be shown as open next, it redirects back to the edit view, with a parameter open_section:

open_section = Partners::NextStepService.new(current_partner, submitted_partial).call
redirect_to edit_partners_profile_path(open_section: open_section)

For example, after clicking on "Save and Next" from the first section Agency Information, you're redirected to:

http://localhost:3000/partners/profile/edit?open_section=program_delivery_address

The partners profile edit view uses that parameter open_section to target the bootstrap css classes and data attributes to open the appropriate section and ensure all the others are closed.

So currently, the only communication to the server is via the "Save and Next" action buttons that appear on each form (or the Submit for Approval buttons).

On the other hand, the up/down caret icons at the right hand end of each section do not perform any server communication (although they could be modified to do so via a Stimulus controller):

In this first attempt, those caret icons are only executing the out-of-the-box Bootstrap JavaScript to open and close sections, so this "state" is maintained only in the client/browser.

If we do want server side behaviour associated with these caret icons, it's a little tricky because opening one closes the others, but these are two separate JavaScript events that would have to be co-ordinated.

Note that it might get messy to submit the forms via ajax from these caret icon clicks because then we'd lose the benefit of server rendering of the form validation errors.

It is possible from JavaScript to submit that section's form exactly as if the user had clicked "Submit for Approval" button. But then we'd need to reconsider the existing logic.

Because currently when user clicks "Submit for Approval" on for example Agency Information, the server responds with:

http://localhost:3000/partners/profile/edit?open_section=program_delivery_address

Which tells the edit view to open Program Delivery Address.

On the other hand if user has started filling out Agency Information, and rather than clicking Save and Next, they randomly click the open caret on any other section such as Population Served, we'd need to figure out how to tell the server that even though Agency Information is being saved, the user wants to come back to:

http://localhost:3000/partners/profile/edit?open_section=population_served

It's tricky because this would require co-ordination and trying to maintain state on the client between two different Bootstrap JavaScript events. Bootstrap fires separate open and closed events when these sections are expanded/collapsed.

It might be possible to co-ordinate the JavaScript Bootstrap events to do what we want, I can do some more research in this area.

[danielabar]

Another thing to think about if we start connecting server-side behaviour to the open/close caret icons:

User could simply close the section they're currently working on, without opening anything else. Then would that mean save that section and return to edit view with all sections closed? That could be tricky with the trying to co-ordinate JS events because there would never be an open event.

[cielf]

I haven't worked through all of the above yet -- this is just a quick response to your last comment. As a user, I can legit see me collapsing a section before going to another section -- I'm pretty sure that's the first thing I did when I was looking at it! So there could definitely be a point at which all the sections are closed.

[danielabar]

The last commit on this branch has a first attempt at connecting the server-side submit behavior with the accordion open/close events.

The following scenario should now work:

1. User starts editing a section, for example Agency Information, fills in a few fields
2. Then user cilcks the Open caret for some other section further down, for example Agency Stability (i.e. does not click Submit for Approval in their current section Agency Information)
3. In this case, the new Stimulus controller I added for the accordion detects that the Agency Information section has closed, followed by the Agency Stability section opening. So it will submit the Agency Information form just as if user had clicked Sve and Next. So whatever work in progress user entered in Agency Information will be saved.

BUT it also adds a hidden field for open_section_override being Agency Stability (because thats where user decided to open). So the server will respond with Agency stability being the next open section (rather than Program Delivery Address that would have been the next logical open section).

It does not yet handle the case of user simply closing their current section without opening anything else, but that can be iterated on.

[cielf]

(laughs) Progress! I was writing a comment that tried to explain the naive model in my head, which just had a "to_be_opened" that would be set by either the "Save and Next" or clicking on then open caret, and a "was_open", but it sounds like you got to something like the same destination.

I've got company, so I might not get to looking at what you have today. Do you need feedback to continue?

[danielabar]

I'll probably continue tomorrow morning. I can try to implement the "user closed everything" - save whatever the open section was and return to the view with everything closed.