Usability testing for CTJ skills page design

[itsmanjuchawla]

trafficstars

Overview

This issue will outline the workflow for the usability testing of CTJ's skills and calendar page. Participants in this test will be asked to log into the CTJ page and select the ratings of their skills based on their best knowledge. Based on their skills rating, they will be directed to the projects that fit well with their skill level. Participants will also be asked to select dates and times of their weekly availability.

Action Items

• [ ] Researchers meet with the design team to understand the userflow of the design and to understand the goals designers intend to achieve from the usability testing.
• [ ] Researchers go over the old examples of the research plan of CTJ onboarding
• [ ] Create a new research plan with new script and instructions for participant selection
• [ ] Pilot test with the design team
• [ ] Participant recruitment (minimum 5 participants)
• [ ] Testing

Resources/Instructions

Research Plan is in progress: Research Plan CTJ Task List is in progress: Task List

[kcoronel]

@itsmanjuchawla @xxl228 hey please update with the work you have done on this issue - thank you!

[xxl228]

Discussed with PM about the current user flow for MVP1. We determined the final user flow for this upcoming usability testing is to test folks who has recently onboarded, the user flow will be login->verification for the email for either onboarding info or already onboarded => proceed to skill matrix => see full job postings and able to apply them. There will be no variation of the number of jobs to see with different skill levels with MVP1. The limited view version is already tested, which is for everyone who comes to the site. The limited view would not allow people to apply, which is the only difference. Current user testing should focus on the skill matrix.

[xxl228]

Met with design and discussed the user flow, verified with PM for final user flow, may add a page before the sign in page to communicate expectations with users, also need to communicate the expectations to users (onboarding first, then create account with same email) under How to Join (if that make sense), will discuss more details next week when meeting with design. Discussed usability testing details, will start to create a screener for the upcoming usability testing while continue to work on the usability testing plan.

[xxl228]

Making progress on Usability Test, completed draft screener, task sheet is almost complete, draft research plan is in progress, will finalize it in the coming weeks.

[xxl228]

Met today as a team to finalize the Usability Research plan, discussed task list, and get the screener ready. Discussed meeting with Design team to receive the final prototype for usability test. Also discussed plans for pilot testing and recruitment steps.

[xxl228]

Pilot tested the usability testing script, research questions, recruiting screener, consent form with Rishi as a pilot test. Asked Manju to do another pilot test with someone else this coming week. For the following week, we plan to finalize all details for the Usability Testing. Items still need to be finalized:

• [x] Date and Time for recruitment
• [x] Recruitment messages
• [x] Date and Time where researchers can be available for testing
• [x] A secured google drive for storing user data

[kcoronel]

Onboarding sessions: https://www.meetup.com/hackforla/

[xxl228]

Met with Karen today to discuss the onboarding details for the upcoming Usability Test. Will complete the following items for the upcoming week:

• [ ] Write an recruitment script for Karen to use at onboarding 9/9/2024
• [ ] Make a Caldeley signup for usability test with Vivian's personal Zoom, make sure to record to computer and upload it to the protected Google drive
• [ ] Send Karen the screener, recruitment message, along with the Calendly invite by next Friday 8/30
• [ ] Make sure to post it in slack channel the day following onboarding 9/10/24

[xxl228]

Recruitment Script for 9/9 onboarding: Are you a designer who has recently joined Hack for LA? We’re looking for participants to join a 30-minute moderated usability test for the Civic Tech Job (CTJ) website. This will be the future home website for Hack for LA. Your insights will help us improve the user experience for new team members like yourself. If you're interested in participating, please sign up here! We appreciate your time and feedback!

[ExperimentsInHonesty]

Review notes for the form: CivicTechJob Usability Test 2024

Branding missing from the top

Have you attended onboarding with Hacks for LA?

• Anyone who is invited to take this at the end of onboarding, is already onboarded
• Anyone who has been invited to attend from the Slack channel has already been onboarded
• The questions has Hack for LA misspelled as Hacks for LA

Please type your email address (the one used during onboarding)?

• We ask them for two email addresses during onboarding, so this question is confusing
  • A Gmail Address (so we can send google drive invites)
  • A Preferred Email Address (if they want their calendar invites sent somewhere else)

Please select the area of practice you are interested in volunteering with Hack for LA. (please select one area).

• Since these CoPs are not included, and this is intended to just be fielded to designers, I recommend adding that qualification at the top section (in the description text) and getting rid of this question. If you want to keep the question because you intend to use the same survey again for a different practice area, please change the wording of the question to include the word primary: Please select the primary area of practice you are interested in volunteering with Hack for LA. (please select one area).
  • Marketing
  • Fundraising/Revenue

How many years of experience (e.g. professional experience, part-time experience, internship experience, volunteer experience, etc.) do you have in the area of practice you selected?

• they can choose multiple practice areas

Please proceed to sign-up for one 30 min session to participate in the usability test.

• I am confused by the Get Link button CTA. I would the usability of this form with people outside your project, because you might get people not submitting after signing up.

[xxl228]

Thank you for your feedback, Bonnie! We really appreciate the detailed feedback you provided for the screener. We will keep the details in mind for a time when we need to do a survey at a broader scale.

After discussion with the team, we decided to eliminate the screener for the usability test recruitment process. We will recruit in the ui-ux channel with the following message:

Please see the following message for Usability Testing recruitment materials: Are you a designer volunteering with Hack for LA? We’re looking for participants to join a 30-minute moderated usability test for the new CTJ website. Your insights will help us improve the user experience for new team members like yourself.

If you're interested in participating, please sign up here!

We will follow up with you for more detailed information about the usability test. We appreciate your time and feedback! If you have any questions, please reach out to @Xiaotian Li in Slack.

Please let us know if you have more feedback. Thank you so much!

[kcoronel]

Posted message in UI/UX on 9/16 and received 6 responses! If all participants show up and quality interviews, expect to have analysis by end of October.

If needed, will rely on project design leads for support in gathering participants.

[xxl228]

Vivian completed 3 usability test this week, 1 cancel. Rishi has 2 more usability test scheduled for next week. Vivian will send the recruitment message in ui-ux channel again on Monday 9/23 to recruit more participants.

[xxl228]

We did not have any luck with volunteers showing up for the usability testing sessions this past week. We will continue to recruit volunteers through the following ways:

1. Connect with volunteers who missed the sessions through Slack messaging and see if they would be willing to set up a testing session another time.
2. Vivian will follow up with CoP leads and ask them to spread the words about recruiting volunteers. She will also repost the recruitment message again in the ux-ui channel with new available times.
3. @kcoronel please help us to spread the words by sharing the new recruitment message once it is reposted on Monday 9/30. And let us know if you have suggestions for recruitment.

[kcoronel]

hey @xxl228 what were you thinking I could do? I can't think of where else to post since we are only gathering feedback from Designers at the moment. I could engage with the post if you'd like? I can also repost later in the week in the UI/UX community.

[xxl228]

Will recruit again after Oct 7 onboarding, preferably with the two new designers that we will onboard but would take any volunteers. Vivian will repost the recruitment message on Oct 8. Reached out to CoP leads, they will help push for recruitment on in CoP meetings on Wed Oct 9

[xxl228]

-Prefer to recruit the new designers as participants for the remainder usability testing volunteers. We need at least 1 more person. -If this doesn't pan out, will wait to recruit after the Oct 22 onboarding event (mainly want to give a bit more time before the recruitment message show up again in the channel and having more new volunteers)

[xxl228]

Vivian will repost the message for recruiting more participants for the usability test after Monday 10/22 onboarding event. If we don't get any more volunteers after this, then we will wait for a new designer being onboarded and test on that person.

[xxl228]

Reposted recruitment messages 11/4/2024, aiming to get one more participant. If not, I will have to individually reach out to people for help.

[xxl228]

Reached out to individual designers on other teams, and hoping to get 1-2 more participants. Here is the recruitment message: Hi___, My name is Vivian Li, I am a Lead UXR for CivicTechJob. We are testing a prototype, and wonder if you would be able to spare 30mins to help out with the usability test. Please feel free to find a time you are available here. If you have any questions, please feel free to DM me. Thank you so much for your time. I really appreciate it.

[xxl228]

Reached out to Ryan Chase to get in touch with 311 data project design lead, shared recruitment message. Will ask Karen to share the recruitment message with the PMs at the meeting on Friday 11/22 for a last try.

[xxl228]

Recruited two more participants and able to complete the testing in the past week. We will be working on the report throughout December.

[kcoronel]

@xxl228 add link to report to this issue