New User Survey: Adds a new user survey for new user signups right after they signup

[ddc22]

Related to pau2Xa-5AZ-p2

Proposed Changes

• Adds a new user survey as an experiment to fulfil a customer data gathering exercise

• The experiment is loaded on load of the start page. The experiment is loaded only if the flow is onboarding and the user is not currently signed up. https://github.com/Automattic/wp-calypso/blob/4c61fe774e6cfa3a197243f2e5488ba48079cec3/client/signup/controller.js#L237-L248

• The survey will be excluded in all other instances and the control variant.

• The data will be submitted as tracks props

Testing Instructions

Assign yourself to the variant in 21652-explat-experiment

• Try with a fresh user on the onboarding flow /start the survey view should be visible
• Try with a fresh user on the any other signup flow /start the survey view should NOT be visible
• Try with an existing user (user with at least 1 active site) start logged out and go to /start survey should not be visible
• With a fresh user fill the survey make sure they are accurately reflected when submitted.

Assign yourself to the control in 21652-explat-experiment

• Try with a fresh user on the onboarding flow /start the survey view should NOT be visible

Pre-merge Checklist

• [ ] Has the general commit checklist been followed? (PCYsg-hS-p2)
• [ ] https://wpcalypso.wordpress.com/devdocs/docs/testing/index.md for your changes?
• [ ] Have you tested the feature in Simple (P9HQHe-k8-p2), Atomic (P9HQHe-jW-p2), and self-hosted Jetpack sites (PCYsg-g6b-p2)?
• [ ] Have you checked for TypeScript, React or other console errors?
• [ ] Have you used memoizing on expensive computations? More info in Memoizing with create-selector and Using memoizing selectors and Our Approach to Data
• [ ] Have we added the "[Status] String Freeze" label as soon as any new strings were ready for translation (p4TIVU-5Jq-p2)?
• [ ] For changes affecting Jetpack: Have we added the "[Status] Needs Privacy Updates" label if this pull request changes what data or activity we track or use (p4TIVU-aUh-p2)?

[github-actions[bot]]

Calypso Live (direct link)

https://calypso.live?image=registry.a8c.com/calypso/app:build-99249

Jetpack Cloud live (direct link)

https://calypso.live?image=registry.a8c.com/calypso/app:build-99249&env=jetpack

[matticbot]

Here is how your PR affects size of JS and CSS bundles shipped to the user's browser:

App Entrypoints (~220 bytes added 📈 [gzipped])

name                   parsed_size           gzip_size
entry-subscriptions         +449 B  (+0.0%)     +220 B  (+0.0%)
entry-stepper               +381 B  (+0.0%)     +200 B  (+0.0%)
entry-main                  +381 B  (+0.0%)     +200 B  (+0.0%)
entry-login                 +381 B  (+0.0%)     +200 B  (+0.0%)
entry-domains-landing       +381 B  (+0.1%)     +200 B  (+0.1%)
entry-browsehappy           +381 B  (+0.3%)     +200 B  (+0.5%)

Common code that is always downloaded and parsed every time the app is loaded, no matter which route is used.

Sections (~724 bytes added 📈 [gzipped])

name                             parsed_size           gzip_size
signup                               +2387 B  (+0.7%)     +341 B  (+0.5%)
update-design-flow                   +2021 B  (+0.2%)     +221 B  (+0.1%)
link-in-bio-tld-flow                 +2021 B  (+0.2%)     +221 B  (+0.1%)
videopress-flow                       +171 B  (+0.0%)      +53 B  (+0.0%)
sensei-flow                           +171 B  (+0.0%)      +53 B  (+0.0%)
podcasts-flow                         +171 B  (+0.0%)      +53 B  (+0.0%)
domains                               +171 B  (+0.0%)      +53 B  (+0.0%)
copy-site-flow                        +171 B  (+0.0%)      +53 B  (+0.0%)
checkout                              +171 B  (+0.0%)      +45 B  (+0.0%)
jetpack-connect                       +103 B  (+0.0%)      +30 B  (+0.0%)
gutenberg-editor                      +103 B  (+0.0%)      +30 B  (+0.0%)
accept-invite                         +103 B  (+0.1%)      +30 B  (+0.1%)
subscribers                            +68 B  (+0.0%)      +11 B  (+0.0%)
stats                                  +68 B  (+0.0%)      +11 B  (+0.0%)
site-purchases                         +68 B  (+0.0%)      +19 B  (+0.0%)
site-blocks                            +68 B  (+0.0%)      +29 B  (+0.0%)
settings-writing                       +68 B  (+0.0%)      +12 B  (+0.0%)
settings-podcast                       +68 B  (+0.0%)      +19 B  (+0.0%)
settings-newsletter                    +68 B  (+0.0%)      +22 B  (+0.0%)
settings-jetpack                       +68 B  (+0.0%)      +18 B  (+0.0%)
settings                               +68 B  (+0.0%)      +19 B  (+0.0%)
security                               +68 B  (+0.0%)      +29 B  (+0.0%)
scan                                   +68 B  (+0.0%)      +13 B  (+0.0%)
purchases                              +68 B  (+0.0%)      +29 B  (+0.0%)
privacy                                +68 B  (+0.0%)      +29 B  (+0.0%)
plugins                                +68 B  (+0.0%)      +27 B  (+0.0%)
people                                 +68 B  (+0.0%)      +22 B  (+0.0%)
notification-settings                  +68 B  (+0.0%)      +29 B  (+0.0%)
media                                  +68 B  (+0.0%)      +19 B  (+0.0%)
me                                     +68 B  (+0.0%)      +29 B  (+0.0%)
marketing                              +68 B  (+0.0%)      +25 B  (+0.0%)
jetpack-social                         +68 B  (+0.0%)      +25 B  (+0.0%)
jetpack-cloud-settings                 +68 B  (+0.0%)      +24 B  (+0.0%)
jetpack-cloud-plugin-management        +68 B  (+0.0%)      +27 B  (+0.0%)
jetpack-cloud-partner-portal           +68 B  (+0.0%)      +27 B  (+0.0%)
jetpack-cloud-agency-sites-v2          +68 B  (+0.0%)      +18 B  (+0.0%)
jetpack-cloud-agency-signup            +68 B  (+0.1%)      +24 B  (+0.1%)
import-hosted-site-flow                +68 B  (+0.0%)      +13 B  (+0.0%)
import-flow                            +68 B  (+0.0%)      +13 B  (+0.0%)
hosting                                +68 B  (+0.0%)      +29 B  (+0.0%)
help                                   +68 B  (+0.0%)      +29 B  (+0.0%)
earn                                   +68 B  (+0.0%)      +11 B  (+0.0%)
domain-user-transfer-flow              +68 B  (+0.0%)      +28 B  (+0.1%)
developer                              +68 B  (+0.0%)      +29 B  (+0.0%)
concierge                              +68 B  (+0.0%)      +29 B  (+0.0%)
comments                               +68 B  (+0.0%)      +16 B  (+0.0%)
backup                                 +68 B  (+0.0%)      +18 B  (+0.0%)
activity                               +68 B  (+0.0%)      +18 B  (+0.0%)
account-close                          +68 B  (+0.0%)      +29 B  (+0.0%)
account                                +68 B  (+0.0%)      +29 B  (+0.0%)

Sections contain code specific for a given set of routes. Is downloaded and parsed only when a particular route is navigated to.

Async-loaded Components (~8397 bytes added 📈 [gzipped])

name                                                                              parsed_size           gzip_size
async-load-signup-steps-new-user-survey                                              +26869 B    (new)    +8047 B    (new)
async-load-signup-steps-new-user-survey-components                                    +4551 B    (new)    +1708 B    (new)
async-load-calypso-my-sites-checkout-modal                                             +171 B  (+0.0%)      +45 B  (+0.0%)
async-load-calypso-blocks-app-banner                                                   +103 B  (+0.0%)      +30 B  (+0.1%)
async-load-store-app-store-stats-listview                                               +68 B  (+0.0%)      +25 B  (+0.0%)
async-load-store-app-store-stats                                                        +68 B  (+0.0%)      +29 B  (+0.0%)
async-load-signup-steps-website-content-section-types                                   +68 B  (+0.1%)      +20 B  (+0.1%)
async-load-signup-steps-website-content                                                 +68 B  (+0.1%)      +20 B  (+0.1%)
async-load-signup-steps-domains                                                         +68 B  (+0.0%)      +23 B  (+0.0%)
async-load-signup-steps-clone-point                                                     +68 B  (+0.0%)      +18 B  (+0.0%)
async-load-quick-language-switcher                                                      +68 B  (+0.1%)      +11 B  (+0.0%)
async-load-design-playground                                                            +68 B  (+0.0%)      +19 B  (+0.0%)
async-load-design-blocks                                                                +68 B  (+0.0%)      +44 B  (+0.0%)
async-load-design                                                                       +68 B  (+0.0%)      +27 B  (+0.0%)
async-load-calypso-reader-site-subscriptions-manager-pending-subscriptions-ma...        +68 B  (+0.1%)      +15 B  (+0.1%)
async-load-calypso-reader-site-subscriptions-manager-comment-subscriptions-ma...        +68 B  (+0.1%)      +15 B  (+0.1%)
async-load-calypso-post-editor-editor-media-modal                                       +68 B  (+0.0%)      +19 B  (+0.0%)
async-load-calypso-lib-account-settings-helper                                          +68 B  (+0.0%)      +29 B  (+0.1%)
async-load-calypso-layout-community-translator                                          +68 B  (+0.2%)      +26 B  (+0.3%)
async-load-calypso-blocks-editor-checkout-modal                                         +68 B  (+0.0%)      +15 B  (+0.0%)

React components that are loaded lazily, when a certain part of UI is displayed for the first time.

Legend

What is parsed and gzip size?

Parsed Size: Uncompressed size of the JS and CSS files. This much code needs to be parsed and stored in memory. Gzip Size: Compressed size of the JS and CSS files. This much data needs to be downloaded over network.

Generated by performance advisor bot at iscalypsofastyet.com.

[ddc22]

Self note: Set default state as null so that we have an indication of what fields the user interacts with.

[ddc22]

@southp sorry I had to short circuit the PR to allow Rita to see the form as you mentioned! A feature flag is a better alternative for this to force it to show. Will work on that!

Thanks for the other feedback. I will explain this a bit more.

[matticbot]

This PR modifies the release build for the following Calypso Apps:

For info about this notification, see here: PCYsg-OT6-p2

• blaze-dashboard
• editing-toolkit
• odyssey-stats
• wpcom-block-editor

To test WordPress.com changes, run install-plugin.sh $pluginSlug add/ui-shell-for-survey on your sandbox.

[ddc22]

If we just need a place to load the experiment on entering a flow, we do have a simple callback now here

The problem with this is that experiment loading is async and the entire signup framework is based on sync workflow. So we cannot introduce any experiment baed mechanics to leave out a step from within the framework

backend experiment

I will explore this option.

[aaronyan]

Hi @ddc22 and @southp! I also wasn't able to assign myself to the treatment variation as well when running this branch locally at http://calypso.localhost:3000/start/. I noticed that the tk_ai does not exist, which is required in order to receive assignment as a logged-out user. I turned on cookies and turned off my ad-block, and I still do not receive tk_ai. Any ideas why?

@ddc22: while interacting with the survey form, the answer responses change order each interaction. I think the intended design is for the answer choices to be randomly ordered, but then remain static if a user interacts with them. I think the fill in the blank should always be the last option too.

[ddc22]

I also wasn't able to assign myself to the treatment

Thanks for attempting to test this @southp @aaronyan. Have you pulled in the latest changes so that the latest experiment information is reflected in your sandbox cache? :)

while interacting with the survey form, the answer responses change order each interaction.

Haha 😆 , sorry about that! I Hadn't tested that change and forgot to mention the fact.

[aaronyan]

I got it working! Not sure why, but the assignment takes around a minute to set and be callable after manually assigning using the bookmarklet. cc: @jessie-ross in case they know why this might be. Maybe it has to do with using loadExperimentAssignment or the bookmarklet on logged-out users?

This worked both proxy only and proxied + sandboxed API:

1. Comment out onboarding/new-user-survey flag
2. Load http://calypso.localhost:3000/start/ and allow third-party cookies and turn off adblocker
3. Assign yourself treatment using the bookmarklet
4. Wait 1 minute
5. Reload the page, and you should be assigned to treatment (console.log() to confirm)
6. Type in email and click Continue
7. See survey!

[ddc22]

@southp I did not have time to investigate an alternative approach do the experiment assignment in the backend. Can we go ahead with the current approach ?

[aaronyan]

I've tested the random assignment for this (by commenting out and adding logging to sandboxed assigner code), and it works for me! I successfully was assigned to both experiences and they work as expected.

I wasn't able to test different locales, but I can confirm I am being assigned under en browswerlang.

[aaronyan]

@ddc22, while testing the experimentation implementation, I had a few notes about the the None of the above form:

• The preview text is cut off for me. Consider shorter suggestion.
• When the form is filled, the text should be the same color and size as the other options (and different color than the preview text).
• If someone types something in the form, the box should autocheck
• If someone clears the form, the box should uncheck

[ddc22]

Thanks for testing @aaronyan , some of those issues were regressions. They have now been fixed. If there is nothing else to fix I am planning to send these to be translated.

[southp]

I did not have time to investigate an alternative approach do the experiment assignment in the backend. Can we go ahead with the current approach ?

Sure. According to the latest summary from @aaronyan, it will likely run only for days pau2Xa-5AZ-p2#comment-14969 so let's go ahead with the current approach.

[ddc22]

@aaronyan I got a request from Rita to use the randomize the survey option to only 50% of the users (Tentative). I am planning to only shuffle if Math.random() < 50%. Are you ok with this Approach

Just saw your comment about setting up multiple cohorts for this.

[aaronyan]

I got a request from Rita to use the randomize the survey option to only 50% of the users (Tentative). I am planning to only shuffle if Math.random() < 50%. Are you ok with this Approach

Rita, Dehan, Matt Miller, and I hopped on a call to clarify a few points (pau2Xa-5AZ-p2#comment-15001). We will have a new variation treatment_scramble that will have randomized responses, and the experiment will be an A/B/C:

• control - no survey
• treatment - static responses
• treatment_scramble - randomized responses

[ddc22]

Hey @aaronyan can you do some early testing here. The code review is bound to be intense! Some early testing would be really appreciated.

I have added two feature flags for your convenience, but it's better if you could do the actual assignments.

		"onboarding/new-user-survey": false,
		"onboarding/new-user-survey-scrambled": false,

[aaronyan]

Hi @ddc22! I've used manual assignment to test all three variations, and only control works as expected.

• control returns no survey and directs to domain selection. ✅
• treatment returns the survey with scrambled responses 👎
• treatment_scrambled returns no survey 👎

I have added two feature flags for your convenience

I was looking for the feature flags to disable them, and I did not find new-user-survey-scrambled.

[ddc22]

Really sorry about that ! The push had failed due to a network issue.

[ddc22]

@oswian thank you so much jumping into an unknown context to review. The feedback is appreciated however since they are minor I will ship this PR and pick them up as a followup. I do this because I want to commit a working state of the PR without doing any changes.

[a8ci18n]

This Pull Request is now available for translation here: https://translate.wordpress.com/deliverables/11499075

Some locales (Hebrew, Japanese) have been temporarily machine-translated due to translator availability. All other translations are usually ready within a few days. Untranslated and machine-translated strings will be sent for translation next Monday and are expected to be completed by the following Friday.

Thank you @ddc22 for including a screenshot in the description! This is really helpful for our translators.

[a8ci18n]

Translation for this Pull Request has now been finished.