mozilla-vpn-client
mozilla-vpn-client copied to clipboard
mozilla-mobile
Some Subscription related metrics were not implemented
As reported by [~accountid:60ec9eaaf314ff006a44bb42] on Slack, these are the missing metrics:
- outcome.subscription_started
- outcome.subscription_completed
Looks like oversight.
Miro board:https://miro.com/app/board/uXjVM0BZcnc=/
Update 5/3/2024: [~accountid:60c36c45dae567006809d3e2] [~accountid:634d8e68fe5ff375235781b3] and I discussed and agreed on the following:
- We will remove the interaction event connected to the cancel button in the
continue_in_browserscreen. This change is already done in the Miro board. - We will add an
outcome.subscription_failedevent with an additional key calledreasonthat will contain the reason for failing to set up a Subscription. A non-exhaustive list of reasons are: canceled_by_user, subscription_already_exists, subscription_not_validated, and billing_not_available. - We will remove the legacy telemetry events that used to capture the various reasons for subscription failure as distinct events.
┆Issue is synchronized with this Jira Bug ┆Reporter: Beatriz Rizental Machado
➤ Beatriz Rizental Machado commented:
re: interaction.manage_subscription_selected This one is in there. I can even see some counts in Looker: https://mozilla.cloud.looker.com/explore/mozilla_vpn/event_counts?qid=vzNQGs1XvAEI1J2oS55StG&toggle=fil,vis ( https://mozilla.cloud.looker.com/explore/mozilla_vpn/event_counts?qid=vzNQGs1XvAEI1J2oS55StG&toggle=fil,vis ) – very low numbers though.
➤ Beatriz Rizental Machado commented:
re: The subscription outcomes. As I add these metrics I notice there are some missing outcomes in there, here they are:
subscription_failed: type: event lifetime: ping send_in_pings: - main description: | The subscription process has errored. bugs: - https://mozilla-hub.atlassian.net/browse/VPN-5251 data_reviews: - TODO data_sensitivity: - interaction notification_emails: - [email protected] - [email protected] expires: never subscription_not_validated: type: event lifetime: ping send_in_pings: - main description: | The subscription could not be validated. bugs: - https://mozilla-hub.atlassian.net/browse/VPN-5251 data_reviews: - TODO data_sensitivity: - interaction notification_emails: - [email protected] - [email protected] expires: never subscription_canceled: type: event lifetime: ping send_in_pings: - main description: | The subscription was cancelled by the user. bugs: - https://mozilla-hub.atlassian.net/browse/VPN-5251 data_reviews: - TODO data_sensitivity: - interaction notification_emails: - [email protected] - [email protected] expires: never subscription_blocked: type: event lifetime: ping send_in_pings: - main description: | The subscription was blocked because the user is already a subscriber. bugs: - https://mozilla-hub.atlassian.net/browse/VPN-5251 data_reviews: - TODO data_sensitivity: - interaction notification_emails: - [email protected] - [email protected] expires: neverShould I go ahead and add these as well? Do these metrics make sense as they are or should we have single subscription_failed one? cc Santiago Andrigo
➤ Beatriz Rizental Machado commented:
re: The confirming_subscription one. That screen doen’t seem to exist in the app anymore?
➤ Santiago Andrigo commented:
Beatriz Rizental Machado
- I don’t see any confirming_subscription on neither the old telemetry miro ( https://miro.com/app/board/uXjVMVLwsKo=/ ) nor the new one ( https://miro.com/app/board/uXjVM0BZcnc=/ ). I also don’t see any events on Looker ( https://mozilla.cloud.looker.com/explore/mozilla_vpn/event_counts?qid=yShncLFtfXiMIaoxbltski&toggle=fil,vis ) in the last 180 days, so I’m guessing that whatever screen this corresponded to, it’s no longer there.
- We can have a outcome.subscription_failed on mobile, but on Desktop (given that Subscription happens in FxA), wouldn’t the failure be tracked by FxA themselves?
- As to the explosion of failure types … is this before the user attempts to create a new Subscription?
- subscription_not_validated: what does it mean? can we have a more human readable / intuitive error name?
- subscription_canceled: this is not a failure state imho. it’s just a particular beginning state for a new subscription. I wouldn’t track this.
- subscription_blocked: can we use the name subscription_already_present? Also, this is probably accompanied by a UI, right? That’s missing in the Miro board. Do you have it?
➤ Beatriz Rizental Machado commented:
{quote}I don’t see any confirming_subscription on neither the old telemetry miro ( https://miro.com/app/board/uXjVMVLwsKo=/ ) nor the new one ( https://miro.com/app/board/uXjVM0BZcnc=/ ). I also don’t see any events on Looker ( https://mozilla.cloud.looker.com/explore/mozilla_vpn/event_counts?qid=yShncLFtfXiMIaoxbltski&toggle=fil,vis ) in the last 180 days, so I’m guessing that whatever screen this corresponded to, it’s no longer there.{quote}
On the new one there is such a screen. See the mobile flow.
{quote}subscription_not_validated: what does it mean? can we have a more human readable / intuitive error name?{quote}
This happens when the subscription attempt fails, not sure what makes this different from the subscrition_failed event but there s a distinction in the code.
➤ Santiago Andrigo commented:
Ah sorry, I was looking for the outcome events, but this was just a regular one. Gotcha. Are you saying that that screen (“You are all set”) no longer exists?
When, in the Miro flow, would subscription_failed go in? Want to make sure we are tracking there as it becomes the easiest way for somebody not looking at the code to know what’s available for telemetry.
➤ Valentina Virlics commented:
Currently (latest 2.23), if we use the Telemetry debugging (Dev tools) option to test this, every time we sign out, the tag is removed.
If we add the tag back, and click the “Cancel” button while in the “waiting to log in on the web“ client state, the tag is removed as well. (see ss below)
!image-20240515-121509.png|width=1026,height=785,alt="image-20240515-121509.png"!
We are not sure how we could test this? Are we missing something in the setup process? Thank you!
Santiago Andrigo Beatriz Rizental Machado
➤ Beatriz Rizental Machado commented:
Hm, I wonder why the tag is removed on logout. I will check it out. Can you test it by just logging the pings instead of sending them to the debugviewer for the time being?
➤ Beatriz Rizental Machado commented:
Another workround would be to set GLEAN_DEBUG_VIEW_TAG="a-tag" in the environment the application is running, that should work. Let me take a look first though.
➤ Valentina Virlics commented:
Beatriz Rizental Machado I can try, but how can I log the pings? 😳 And how to set the GLEAN_DEBUG_VIEW_TAG="a-tag" in the environment the application is running? Thank you!
➤ Valentina Virlics commented:
I managed to have a quick call with Bea in order to understand how to set a tag via Terminal. Thank you, Bea!!:smiling_face_with_3_hearts: We created the tag with success!
After making the following steps:
- click the “Sign in” button from the client
- click “Cancel” from the waiting screen that loads in the client until the log in web page is loaded
- click the Sign in again and finalize log in;
The tag/pings are visible in Glean, but no events like outcome.subscription_started/outcome.subscription_completed or outcome.subscription_failed (with specific reasons) are displayed.
!Screenshot 2024-05-17 095417.png|width=802,height=234,alt="Screenshot 2024-05-17 095417.png"!
Thinking into this, are we sure that - while on desktop the In App Auth is moved outside the app - should we still see those subscription events?
Beatriz Rizental Machado Santiago Andrigo
➤ Beatriz Rizental Machado commented:
Yes, we should still see the events even with browser subscription. There must be some implementation or testing bug 🤔
➤ Beatriz Rizental Machado commented:
Huh, ok. I tried it out and I also don’t see the outcome events. We might need to put this back in progress. I wonder why the tests pass 🤔
➤ Beatriz Rizental Machado commented:
Ok, the events are only recorded when we get to the “Subscription Needed” view. If we never get there there is no way for the app to know if the user is just logging in or subscribing. Therefore, you are correct Valentina Virlics , with authentication moved ton in browser I think don’t have reliable way to record these events. I have only briefly looked at this, maybe Lesley Norton can take a look later on as she knows more about in browser auth.
➤ Valentina Virlics commented:
Lesley Norton We should either move this back to Backlog, if we want to fix this in the future, or close it as Cancelled, as currently (with web login) those events are not visible.