Verify Correctness of Telemetry Data

[Sdaswani]

We haven't tested out telemetry data for correctness (e.g., do we fire events when we expect them to, does that data show up in redash as expected, etc.), so we should build some infrastructure to test this.

[mcomella]

We haven't tested out telemetry data for correctness

Caveats:

• We manually tested this when it was initially implemented and the code hasn't changed much since then
• The code is very similar (identical?) to Fire TV so if it's correct on TV, it's probably correct here

so we should build some infrastructure to test this.

See also https://github.com/mozilla-mobile/firefox-tv/issues/625. Note that No-Jun and the QA team is interesting in automated testing (or any kind of testing) for telemetry as well! Initial thoughts are a test that pokes around the whole app and verifies the expected telemetry methods are called exactly the number of times they should be for each event (maybe we can build this into the robots?).

[nojunpark]

@mmccorks created a draft doc to outline what can be done here: https://docs.google.com/document/d/1rQhW275r2pfbHQfxvdvG7_CT6JnjKOAeX3iS99ZYNz4/edit?usp=sharing We might be asking devs soon about some questions here.

[athomasmoz]

@npark-mozilla can you loop @mcomella into the conversations on the test plan? This is a good issue to address this quarter

[nojunpark]

Will do, we have about 2 or 3 questions to ask devs, @mmccorks or I will send it out soon

[mcomella]

@npark-mozilla @mmccorks To address a few items in the doc from an engineering perspective:

• There is telemetry documentation in each products repo, e.g. firefox-tv
• We can write automation to verify telemetry. From easiest to hardest:
  • We can add log statements for each telemetry event (on debug builds only?) and QA can manually verify the right events are added
  • We can automate ^. We write automated tests that click through the UI and verify the appropriate telemetry events are created at the appropriate times; this doesn't verify upload is working correctly, however (which can be okay - it doesn't need to be end-to-end). This can run in automation
  • We can set up a staging server locally and point the app to it. QA (or automated tests) can verify the staging server has the appropriate content
  • We automate ^: we have the staging server programmatically verify the events it receives. Historically, this type of testing has been really flaky because you have to connect a mobile device to another device automatically and gracefully handle errors

[Sdaswani]

@mcomella @npark-mozilla given that we run automated tests for all of our products, I wonder if we can combine them with the automated staging server solution to test that the automated test results in an expected telemetry script? Perhaps that staging server can reside locally on the machine that is running the automated test.

This may be complex but I’m just throwing ideas out there.

[nojunpark]

when @rpappalax returns from PTO, we can discuss more, but a couple things:

• telemetry documentation was very helpful, thanks! so I'm assuming they are all up to date for Focus, FFTV, and connect?
• I was initially thinking about having a way to manually verify pings and its contents first, then translate that into automation. On our next meeting, we'll talk about QA having access to the ping server to analyze the results
• I believe desktop qa team does have this kind of automation in place, and Richard will gather more information regarding how this was done reliably. I think we do need some help from devs to create a startup pref to redirect the pings to the local webserver. And @Sdaswani yes I think this could easily be a family of a UI test - redirect to local telemetry server, do UI, collect ping, analyze json and assert on its contents - although this may take a bit of an effort, but probably worth doing it, and will be able to reuse on other android products as well.

[mcomella]

I wonder if we can combine them with the automated staging server solution to test that the automated test results in an expected telemetry script?

Right, this is similar to my fourth bullet point.

On Thu, Nov 8, 2018 at 1:18 PM No-Jun Park [email protected] wrote:

• telemetry documentation was very helpful, thanks! so I'm assuming they are all up to date for Focus, FFTV, and connect?

Should be, yes: it's a requirement to have these docs up-to-date and if they're not, it's a mistake.