metamask-extension icon indicating copy to clipboard operation
metamask-extension copied to clipboard
verified publisher icon MetaMask

test: new switchToWindowWithTitle w/ Extension communication

Open HowardBraham opened this issue 1 year ago • 9 comments

Description

This creates a Socket between Mocha/Selenium and the Extension background script (service worker in MV3).

I had originally hoped that we could send a message to the Extension, and make the Extension switch windows. But it seems like Selenium's idea of "which tab/window are we looking at" is not the same as which one the browser has active. In fact, the "real user" can change tabs, and Selenium still works on the old tab. So an attempt to change tabs in the browser effectively does nothing.

My second attempt was to send back a list of open tabs and their names. This almost works, except the order of the tab lists is different between the browser list and the Selenium list. So we can't really act on that information to switch to a tab.

So the thing I went for is to ask the Extension to wait until there's a window open with the target title or url, and then inform Mocha/Selenium that it exists and to look for it.

These changes made a small number of tests more flaky, so I made some minor changes to them to work better with the new system.

Open in GitHub Codespaces

Related issues

Manual testing steps

This is a special CircleCI workflow where I run every E2E test 4 times each. Some of these jobs took 40 minutes, with very few failures (and only on existing flaky tests). This PR makes things more stable. https://app.circleci.com/pipelines/github/MetaMask/metamask-extension/87754/workflows/02b77156-ef91-45d3-9367-e8ebe2ee2f1f

Screenshots/Recordings

Before

After

Pre-merge author checklist

Pre-merge reviewer checklist

  • [ ] I've manually tested the PR (e.g. pull and build branch, run the app, test code being changed).
  • [ ] I confirm that this PR addresses all acceptance criteria described in the ticket it closes and includes the necessary testing evidence such as recordings and or screenshots.

HowardBraham avatar Jun 17 '24 05:06 HowardBraham

CLA Signature Action: All authors have signed the CLA. You may need to manually re-run the blocking PR check if it doesn't pass in a few minutes.

github-actions[bot] avatar Jun 17 '24 05:06 github-actions[bot]

New and removed dependencies detected. Learn more about Socket for GitHub ↗︎

Package New capabilities Transitives Size Publisher
npm/@types/[email protected] None 0 698 kB types
npm/@types/[email protected] None 0 24.9 kB types
npm/@types/[email protected] None 0 8.34 kB types
npm/@types/[email protected] None 0 32.4 kB types
npm/@types/[email protected] None 0 21.2 kB types
npm/[email protected] environment, network 0 141 kB lpinca

🚮 Removed packages: npm/@types/[email protected], npm/[email protected]

View full report↗︎

socket-security[bot] avatar Jun 17 '24 05:06 socket-security[bot]

Codecov Report

Attention: Patch coverage is 0% with 4 lines in your changes missing coverage. Please review.

Project coverage is 69.67%. Comparing base (a3c4333) to head (bef19fe). Report is 9 commits behind head on develop.

Files Patch % Lines
app/scripts/background.js 0.00% 2 Missing :warning:
app/scripts/offscreen.js 0.00% 2 Missing :warning:
Additional details and impacted files 
@@             Coverage Diff             @@
##           develop   #25362      +/-   ##
===========================================
- Coverage    69.68%   69.67%   -0.01%     
===========================================
  Files         1405     1405              
  Lines        49701    49705       +4     
  Branches     13738    13740       +2     
===========================================
  Hits         34630    34630              
- Misses       15071    15075       +4

:umbrella: View full report in Codecov by Sentry.
:loudspeaker: Have feedback on the report? Share it here.

codecov[bot] avatar Jun 17 '24 05:06 codecov[bot]

Builds ready [4fae996]
Page Load Metrics (201 ± 283 ms)
PlatformPageMetricMin (ms)Max (ms)Average (ms)StandardDeviation (ms)MarginOfError (ms)
ChromeHomefirstPaint692871034622
domContentLoaded98518178
load402768201590283
domInteractive98518178
Bundle size diffs [🚨 Warning! Bundle size has increased!]
  • background: 2.48 KiB (0.07%)
  • ui: 0 Bytes (0.00%)
  • common: 0 Bytes (0.00%)

metamaskbot avatar Jun 18 '24 09:06 metamaskbot

Builds ready [fba217e]
Page Load Metrics (141 ± 186 ms)
PlatformPageMetricMin (ms)Max (ms)Average (ms)StandardDeviation (ms)MarginOfError (ms)
ChromeHomefirstPaint5811585136
domContentLoaded8141121
load381827141387186
domInteractive8141121
Bundle size diffs [🚨 Warning! Bundle size has increased!]
  • background: 2.48 KiB (0.07%)
  • ui: 0 Bytes (0.00%)
  • common: 0 Bytes (0.00%)

metamaskbot avatar Jun 18 '24 10:06 metamaskbot

Builds ready [a55651a]
Page Load Metrics (45 ± 2 ms)
PlatformPageMetricMin (ms)Max (ms)Average (ms)StandardDeviation (ms)MarginOfError (ms)
ChromeHomefirstPaint67987674
domContentLoaded9111010
load40614552
domInteractive9111010
Bundle size diffs [🚨 Warning! Bundle size has increased!]
  • background: 2.5 KiB (0.07%)
  • ui: 0 Bytes (0.00%)
  • common: 0 Bytes (0.00%)

metamaskbot avatar Jun 21 '24 07:06 metamaskbot

But it seems like Selenium's idea of "which tab/window are we looking at" is not the same as which one the browser has active

Really nice catch :heart_eyes: this is precisely what happens in the vault decrypt test, where we had 2 windows and webdriver active window didn't match chrome's active window (this)

seaona avatar Jun 21 '24 10:06 seaona

Builds ready [11fb79a]
Page Load Metrics (48 ± 2 ms)
PlatformPageMetricMin (ms)Max (ms)Average (ms)StandardDeviation (ms)MarginOfError (ms)
ChromeHomefirstPaint69898063
domContentLoaded9161111
load40624852
domInteractive9161111
Bundle size diffs [🚨 Warning! Bundle size has increased!]
  • background: 2.5 KiB (0.07%)
  • ui: 0 Bytes (0.00%)
  • common: 0 Bytes (0.00%)

metamaskbot avatar Jun 21 '24 21:06 metamaskbot

Builds ready [5ea1bc5]
Page Load Metrics (63 ± 14 ms)
PlatformPageMetricMin (ms)Max (ms)Average (ms)StandardDeviation (ms)MarginOfError (ms)
ChromeHomefirstPaint62188984220
domContentLoaded95717136
load41136633014
domInteractive95717136
Bundle size diffs [🚨 Warning! Bundle size has increased!]
  • background: 2.5 KiB (0.07%)
  • ui: 0 Bytes (0.00%)
  • common: 0 Bytes (0.00%)

metamaskbot avatar Jun 26 '24 09:06 metamaskbot

Builds ready [299ce1a]
Page Load Metrics (239 ± 236 ms)
PlatformPageMetricMin (ms)Max (ms)Average (ms)StandardDeviation (ms)MarginOfError (ms)
ChromeHomefirstPaint683021044923
domContentLoaded96224147
load421803239491236
domInteractive96224147
Bundle size diffs [🚨 Warning! Bundle size has increased!]
  • background: 2.48 KiB (0.08%)
  • ui: 0 Bytes (0.00%)
  • common: 0 Bytes (0.00%)

metamaskbot avatar Jul 17 '24 01:07 metamaskbot

Builds ready [09533c6]
Page Load Metrics (196 ± 221 ms)
PlatformPageMetricMin (ms)Max (ms)Average (ms)StandardDeviation (ms)MarginOfError (ms)
ChromeHomefirstPaint941681302211
domContentLoaded127634168
load622203196461221
domInteractive127634168
Bundle size diffs [🚨 Warning! Bundle size has increased!]
  • background: 3.01 KiB (0.09%)
  • ui: 0 Bytes (0.00%)
  • common: 0 Bytes (0.00%)

metamaskbot avatar Jul 23 '24 00:07 metamaskbot

Builds ready [8b9b1a4]
Page Load Metrics (171 ± 185 ms)
PlatformPageMetricMin (ms)Max (ms)Average (ms)StandardDeviation (ms)MarginOfError (ms)
ChromeHomefirstPaint674611178139
domContentLoaded98628209
load441831171384185
domInteractive98628209
Bundle size diffs [🚨 Warning! Bundle size has increased!]
  • background: 3.01 KiB (0.09%)
  • ui: 0 Bytes (0.00%)
  • common: 0 Bytes (0.00%)

metamaskbot avatar Jul 23 '24 15:07 metamaskbot

Quality Gate Passed Quality Gate passed

Issues
0 New issues
0 Accepted issues

Measures
0 Security Hotspots
0.0% Coverage on New Code
0.0% Duplication on New Code

See analysis details on SonarCloud

sonarqubecloud[bot] avatar Jul 23 '24 15:07 sonarqubecloud[bot]

Builds ready [bef19fe]
Page Load Metrics (166 ± 202 ms)
PlatformPageMetricMin (ms)Max (ms)Average (ms)StandardDeviation (ms)MarginOfError (ms)
ChromeHomefirstPaint662871074722
domContentLoaded979312110
load401996166420202
domInteractive979312110
Bundle size diffs [🚨 Warning! Bundle size has increased!]
  • background: 3.01 KiB (0.09%)
  • ui: 0 Bytes (0.00%)
  • common: 0 Bytes (0.00%)

metamaskbot avatar Jul 23 '24 16:07 metamaskbot