web-monetization-projects
web-monetization-projects copied to clipboard
interledger
Testing FF extensions with puppeteer
Note
Comment was overwritten with old version. For full details, see the screenshots of the diff (only thing accessible from Github) @ https://github.com/coilhq/web-monetization-projects/issues/2659
Background
It used to be the case that you could use web-ext as a module, to programmatically launch Firefox from a script, configured to start Juggler. Juggler was the method with which puppeteer connected to and controlled Firefox, though it is now deprecated
During the Juggler days, support to connect to Firefox was implemented via the puppeteer-firefox module. We used to use this before we migrated to Microsoft Playwright (built by the ex puppeteer team from Google) which (unofficially, at least) promised to support testing extensions with multiple browsers. At some point Playwright officially dropped support for extensions. We then reverted back to puppeteer, which by this time, had merged support for Firefox into the main puppeteer module. However, it no longer used Juggler. This meant that our puppeteer tests no longer worked, and so were disabled on CI.
Problem 1: How to connect ?
We need to figure out a way to connect with the updated version of puppeteer (now at v13.4) to an instance of Firefox that was launched by web-ext. The reason we use web-ext is that there's no known alternative to load a folder as a temporary add-on. web-ext uses an RDP (TODO is this just CDP without the Chrome??) connection to load the extension dist folder as a temporary add-on. web-ext also provides live reload facility. The use of web-ext is therefore justified.
Problem 2: How to configure the view port size?
It seems that even upon connecting there is still the issue of sending the viewport size options, usually sent to puppeteer.launch()
web-ext#run default args
web-ext#run itself passes along two args of interest to Firefox when it launches it:
-
--no-remote -
--start-debugger-server
--no-remote
As can be seen in the web-ext code here:
const results = await fxRunner({
// if this is falsey, fxRunner tries to find the default one.
'binary': firefoxBinary,
'binary-args': binaryArgs,
// This ensures a new instance of Firefox is created. It has nothing
// to do with the devtools remote debugger.
'no-remote': true,
The --no-remote arg is used to ensure a new instance of Firefox is launched.
--start-debugger-server
firefox --help shows the following:
--start-debugger-server [ws:][ <port> | <path> ] Start the devtools server on
a TCP port or Unix domain socket path. Defaults to TCP port
6000. Use WebSocket protocol if ws: prefix is specified.
Firefox about:config options related to debugging
- devtools.debugger.remote-enabled
- devtools.inspector.remote # TODO: ???
- devtools.debugger.prompt-connection
- remote.log.level # set to Trace, default is Info
Experiments to try
-
Use puppeteer.launch() to launch Firefox
- Use ps or some other method of inspecting the command line options used to launch FF
501 40446 40445 0 11:48AM ?? 0:04.66 /Users/nicholasdudfield/projects/web-monetization/.yarn/unplugged/puppeteer-npm-13.4.1-6e88e435df/node_modules/puppeteer/.local-firefox/mac-99.0a1/Firefox Nightly.app/Contents/MacOS/firefox --no-remote --foreground about:blank --remote-debugging-port=0 --profile /var/folders/wq/0t68fgjn491cxqphq1pg7ht00000gn/T/puppeteer_dev_firefox_profile-Jmk8py- Use the API to log the WebSocket connection url
const ff = await puppeteer.launch({product: 'firefox', headless: false}) console.log(ff.wsEndpoint()) > ws://localhost:54907/devtools/browser/01d4bbb0-d672-49b7-858d-7ecec266c73e- Curl Firefox at /json/version on the same port as the WebSocket url
curl http://localhost:54907/json/version { "Browser": "Firefox/99.0a1", "Protocol-Version": "1.0", "User-Agent": "Mozilla/5.0 (Macintosh; Intel Mac OS X 10.15; rv:99.0) Gecko/20100101 Firefox/99.0", "V8-Version": "1.0", "WebKit-Version": "1.0", "webSocketDebuggerUrl": "ws://localhost:54907/devtools/browser/01d4bbb0-d672-49b7-858d-7ecec266c73e" }% -
Check what ports are open
-
lsof -i -P | grep -i "listen"
-
puppeteer#connect options
export declare interface ConnectOptions extends BrowserConnectOptions {
browserWSEndpoint?: string;
browserURL?: string;
...
}
When configuring browserURL, an HTTP request will be made to /json/version to get the main browser target WebSocket connection url.
Notes:
- See Searchfox to search Firefox source code @ https://searchfox.org/mozilla-central/source/remote/cdp/JSONHandler.jsm#27-29
TODO:
- [x] Document experimentations to try and get FF connecting
Patch with experiment
Note this errantly uses remote-debugger-port rather than the correct remote-debugging-port
diff --git a/.pnp.cjs b/.pnp.cjs
index 748d7a97f..3a89e060e 100755
--- a/.pnp.cjs
+++ b/.pnp.cjs
@@ -29515,7 +29515,7 @@ function $$SETUP_STATE(hydrateRuntimeState, basePath) {
"linkType": "HARD",
}],
["npm:6.7.0", {
- "packageLocation": "./.yarn/cache/web-ext-npm-6.7.0-63737fba2a-be81319210.zip/node_modules/web-ext/",
+ "packageLocation": "./.yarn/unplugged/web-ext-npm-6.7.0-63737fba2a/node_modules/web-ext/",
"packageDependencies": [
["web-ext", "npm:6.7.0"],
["@babel/runtime", "npm:7.13.9"],
diff --git a/package.json b/package.json
index e727ef2ed..898e2762f 100644
--- a/package.json
+++ b/package.json
@@ -103,5 +103,10 @@
"engines": {
"node": ">= 14.0.0"
},
- "packageManager": "[email protected]"
+ "packageManager": "[email protected]",
+ "dependenciesMeta": {
+ "[email protected]": {
+ "unplugged": true
+ }
+ }
}
diff --git a/packages/coil-extension/test.sh b/packages/coil-extension/test.sh
index 83befb877..fe2815eb2 100755
--- a/packages/coil-extension/test.sh
+++ b/packages/coil-extension/test.sh
@@ -39,6 +39,6 @@ else
COMMAND="ts-node-dev -r tsconfig-paths/register -P test/tsconfig.json --respawn --transpile-only"
fi
-export DEBUG='coil*'
+export DEBUG=${DEBUG:-'coil*'}
-retry 3 yarn "$COMMAND" "$TESTFILE"
+retry 1 yarn "$COMMAND" "$TESTFILE"
diff --git a/packages/coil-puppeteer-utils/src/lib/initBrowser.ts b/packages/coil-puppeteer-utils/src/lib/initBrowser.ts
index 377390280..7f8693828 100644
--- a/packages/coil-puppeteer-utils/src/lib/initBrowser.ts
+++ b/packages/coil-puppeteer-utils/src/lib/initBrowser.ts
@@ -7,6 +7,7 @@ import { BrowserContext, default as puppeteer } from 'puppeteer'
import * as env from './env'
import { debug } from './debug'
+import { timeout } from './timeout'
const JUGGLER_MESSAGE = `Juggler listening on`
@@ -24,6 +25,7 @@ function jugglerEndpointWatcher() {
return () => {
// Parse firefox logs and extract juggler endpoint.
const captured = webExt.util.logger.consoleStream.capturedMessages
+ captured.forEach((m) => console.log('captured:', m))
const message = captured.find(msg => msg.includes(JUGGLER_MESSAGE))
if (!message) {
throw new Error(`Can not find Juggler endpoint:\n ${captured.join('\n')}`)
@@ -75,7 +77,8 @@ export async function initBrowser({
return launched.defaultBrowserContext()
} else {
const port = await getPort()
- const getJugglerEndpoint = jugglerEndpointWatcher()
+ webExt.util.logger.consoleStream.makeVerbose()
+
// eslint-disable-next-line @typescript-eslint/no-explicit-any
const puppeteerAny = puppeteer as any
const root: string = puppeteerAny._launcher._projectRoot
@@ -85,21 +88,29 @@ export async function initBrowser({
const exec =
os.platform() === 'linux'
? `${localFF}/${revision}/firefox/firefox`
- : puppeteerAny.executablePath()
+ : `${localFF}/${revision}/Firefox Nightly.app/Contents/MacOS/firefox`
const options: RunOptions = {
firefox: exec,
sourceDir: env.EXTENSION_PATH,
- args: [`-juggler=${port}`]
+ args: [`--remote-debugger-port=${port}`, '--verbose'],
}
+ // const ff = await
+ // puppeteer.launch({product: 'firefox'})
+ // console.log(ff.wsEndpoint())
+
debug('launching firefox, with options: ', options)
- await webExt.cmd.run(options, {
+ const runner = await webExt.cmd.run(options, {
shouldExitProgram: false
})
-
+ console.log('remoteFirefox', runner.extensionRunners[0].remoteFirefox.client._rdpConnection)
+ const {debuggerPort, ...others} = runner.extensionRunners[0].runningInfo
+ debug('finished launching', debuggerPort, JSON.stringify(others))
+ await timeout(10e3)
+ // const jugglerEndpoint = getJugglerEndpoint()
const ff = await puppeteer.connect({
- browserWSEndpoint: getJugglerEndpoint()
+ browserURL: `http://localhost:${port}`,
})
return ff.defaultBrowserContext()
diff --git a/yarn.lock b/yarn.lock
index b2b3fce91..c634f62b1 100644
--- a/yarn.lock
+++ b/yarn.lock
@@ -23152,6 +23152,9 @@ sjcl@sublimator/sjcl:
webpack-cli: 4.9.2
webpack-dev-server: 4.7.4
webpack-merge: 5.8.0
+ dependenciesMeta:
+ [email protected]:
+ unplugged: true
languageName: unknown
linkType: soft
