mongosh icon indicating copy to clipboard operation
mongosh copied to clipboard

chore(e2e-tests): add `TestShell#eventually` and refactor `TestShell#executeLine` and `TestShell#waitForPrompt`

Open kraenhansen opened this issue 1 year ago • 1 comments

This adds a TestShell#eventually which is designed to be a drop-in replacement for the timing based eventually utility.

Tests can now call await shell.eventually(() => { /* block which conditionally throws */ }) and pass a callback which will get called whenever the child process spawned by the TestShell receives output. This effectively acts as a push-stream to allow for faster execution of tests (5min 4sec before vs 4min 20sec after) and less noise if logging in these callbacks while developing tests.

This refactors TestShell's executeLine to await a prompt before writing the input instead of simply awaiting the prompt afterwards. This is to fix a race which occurs if shell.writeInputLine is called directly from a test and an eventually polls for the output to contain something. In which case the prompt won't be awaited and an executeLine will incorrectly pick up the prompt from the previous command: https://github.com/mongodb-js/mongosh/blob/a520eb1d3683c51e49951fb797899d9787ae6994/packages/e2e-tests/test/e2e-auth.spec.ts#L738-L749

This also refactors TestShell's waitForPrompt to use the new shell.eventually (push streaming) and only consider the last line of the output determining if the output has a prompt. I believe this is now possible because of a fix for the race in executeLine.

Note: This is based on the changes introduced in #2170 and needs a rebase once that is merged.

Note: This also sneaks in a bump of the timeout passed to eventually in the "e2e Analytics Node" before hook, as it was often failing locally and the timeout of the hook itself was already 60s.

kraenhansen avatar Sep 18 '24 13:09 kraenhansen

It seems this is agitating a few of the flaky tests a bit more. Specifically, I'm seeing these failures quite frequently:

  • FLE tests 7.0+ allows explicit enryption with bypassQueryAnalysis: AssertionError: expected '... ... ... ... \n' to include 'ghjk'
  • e2e config, logging and rc file in fully accessible environment update notification shows an update notification if a newer version is available: Error: Timed out (waited 10000ms): ENOENT: no such file or directory, open '/data/mci/907dee6684bf664703662d882f050b2c/src/tmp/test/cli-repl-home-1727870787014-0.3484944880719283/.mongodb/REDACTED:notary_signing_key_name/update-metadata.json'

kraenhansen avatar Oct 02 '24 12:10 kraenhansen