ember-test-waiters icon indicating copy to clipboard operation
ember-test-waiters copied to clipboard
verified publisher icon emberjs

Upgrade from 3.1.0 -> 4.0.0 causes ~10% of tests to intermittently fail

Open kategengler opened this issue 1 year ago • 13 comments

Still investigating and endeavoring to reproduce. The upgrade on an app (full webpack Embroider) has a failing dependabot PR that I finally got a chance to look at -- ~10% of the tests intermittently fail with the upgrade when the suite is run as a whole but pass when run individually.

kategengler avatar Dec 18 '24 20:12 kategengler

what is the output of pnpm why -r @ember/test-waiters?

NullVoxPopuli avatar Dec 18 '24 21:12 NullVoxPopuli

Experiencing the same issue, also on latest embroider/webpack etc.

Running pnpm why -r @ember/test-waiters I could see that test-waiters v3 was being used by various addons still, so I added test-waiters v4 to my pnpm overrides to see if I could get around that........ but then I hit @ember-data/request needs to depend on ember-auto-import in order to use @ember/test-waiters which should be fixed by https://github.com/emberjs/data/pull/9614 and https://github.com/emberjs/data/pull/9619 but I'm stuck on ember-data 5.3.3 due to other problems.

Hope that helps

Techn1x avatar Jan 14 '25 03:01 Techn1x

I finally got back to this and had the same experience as @Techn1x

kategengler avatar Jan 31 '25 20:01 kategengler

@kategengler problems resulting in needing an ember-data upgrade?

does the problem still exist with test-helpers 4.0.5? (or v5.1)

NullVoxPopuli avatar Jan 31 '25 21:01 NullVoxPopuli

When I add the override to force @ember/test-waiters v4, I get @ember-data/request needs to depend on ember-auto-import in order to use @ember/test-waiters, which it sounds like would require an ember-data upgrade to resolve. That's not an option for the project I am on, right now.

@ember/test-helpers is at 4.1.0, I haven't tried 5.1 yet.

kategengler avatar Jan 31 '25 22:01 kategengler

is everyone using embroider?

NullVoxPopuli avatar Jan 31 '25 22:01 NullVoxPopuli

Also experiencing tests failing on random things, for example e.g.

await click('button');

console.log(find('.modal')); // randomly null 

await new Promise((resolve) => setTimeout(resolve, 10));

console.log(find('.modal')); // present

amk221 avatar Feb 25 '25 11:02 amk221

I got lucky and was able to update the projects I'm looking after to later versions of ember-data, from 5.3.3 -> 5.3.11, and I no longer have this issue. Big relief to get that update done, unblocked on a lot of things and the types are so much better.

Was able to do this without adding test-waiters 4 to pnpm overrides

is everyone using embroider?

yes

Techn1x avatar Mar 10 '25 05:03 Techn1x

For everyone with this issue, could you comment back with two pieces of information?

pnpm why @ember/test-waiters

pnpx y-which @ember/test-waiters

Or, a repro

Thanks!

NullVoxPopuli avatar Mar 10 '25 12:03 NullVoxPopuli

It's a full app reproduction, but you can see this in https://github.com/ilios/frontend/pull/8418. and I'm guessing our mixed list of versions is part of the issue so I've been waiting on an ember-data release with this upgrade in it. In case it helps though, here is what I've got.

$ pnpm --filter frontend why @ember/test-waiters

Legend: production dependency, optional only, dev only

[email protected] /Users/jrjohnson/Documents/code/ilios/frontend/packages/frontend (PRIVATE)

devDependencies:
@ember-data/adapter 5.3.11
├─┬ @ember-data/legacy-compat 5.3.11 peer
│ ├─┬ @ember-data/graph 5.3.11 peer
│ │ └─┬ @ember-data/store 5.3.11 peer
│ │   └─┬ @ember-data/request 5.3.11 peer
│ │     └── @ember/test-waiters 4.0.0
│ ├─┬ @ember-data/json-api 5.3.11 peer
│ │ ├─┬ @ember-data/graph 5.3.11 peer
│ │ │ └─┬ @ember-data/store 5.3.11 peer
│ │ │   └─┬ @ember-data/request 5.3.11 peer
│ │ │     └── @ember/test-waiters 4.0.0
│ │ └─┬ @ember-data/store 5.3.11 peer
│ │   └─┬ @ember-data/request 5.3.11 peer
│ │     └── @ember/test-waiters 4.0.0
│ ├─┬ @ember-data/request 5.3.11 peer
│ │ └── @ember/test-waiters 4.0.0
│ ├─┬ @ember-data/store 5.3.11 peer
│ │ └─┬ @ember-data/request 5.3.11 peer
│ │   └── @ember/test-waiters 4.0.0
│ └── @ember/test-waiters 4.0.0 peer
└─┬ @ember-data/store 5.3.11 peer
  └─┬ @ember-data/request 5.3.11 peer
    └── @ember/test-waiters 4.0.0
@ember-data/json-api 5.3.11
├─┬ @ember-data/graph 5.3.11 peer
│ └─┬ @ember-data/store 5.3.11 peer
│   └─┬ @ember-data/request 5.3.11 peer
│     └── @ember/test-waiters 4.0.0
└─┬ @ember-data/store 5.3.11 peer
  └─┬ @ember-data/request 5.3.11 peer
    └── @ember/test-waiters 4.0.0
@ember-data/legacy-compat 5.3.11
├─┬ @ember-data/graph 5.3.11 peer
│ └─┬ @ember-data/store 5.3.11 peer
│   └─┬ @ember-data/request 5.3.11 peer
│     └── @ember/test-waiters 4.0.0
├─┬ @ember-data/json-api 5.3.11 peer
│ ├─┬ @ember-data/graph 5.3.11 peer
│ │ └─┬ @ember-data/store 5.3.11 peer
│ │   └─┬ @ember-data/request 5.3.11 peer
│ │     └── @ember/test-waiters 4.0.0
│ └─┬ @ember-data/store 5.3.11 peer
│   └─┬ @ember-data/request 5.3.11 peer
│     └── @ember/test-waiters 4.0.0
├─┬ @ember-data/request 5.3.11 peer
│ └── @ember/test-waiters 4.0.0
├─┬ @ember-data/store 5.3.11 peer
│ └─┬ @ember-data/request 5.3.11 peer
│   └── @ember/test-waiters 4.0.0
└── @ember/test-waiters 4.0.0 peer
@ember-data/model 5.3.11
├─┬ @ember-data/graph 5.3.11 peer
│ └─┬ @ember-data/store 5.3.11 peer
│   └─┬ @ember-data/request 5.3.11 peer
│     └── @ember/test-waiters 4.0.0
├─┬ @ember-data/json-api 5.3.11 peer
│ ├─┬ @ember-data/graph 5.3.11 peer
│ │ └─┬ @ember-data/store 5.3.11 peer
│ │   └─┬ @ember-data/request 5.3.11 peer
│ │     └── @ember/test-waiters 4.0.0
│ └─┬ @ember-data/store 5.3.11 peer
│   └─┬ @ember-data/request 5.3.11 peer
│     └── @ember/test-waiters 4.0.0
├─┬ @ember-data/legacy-compat 5.3.11 peer
│ ├─┬ @ember-data/graph 5.3.11 peer
│ │ └─┬ @ember-data/store 5.3.11 peer
│ │   └─┬ @ember-data/request 5.3.11 peer
│ │     └── @ember/test-waiters 4.0.0
│ ├─┬ @ember-data/json-api 5.3.11 peer
│ │ ├─┬ @ember-data/graph 5.3.11 peer
│ │ │ └─┬ @ember-data/store 5.3.11 peer
│ │ │   └─┬ @ember-data/request 5.3.11 peer
│ │ │     └── @ember/test-waiters 4.0.0
│ │ └─┬ @ember-data/store 5.3.11 peer
│ │   └─┬ @ember-data/request 5.3.11 peer
│ │     └── @ember/test-waiters 4.0.0
│ ├─┬ @ember-data/request 5.3.11 peer
│ │ └── @ember/test-waiters 4.0.0
│ ├─┬ @ember-data/store 5.3.11 peer
│ │ └─┬ @ember-data/request 5.3.11 peer
│ │   └── @ember/test-waiters 4.0.0
│ └── @ember/test-waiters 4.0.0 peer
└─┬ @ember-data/store 5.3.11 peer
  └─┬ @ember-data/request 5.3.11 peer
    └── @ember/test-waiters 4.0.0
@ember-data/request 5.3.11
└── @ember/test-waiters 4.0.0
@ember-data/serializer 5.3.11
├─┬ @ember-data/legacy-compat 5.3.11 peer
│ ├─┬ @ember-data/graph 5.3.11 peer
│ │ └─┬ @ember-data/store 5.3.11 peer
│ │   └─┬ @ember-data/request 5.3.11 peer
│ │     └── @ember/test-waiters 4.0.0
│ ├─┬ @ember-data/json-api 5.3.11 peer
│ │ ├─┬ @ember-data/graph 5.3.11 peer
│ │ │ └─┬ @ember-data/store 5.3.11 peer
│ │ │   └─┬ @ember-data/request 5.3.11 peer
│ │ │     └── @ember/test-waiters 4.0.0
│ │ └─┬ @ember-data/store 5.3.11 peer
│ │   └─┬ @ember-data/request 5.3.11 peer
│ │     └── @ember/test-waiters 4.0.0
│ ├─┬ @ember-data/request 5.3.11 peer
│ │ └── @ember/test-waiters 4.0.0
│ ├─┬ @ember-data/store 5.3.11 peer
│ │ └─┬ @ember-data/request 5.3.11 peer
│ │   └── @ember/test-waiters 4.0.0
│ └── @ember/test-waiters 4.0.0 peer
└─┬ @ember-data/store 5.3.11 peer
  └─┬ @ember-data/request 5.3.11 peer
    └── @ember/test-waiters 4.0.0
@ember-data/store 5.3.11
└─┬ @ember-data/request 5.3.11 peer
  └── @ember/test-waiters 4.0.0
@ember/test-helpers 4.0.5
└── @ember/test-waiters 4.0.0
@embroider/router 2.1.8
└── @ember/test-waiters 3.1.0
ember-a11y-testing 7.1.0
├─┬ @ember/test-helpers 4.0.5 peer
│ └── @ember/test-waiters 4.0.0
└── @ember/test-waiters 3.1.0
ember-async-data 2.0.0
└── @ember/test-waiters 4.0.0
ember-cli-page-object 2.3.1
└─┬ @ember/test-helpers 4.0.5 peer
  └── @ember/test-waiters 4.0.0
ember-exam 9.1.0
└─┬ ember-qunit 9.0.1 peer
  └─┬ @ember/test-helpers 4.0.5 peer
    └── @ember/test-waiters 4.0.0
ember-mirage 0.3.3
└─┬ @ember/test-helpers 2.9.4
  └── @ember/test-waiters 3.1.0
ember-qunit 9.0.1
└─┬ @ember/test-helpers 4.0.5 peer
  └── @ember/test-waiters 4.0.0
ilios-common link:../ilios-common
├── @ember/test-waiters 4.0.0
├─┬ ember-async-data 2.0.0
│ └── @ember/test-waiters 4.0.0
├─┬ ember-cli-page-object 2.3.1
│ └─┬ @ember/test-helpers 4.0.5 peer
│   └── @ember/test-waiters 4.0.0
├─┬ ember-file-upload 9.3.0
│ ├─┬ @ember/test-helpers 4.0.5 peer
│ │ └── @ember/test-waiters 4.0.0
│ └── @ember/test-waiters 4.0.0
├─┬ ember-intl 7.1.4
│ └─┬ @ember/test-helpers 4.0.5 peer
│   └── @ember/test-waiters 4.0.0
├─┬ ember-simple-auth 7.1.3
│ ├─┬ @ember/test-helpers 4.0.5 peer
│ │ └── @ember/test-waiters 4.0.0
│ └── @ember/test-waiters 3.1.0
└─┬ ember-simple-charts 12.1.1
  └─┬ ember-async-data 2.0.0
    └── @ember/test-waiters 4.0.0

As it's installed in an add-on in this monorepo I switched to that package first packages/ilios-common and then ran

pnpx y-which @ember/test-waiters

  Node resolves @ember/test-waiters 
       to   ~/Documents/code/ilios/frontend/node_modules/.pnpm/@[email protected]/node_modules/@ember/test-waiters/dist/index.js
       which is @ 4.0.0

     from   ilios-common @ 87.1.0 
       at   ~/Documents/code/ilios/frontend/packages/ilios-common

Edit: updated after removing and re-creating lock file.

jrjohnson avatar Mar 10 '25 16:03 jrjohnson

test-helpers supports v4:

  • https://github.com/emberjs/ember-test-helpers/blob/master/addon/package.json#L54

ember-async-data supports v4

  • https://github.com/tracked-tools/ember-async-data/blob/main/ember-async-data/package.json#L54 (only in latest release)

your version of ember-data supports v4:

  • https://github.com/emberjs/data/blob/v5.3.11/packages/request/package.json#L46

try deleting all occurances of test-waiters from your lockfile and re-install. (or if you can delete your lockfile entirely, that would be easier)

NullVoxPopuli avatar Mar 10 '25 16:03 NullVoxPopuli

🤦, of course. I editing my above comment with the re-created lock file's version of why and pushed the branch back up.

jrjohnson avatar Mar 10 '25 17:03 jrjohnson

This appears like it could be sus:

ember-a11y-testing 7.1.0
├─┬ @ember/test-helpers 4.0.5 peer
│ └── @ember/test-waiters 4.0.0
└── @ember/test-waiters 3.1.0

Opened a PR here: https://github.com/ember-a11y/ember-a11y-testing/pull/589

in the mean time, you could attempt to use an override for test-waiters v4 -- it would require that all addons depending on test-waiters already support loading v2 addons (via auto-import).

that said,

test-waiters 3.x is supposed to be compatible with 4.x when both loaded at the same time. So I have a hunch something is goofy. that work was done here: https://github.com/emberjs/ember-test-waiters/releases/tag/v3.0.2

NullVoxPopuli avatar Mar 10 '25 17:03 NullVoxPopuli

Still working on this. One Addon at a time! down to ember-a11y-testing (awaiting release) and embroider/router just opened https://github.com/embroider-build/embroider/pull/2443. Then I'll be fully at 4.x, which might help. Or might make it easier to see where there are issues that aren't packaging related.

jrjohnson avatar Apr 07 '25 18:04 jrjohnson

Can now confirm that getting everything in my entire addon chain to 4.x fixes the test failures.

jrjohnson avatar Apr 24 '25 04:04 jrjohnson

Can now confirm that getting everything in my entire addon chain to 4.x fixes the test failures.

Also confirm

amk221 avatar Jul 07 '25 12:07 amk221

Maybe I spoke to soon, still seeing some random ones. Anyone?

amk221 avatar Sep 03 '25 20:09 amk221