opencollective Unreliable Cypress tests

Starting an issue to document the E2E tests that are often failing.

Feb 28 '23 07:02 Betree

Resolved by https://github.com/opencollective/opencollective-frontend/pull/8671

1) Contribute Flow: Stripe Payment Element
   ACH
     Guest:
 AssertionError: Timed out retrying after 15000ms: Expected to find element: `input[type="email"]`, but never found it.
  at Context.eval (webpack:///./test/cypress/integration/13-contributeFlow-stripePaymentElement.test.js:195:9)

Feb 28 '23 07:02 Betree

Another one,

1) Contribute Flow: Stripe Payment Element
       ACH
         Guest:
     AssertionError: Timed out retrying after 15000ms: Expected to find content: 'Processing' but never did.
      at Context.eval (webpack:///./test/cypress/integration/13-contributeFlow-stripePaymentElement.test.js:209:9)

Mar 01 '23 01:03 SudharakaP

@Betree Hello, I want to work and contribute on this issue, please assign me this issue. Thanks.

Mar 01 '23 10:03 Mayuresh0072

Resolved with https://github.com/opencollective/opencollective-frontend/pull/8686 and https://github.com/opencollective/opencollective-api/pull/8620

The E2E logs seems to be filled with,

 warn: Stripe: Webhooks: Received an unsupported event type: customer.created
2023-03-01 13:25:13  <--  [200] POST http://localhost:3060/webhooks/stripe [evt_1MgpefDjPFcHOcTmeFXlzHNU]
2023-03-01 13:25:14   --> connect payment_method.attached [evt_1MgpegDjPFcHOcTmQvZOSy6P]
2023-03-01 13:25:14  <--  [200] POST http://localhost:3060/webhooks/stripe [evt_1MgpegDjPFcHOcTmQvZOSy6P]
2023-03-01 13:25:14   --> payment_method.attached [evt_1MgpegDjPFcHOcTmQvZOSy6P]
2023-03-01 13:25:15   --> setup_intent.created [evt_1MgpehDjPFcHOcTmVPnj1sAO]
2023-03-01 13:25:15   --> connect setup_intent.created [evt_1MgpehDjPFcHOcTmVPnj1sAO]
warn: Stripe: Webhooks: Received an unsupported event type: setup_intent.created
2023-03-01 13:25:15  <--  [200] POST http://localhost:3060/webhooks/stripe [evt_1MgpehDjPFcHOcTmVPnj1sAO]
2023-03-01 13:25:15   --> setup_intent.succeeded [evt_1MgpehDjPFcHOcTmirVhNObo]
2023-03-01 13:25:15   --> connect setup_intent.succeeded [evt_1MgpehDjPFcHOcTmirVhNObo]
warn: Stripe: Webhooks: Received an unsupported event type: setup_intent.succeeded
2023-03-01 13:25:15  <--  [200] POST http://localhost:3060/webhooks/stripe [evt_1MgpehDjPFcHOcTmirVhNObo]
2023-03-01 13:25:17   --> connect customer.created [evt_1MgpejLzdXg9xKNSTTCK1N2p]
warn: Stripe: Webhooks: Received an unsupported event type: customer.created
2023-03-01 13:25:17  <--  [200] POST http://localhost:3060/webhooks/stripe [evt_1MgpejLzdXg9xKNSTTCK1N2p]
2023-03-01 13:25:17   --> customer.created [evt_1MgpejDjPFcHOcTmjYt0ovLM]
2023-03-01 13:25:17   --> connect customer.created [evt_1MgpejDjPFcHOcTmjYt0ovLM]
warn: Stripe: Webhooks: Received an unsupported event type: customer.created

Mar 01 '23 13:03 SudharakaP

Just to note that unfortunately stripe e2e tests at 1.js are dependent on Stripe processing the tests payments during the run. We have some waits on the tests to make that more likely, currently 10 seconds.

Mar 01 '23 14:03 hdiniz

  1) event.create.test.js
       create an event:
     AssertionError: Timed out retrying after 15000ms: Expected to find element: `[data-cy=name]`, but never found it.
      at Context.eval (webpack:///./test/cypress/integration/23-event.create.test.js:43:7)

Mar 07 '23 08:03 Betree

1) host dashboard
       unhost collectives
         cannot unhost collective with balance:
     AssertionError: Timed out retrying after 15000ms: Expected not to find content: 'Finish' within the selector: 'button' but continuously found it.
      at Context.eval (webpack:///./test/cypress/integration/29-host.admin.test.js:95:38)

Mar 07 '23 10:03 hdiniz

   1) host dashboard
       expenses tab
         Process expense:
     AssertionError: Timed out retrying after 15000ms: Expected to find element: `[data-cy="expense-container-2387"]`, but never found it.
      at Context.eval (webpack:///./test/cypress/support/commands.js:454:14)

Mar 07 '23 13:03 Betree

Duplicate of https://github.com/opencollective/opencollective/issues/6484#issuecomment-1457766677

  1) event.create.test.js
       create an event:
     AssertionError: Timed out retrying after 15000ms: Expected to find element: `[data-cy=name]`, but never found it.
      at Context.eval (webpack:///./test/cypress/integration/23-event.create.test.js:43:7)

Mar 09 '23 10:03 hdiniz

1) Collective page
       Updates section
         Shows latest updates:
     AssertionError: Timed out retrying after 15000ms: Expected to find element: `[data-cy=updatesList]`, but never found it.
      at Context.eval (webpack:///./test/cypress/integration/04-collective.test.js:146:38)

EDIT: fixed by https://github.com/opencollective/opencollective-frontend/pull/8659/ - @gustavlrsn

Mar 10 '23 04:03 hdiniz

27-expenses.test.js:

  1) New expense flow
       new expense when logged in
         submit on behalf
           can invite a third-party organization to submit an expense:
     AssertionError: Timed out retrying after 15000ms: Expected to find element: `[data-cy="select-expense-payee"]`, but never found it.

Mar 16 '23 14:03 gustavlrsn

37-accept-financial-contributions.test.js:

  1) Accept financial contributions flow
       Apply to fiscal host
         Successfully applies to a host:
     AssertionError: Timed out retrying after 15000ms: Expected to find content: 'BrusselsTogether is a platform for the new generation of associations transparent by design.' but never did.

Mar 20 '23 16:03 gustavlrsn

Running:  13-contributeFlow-stripePaymentElement.test.js                                 (4 of 11)


  Contribute Flow: Stripe Payment Element
    Card
      1) Guest
      2) User
    ACH
      ✓ Guest (26141ms)
      ✓ User (38539ms)
    SEPA
      ✓ Guest (32178ms)
      ✓ User (33440ms)
    Bancontact
      ✓ Guest (33641ms)
      ✓ User (29602ms)
    Redirects
      3) Redirects to trusted url
      4) Redirects to untrusted url


  6 passing (5m)
  4 failing

  1) Contribute Flow: Stripe Payment Element
       Card
         Guest:
     AssertionError: Timed out retrying after 15000ms: Expected to find element: `[data-cy="order-success"]`, but never found it.
      at Context.eval (webpack:///./test/cypress/support/commands.js:454:14)

  2) Contribute Flow: Stripe Payment Element
       Card
         User:
     AssertionError: Timed out retrying after 15000ms: Expected to find element: `[data-cy="order-success"]`, but never found it.
      at Context.eval (webpack:///./test/cypress/support/commands.js:454:14)

  3) Contribute Flow: Stripe Payment Element
       Redirects
         Redirects to trusted url:

      Timed out retrying after 15000ms
      + expected - actual

      -'http://localhost:3000'
      +'https://opencollective.com/'
      
      at Context.eval (webpack:///./test/cypress/integration/13-contributeFlow-stripePaymentElement.test.js:372:28)

  4) Contribute Flow: Stripe Payment Element
       Redirects
         Redirects to untrusted url:

      Timed out retrying after 15000ms
      + expected - actual

      -'/6d6801bc/donate/payment'
      +'/external-redirect'
      
      at Context.eval (webpack:///./test/cypress/integration/13-contributeFlow-stripePaymentElement.test.js:387:30)

Mar 27 '23 12:03 Betree

13-contributeFlow-stripePaymentElement.test.js

Had a look at these issues a bit.

The Stripe credit card orders are creating a payment method with {type: STRIPE, method: PAYMENT_INTENT}, which create an order with status NEW:

@kewitz @hdiniz Do you have some insight into the expected behavior? The e2e tests are looking for a PAID status. Is the payment method PAYMENT_INTENT correct and should it be status NEW?

The displayed payment method is wrong. We're currently using the deprecated field paymentMethod.providerType that is using the function getLegacyPaymentMethod which does not support method PAYMENT_INTENT, so it is falling back to bank transfer.

If adding type and service to paymentMethod instead of deprecated providerType in the Orders query, we instead show the raw enum PAYMENT_INTENT as the payment method, so maybe something better needs to be added here, if this method correct and something we should display.

Apr 03 '23 10:04 gustavlrsn

@gustavlrsn The payment method type PAYMENT_INTENT and status NEW is correct initially. That is an intermediate state because we dont know the type or status of payment until the Stripe webhook succeeds. But the display type on the UI is indeed wrong.

The status and type are expected to change async through a stripe webhook. This test is flaky as Stripe takes longer than expected to process the payment.

I have tried to refresh the page suring the test for some time while waiting for the status change here: https://github.com/opencollective/opencollective-frontend/pull/8775

Apr 03 '23 13:04 hdiniz

The ACH test is repeatedly failing for me

May 02 '23 08:05 Betree

https://github.com/opencollective/opencollective-api/actions/runs/5321898072/jobs/9637589639?pr=9102#step:31:292

[Network error]: FetchError: request to http://localhost:3060/graphql/v1?api_key=dvl-1510egmf4a23d80342403fb599qd failed, reason: socket hang up
error: GraphQL error: Variable "$slug" of required type "String!" was not provided.
error: {"locations":[{"line":1,"column":22}]}
[GraphQL error]: Message: Variable "$slug" of required type "String!" was not provided., Location: [object Object], Path: undefined
ApolloError: Variable "$slug" of required type "String!" was not provided.
    at new ApolloError (/home/runner/work/opencollective-api/opencollective-api/opencollective-frontend/node_modules/@apollo/client/errors/errors.cjs:31:28)
    at /home/runner/work/opencollective-api/opencollective-api/opencollective-frontend/node_modules/@apollo/client/core/core.cjs:1843:47
    at both (/home/runner/work/opencollective-api/opencollective-api/opencollective-frontend/node_modules/@apollo/client/utilities/utilities.cjs:1072:53)
    at /home/runner/work/opencollective-api/opencollective-api/opencollective-frontend/node_modules/@apollo/client/utilities/utilities.cjs:10***:72
    at new Promise (<anonymous>)
    at Object.then (/home/runner/work/opencollective-api/opencollective-api/opencollective-frontend/node_modules/@apollo/client/utilities/utilities.cjs:10***:24)
    at Object.next (/home/runner/work/opencollective-api/opencollective-api/opencollective-frontend/node_modules/@apollo/client/utilities/utilities.cjs:1073:49)
    at notifySubscription (/home/runner/work/opencollective-api/opencollective-api/opencollective-frontend/node_modules/zen-observable/lib/Observable.js:135:18)
    at onNotify (/home/runner/work/opencollective-api/opencollective-api/opencollective-frontend/node_modules/zen-observable/lib/Observable.js:179:3)
    at SubscriptionObserver.next (/home/runner/work/opencollective-api/opencollective-api/opencollective-frontend/node_modules/zen-observable/lib/Observable.js:235:7) {
  graphQLErrors: [
    {
      message: 'Variable "$slug" of required type "String!" was not provided.',
      locations: [Array],
      extensions: [Object]
    }
  ],
  protocolErrors: [],
  clientErrors: [],
  networkError: null,
  extraInfo: undefined
}
error: GraphQL error: Variable "$slug" of required type "String!" was not provided.
error: {"locations":[{"line":1,"column":22}]}
    1) as a new collective


  0 passing (18s)
  1 failing

  1) apply to host
       as a new collective:
     AssertionError: Timed out retrying after 15000ms: Expected to find content: 'We are fiscally hosting 2 Collectives' but never did.
      at Context.eval (webpack:///./test/cypress/integration/25-host.apply.test.js:11:9)

Jun 20 '23 11:06 hdiniz

Can order as new user almost constantly failing for me

Jul 19 '23 08:07 Betree

1) New expense flow
       new expense when logged in
         submit on behalf
           can invite a third-party organization to submit an expense:
     AssertionError: Timed out retrying after 15000ms: Expected to find content: 'Angola' within the selector: '[data-cy="select-option"]' but never did.
      at Context.eval (webpack:///./test/cypress/integration/27-expenses.test.js:393:11)

Sep 06 '23 11:09 hdiniz

Contribute Flow: Stripe Payment Element > Bancontact consistently failing since a few days.

Oct 16 '23 14:10 Betree

Another one failing regularly:

1) host dashboard
     Pending `Contributions
       Create new pending contribution, edit it and mark it as paid:
   AssertionError: Timed out retrying after 15000ms: Expected to find content: 'Expired' within the selector: '[data-cy=order-status-msg]' but never did.
    at Context.eval (webpack:///./test/cypress/integration/29-host.admin.test.js:160:9)

Oct 17 '23 11:10 Betree

Recurring failures with this one

  1) Expense flow
       new expense with taxes
         can submit with VAT:
     AssertionError: Timed out retrying after 15000ms: Expected to find element: `[data-cy="expense-invoiced-amount"]`, but never found it.
      at Context.eval (webpack://opencollective-frontend/./test/cypress/integration/27-expenses.test.js:521:48)

Dec 01 '23 09:12 Betree

Recurring failures with this one

   1) host dashboard
       Pending `Contributions
         Create new pending contribution, edit it and mark it as paid:
     AssertionError: Timed out retrying after 15000ms: Expected to find content: 'Expired' within the selector: '[data-cy=order-status-msg]' but never did.
      at Context.eval (webpack://opencollective-frontend/./test/cypress/integration/29-host.admin.test.js:161:9)

Dec 29 '23 11:12 Betree

We're randomly getting failures on all tests with this error:

[Network error]: FetchError: request to http://localhost:3060/graphql/v2?api_key=dvl-1510egmf4a23d80342403fb599qd failed, reason: socket hang up

Jan 15 '24 08:01 Betree

Recurring error on this one

   Running:  07-password.test.js                                                           (13 of 16)

  passwords
  1) passwords
       can be used in combination with 2FA:
     AssertionError: Timed out retrying after 15000ms: Expected to find content: '[email protected]' within the selector: '[data-cy="user-menu"]' but never did.
      at Context.eval (webpack://opencollective-frontend/./test/cypress/support/commands.js:432:7)

Jun 10 '24 08:06 Betree

07-password.test.js was fixed by https://github.com/opencollective/opencollective-frontend/pull/10578.

Jul 29 '24 06:07 kewitz

This one is still happening.

1) passwords
       can be used in combination with 2FA:
     AssertionError: Timed out retrying after 15000ms: Expected to find content: '[email protected]' within the selector: '[data-cy="user-menu"]' but never did.
      at Context.eval (webpack://opencollective-frontend/./test/cypress/support/commands.js:432:7)

Sep 10 '24 17:09 hdiniz

This one is still happening.

1) passwords
       can be used in combination with 2FA:
     AssertionError: Timed out retrying after 15000ms: Expected to find content: '[email protected]' within the selector: '[data-cy="user-menu"]' but never did.
      at Context.eval (webpack://opencollective-frontend/./test/cypress/support/commands.js:432:7)

Should hopefully be fixed by https://github.com/opencollective/opencollective-frontend/pull/10770

Oct 25 '24 08:10 Betree

  1) Expense flow
       new expense when logged in
         can play with the exchange rate:
     AssertionError: Timed out retrying after 15000ms: Expected to find element: `[data-cy="mismatch-warning-tooltip"]`, but never found it.
      at Context.eval (webpack://opencollective-frontend/./test/cypress/integration/27-expenses.test.js:316:53)

Oct 25 '24 08:10 Betree

 Running:  22-collective.edit.test.js                                                     (3 of 11)


  edit collective
    1) edit members
      cy:request ✔  POST /api/users/signin
                    Status: 200
                    Request body: {
                      "user": {
                        "email": "[email protected]",
                        "newsletterOptIn": false
                      },
                      "redirect": "/dashboard/52234d4b-665/info",
                      "createProfile": true
                    }
                    Response body: {
                      "success": true,
                      "redirect": "http://localhost:3000/signin/***"
                    }
      cy:command ✔  visit	http://localhost:3000/signin/***
      cy:command ✔  get	[data-cy="menu-item-team"]
        cy:fetch ➟  POST http://localhost:3000/api/users/exchange-login-token
                    Status: 200
        cy:fetch ➟  GET http://localhost:3000/_next/data/fwMGzDsUXnBd8fLU3038o/index.json
                    Status: 200
        cy:fetch ➟  POST http://localhost:3000/api/graphql/v1
                    Status: 200
        cy:fetch ➟  POST http://localhost:3000/api/graphql/v2
                    Status: 200
       cons:warn ❖  Google Maps JavaScript API has been loaded directly without loading=async. This can result in suboptimal performance. For best-practice loading patterns please see https://goo.gle/js-api-loading
      cy:command ✔  new url	http://localhost:3000/dashboard/52234d4b-665/info
          cy:xhr ➟  GET https://maps.googleapis.com/maps/api/mapsjs/gen_204?csp_test=true
                    Status: 200
        cy:fetch ➟  POST http://localhost:3000/api/graphql/v2
                    Status: 200
      cy:command ✔  click	
        cy:fetch ➟  POST http://localhost:3000/api/graphql/v2
                    Status: 200
        cy:fetch ➟  POST http://localhost:3000/api/graphql/v1
                    Status: 200
      cy:command ✔  new url	http://localhost:3000/dashboard/52234d4b-665/team
      cy:command ✔  wait	200
        cy:fetch ➟  POST http://localhost:3000/api/graphql/v2
                    Status: 200
          cy:xhr ➟  POST https://maps.googleapis.com/maps_api_js_slo/log?hasfast=true
                    Status: 200
        cy:fetch ➟  POST http://localhost:3000/api/graphql/v2
                    Status: 200
        cy:fetch ➟  POST http://localhost:3000/api/graphql/v2
                    Status: 200
      cy:command ✔  get	[data-cy="invite-member-btn"]
      cy:command ✔  click	
      cons:error ✘  `DialogContent` requires a `DialogTitle` for the component to be accessible for screen reader users.
                    
                    If you want to hide the `DialogTitle`, you can wrap it with our VisuallyHidden component.
                    
                    For more information, see https://radix-ui.com/primitives/docs/components/dialog
       cons:warn ❖  Warning: Missing `Description` or `aria-describedby={undefined}` for {DialogContent}.
      cy:command ✔  wait	200
      cy:command ✔  get	[data-cy="member-collective-picker"]
      cy:command ✔  click	
      cy:command ✔  get	[data-cy="collective-type-picker-USER"]
      cy:command ✔  click	
          cy:xhr ➟  POST https://maps.googleapis.com/maps_api_js_slo/log?hasfast=true
                    Status: 200
      cy:command ✔  get	[data-cy="create-collective-mini-form"]
      cy:command ✔  wrap	<form.flex.h-full.flex-col.overflow-y-hidden>
      cy:command ✔  find	input[name="email"]
      cy:command ✔  type	[email protected]
      cy:command ✔  wrap	<form.flex.h-full.flex-col.overflow-y-hidden>
      cy:command ✔  find	input[name="name"]
      cy:command ✔  type	AmazingNewUser
      cy:command ✔  wrap	<form.flex.h-full.flex-col.overflow-y-hidden>
      cy:command ✔  find	button[type="submit"]
      cy:command ✔  click	
        cy:fetch ➟  POST http://localhost:3000/api/graphql/v1
                    Status: 200
      cy:command ✔  wait	200
      cy:command ✔  get	[data-cy="create-collective-mini-form"]
      cy:command ✔  assert	expected **[data-cy="create-collective-mini-form"]** not to exist in the DOM
                    Actual: 	"[data-cy=\"create-collective-mini-form\"]"
                    Expected: 	"[data-cy=\"create-collective-mini-form\"]"
      cy:command ✔  get	[data-cy="confirmation-modal-continue"]
      cy:command ✔  click	
        cy:fetch ➟  POST http://localhost:3000/api/graphql/v2
                    Status: 200
      cy:command ✔  get	[data-cy="member-1"] [data-cy="member-pending-tag"]
      cy:command ✔  assert	expected **<div.StyledTag__StyledTagBase-sc-7e1f7747-0.evhhGp>** to exist in the DOM
                    Actual: 	"<div.StyledTag__StyledTagBase-sc-7e1f7747-0.evhhGp>"
                    Expected: 	"<div.StyledTag__StyledTagBase-sc-7e1f7747-0.evhhGp>"
        cy:fetch ➟  POST http://localhost:3000/api/graphql/v2
                    Status: 200
      cy:request ✔  GET http://localhost:1080/api/v1/search
                    Status: 200
                    Request body: <EMPTY>
                    Response body: {
                      "total": 1,
                      "unread": 1,
                      "count": 0,
                      "messages_count": 0,
                      "start": 0,
                      "tags": [
                        "admin-at-opencollective.com",
                        "testuser"
                      ],
                      "messages": []
                    }
      cy:command ✔  assert	expected **{ Object (total, unread, ...) }** to have property **messages_count**
      cy:command ✘  assert	expected **0** to be above **0**


    ✓ edit info (4599ms)
    ✓ edit tiers (4797ms)
    ✓ enables VAT (5080ms)

  edit user collective
    ✓ adds two-factor authentication (2526ms)


  4 passing (24s)
  1 failing

  1) edit collective
       edit members:

      AssertionError: expected 0 to be above 0
      + expected - actual


      at captureUserInvocationStack (http://localhost:3000/__cypress/runner/cypress_runner.js:138199:94)
      at Proxy.<anonymous> (http://localhost:3000/__cypress/runner/cypress_runner.js:138227:9)
      at Proxy.assertAbove (http://localhost:3000/__cypress/runner/cypress_runner.js:79986:12)
      at Proxy.methodWrapper (http://localhost:3000/__cypress/runner/cypress_runner.js:7***70:25)
      at applyChainer (http://localhost:3000/__cypress/runner/cypress_runner.js:117829:26)
      at <unknown> (http://localhost:3000/__cypress/runner/cypress_runner.js:117882:16)
      at arrayReduce (http://localhost:3000/__cypress/runner/cypress_runner.js:18675:21)
      at lodash.reduce (http://localhost:3000/__cypress/runner/cypress_runner.js:27727:14)
      at applyChainers (http://localhost:3000/__cypress/runner/cypress_runner.js:117860:68)
      at tryCatcher (http://localhost:3000/__cypress/runner/cypress_runner.js:1777:23)

Feb 27 '25 15:02 Betree