☂️ Help us with E2E Tests

[cipolleschi]

[!NOTE] This React Native umbrella issue is eligible for Hacktoberfest. Pull requests that contribute to this issue will be labelled as 'hacktoberfest-accepted' and will contribute towards your contest goal.

Description

Hi all, This is an umbrella issue to collect a set of tasks with the goal of improving our E2E testing, focused on improving the quality of React Native Release.

Having better and faster releases with less human intervention, is beneficial for the community:

• Lower chances of human error
• Lower chances of breaking behaviors we can tests

Keep the quality bar for React native high

One of the steps of releasing React Native today is manual testing of some framework functionalities by the release crew.

Manual testing is error prone and it can happen that we miss something while testing a release.

E2E Testing can solve many of those problems:

• It is automatic, so it will systematically go through all the test cases.
• Can tests various configurations in parallel
• Can happen right after the artifacts are ready, reducing the delay between CI and testing

By contributing to this umbrella issue, you can help the community to keep the bar high and ensure that future releases would contain less bugs!

The Current Solution

Few weeks ago, we landed some new Jobs in CI to run tests using Maestro. Maestro is a tool that lets us write E2E tests for React Native using YAML as a definition language. Tests read very natural and are easy to read.

The jobs are already running a very basic test for all the configurations defined above. You can have a look at those test definitions for RNTester

These tests are currently running on the main branch and on stable branches of React Native.

The Tasks

We are presenting below a list of tasks that any of you can pick up and start working on.

Please comment below if you're willing to take the stance on any of those tasks and wait for a confirmation before starting working on it. Some basic rules:

• if you claim a task, but don't submit a PR within 2 weeks, we're going to free up that task for someone else.
• Do not claim more than 2 tasks at the same time

Some tasks will only require you to write a new test using YAML. Some others might require you to touch the JS code of RNTester or of the Template to add some testID or to slightly manipulate the View hierarchy of those apps.

You can have a look at this PR as an example.

How to Test

Before diving into the task list, we want to set you up for success! The best way to do it is to show you how you can test your changes.

Requirements:

• Your environment should be set up to be able to launch an iOS app and an android app on the simulator.
• Run brew install jq to install the jq utility, if it is not installed in your system.
• Install Maestro locally:
  • curl -Ls "https://get.maestro.mobile.dev" | bash
  • brew tap facebook/fb
  • brew install facebook/fb/idb-companion

Test Locally

• Build the app for Release, from the react-native root folder:
  • iOS:

    yarn install
    cd packages/rn-tester
    yarn e2e-build-ios
    

  • Android:

    yarn install
    cd packages/rn-tester
    yarn e2e-build-android
    

• Start maestro:
  • iOS:

    d packages/rn-tester
    yarn e2e-test-ios
    

  • Android:

    cd packages/rn-tester
    yarn e2e-test-android
    

Test In CI

To test in CI, just add a comment on your PR with the text:

/test-e2e

This comment triggers a workflow that starts the Test All workflow with the E2E tests included (they are excluded by default).

Task List

This is the list of tasks. Feel free to add a comment with the task you’d like to work on!

• [x] [ME2E0001] Rename the file packages/rn-tester/.maestro/start.yml from start.yml to modal.yml. Then, add to the tests the lines to tap on the Show Modal button and expect to find a button Close - Assigned to @ahmedbilal008
• [x] [ME2E0002] Depends on ME2E0001. Open the test packages/rn-tester/.maestro/modal.yml and add lines to tap on Close and expect to see the button Show Modal - Assigned to @ahmedbilal008
• [x] [ME2E0003] Update the RNTester example in MyNativeView so that opacity is not random anymore but goes from 1 to 0 in steps of -0.20 (so 1, 0.8, 0.6, 0.4, 0.2, 0). After 0, if the user clicks on Set Opacity, it goes back to 1. - Assigned to @sarthak-d11
• [x] [ME2E0004] Depends on ME2E0003. Modify the test new-arch-example.yml and add steps to Click on Set Opacity, and verify that the HSBA values change accordingly - Assigned to @poonamjain96
• [x] [ME2E0005] Modify the test new-arch-example.yml and add steps to Click on Console.log Measure. Verify that the Interop Layer Measurements changes appropriately.
• [x] [ME2E0006] Update the RNTester example in MyNativeView and add a counter for the LegacyEventStyle fired event. The counter can be rendered right below the button and should increase by 1 every time the button is clicked. - Assigned to @tarunrajput
• [x] [ME2E0007] Depends on ME2E006. Modify the test new-arch-example.yml and add steps to Click on Fire Legacy Style Event. Verify that the counter increases by 1 after the method is called. - Assigned to @tarunrajput
• [ ] [ME2E0008] Add a new file in the packages/rn-tester/.maestro folder called flatlist.yml. Copy over the setup from the new-arch-examples.yml to set up a test that starts the app, scrolls until Flatlist is visible and Clicks on FlatList - Assigned to @l2hyunwoo
• [ ] [ME2E0009] Depends on ME2E0008. Update the packages/rn-tester/.maestro/flatlist.yml to click on the button Basic and scroll down until item 600 is visible. - Assigned to @l2hyunwoo
• [ ] [ME2E0010] Depends on ME2E0008. Update the packages/rn-tester/.maestro/flatlist.yml to click on the switch Empty and verify that the text ”The list is empty :o” is visible - Assigned to @kunalchavhan
• [x] [ME2E0011] Add a new file in the packages/rn-tester/.maestro folder called button.yml. Copy over the setup from the new-arch-examples.yml to set up a test that starts the app, scrolls until Button is visible and Clicks on Button - Assigned to @shubhamguptadream11
• [x] [ME2E0012] Depends on ME2E0012. Update the button.yml to tap on the Submit Application button. Verify that the dialog appears. Click on Ok to dismiss it. - Assigned to @shubhamguptadream11
• [x] [ME2E0013] Add a new file in the packages/rn-tester/.maestro folder called pressable.yml. Copy over the setup from the new-arch-examples.yml to set up a test that starts the app, scrolls until Pressable is visible and Clicks on Pressable - Assigned to @yasir6jan
• [x] [ME2E0014] Depends on ME2E0014. Update the pressable.yml to tap on the first Press Me button and verify that the text changes in the box below. You might need to update the PressableExample.js file to discover which identifier you need to use for the first Press Me button. - Assigned to @yasir6jan
• [x] [ME2E0015] Depends on ME2E0014. Update the pressable.yml to scroll down until the Pressable feedback events box is visible. Tap on the Press me button. Verify that the texts pressIn, press and pressOut are visible. - Assigned to @deepanshushuklad11
• [x] [ME2E0016] Depends on ME2E0014. Update the pressable.yml to scroll down until the Pressable feedback events box is visible. Long press on the Press me button. Verify that the texts pressIn, longPress and pressOut are visible. - Assigned to @deepanshushuklad11
• [x] [ME2E0017] Add a new file in the packages/rn-tester/.maestro folder called text.yml. Copy over the setup from the new-arch-examples.yml to set up a test that starts the app, scrolls until Text is visible and taps on Text. - Assigned to @muskan27797
• [x] [ME2E0018] Depends on ME2E0019. Update the text.yml file to scroll down until the box titled Background Color and Border Width box is completely visible. Then assert that all the 5 texts are visible. - Assigned to @muskan27797
• [x] [ME2E0019] Add a new file in the packages/rn-tester/.maestro folder called image.yml. Copy over the setup from the new-arch-examples.yml to set up a test that starts the app, scrolls until Image is visible and taps on Image. Assert that the Plain Network Image with source prop is visible. - Assigned to @sarthak-d11

[ahmedbilal008]

I would like to claim the following tasks: -[ME2E0001] Rename the file packages/rn-tester/.maestro/start.yml from start.yml to modal.yml. Then, add to the tests the lines to tap on the Show Modal button and expect to find a button Close -[ME2E0002] Depends on ME2E0001. Open the test packages/rn-tester/.maestro/modal.yml and add lines to tap on Close and expect to see the button Show Modal

[l2hyunwoo]

I want to take following tasks

[ME2E0008] Add a new file in the packages/rn-tester/.maestro folder called flatlist.yml. Copy over the setup from the new-arch-examples.yml to set up a test that starts the app, scrolls until Flatlist is visible and Clicks on FlatList

[ME2E0009] Depends on ME2E0008. Update the packages/rn-tester/.maestro/flatlist.yml to click on the button Basic and scroll down until item 600 is visible.

[shubhamguptadream11]

I want to take following tasks:

[ME2E0011] Add a new file in the packages/rn-tester/.maestro folder called button.yml. Copy over the setup from the new-arch-examples.yml to set up a test that starts the app, scrolls until Button is visible and Clicks on Button [ME2E0012] Depends on ME2E0012. Update the button.yml to tap on the Submit Application button. Verify that the dialog appears. Click on Ok to dismiss it.

[muskan27797]

I would like to claim the following tasks:

[ME2E0017] Add a new file in the packages/rn-tester/.maestro folder called text.yml. Copy over the setup from the new-arch-examples.yml to set up a test that starts the app, scrolls until Text is visible and taps on Text. [ME2E0018] Depends on ME2E0019. Update the text.yml file to scroll down until the box titled Background Color and Border Width box is completely visible. Then assert that all the 5 texts are visible.

[sarthak-d11]

I would like to claim the following tasks:

[ME2E0003] Update the RNTester example in MyNativeView so that opacity is not random anymore but goes from 1 to 0 in steps of -0.20 (so 1, 0.8, 0.6, 0.4, 0.2, 0). After 0, if the user clicks on Set Opacity, it goes back to 1.

[ME2E0019] Add a new file in the packages/rn-tester/.maestro folder called image.yml. Copy over the setup from the new-arch-examples.yml to set up a test that starts the app, scrolls until Image is visible and taps on Image. Assert that the Plain Network Image with source prop is visible..

[tarunrajput]

I'd like to work on: [ME2E0005] and [ME2E0006]

[yasir6jan]

I would like to work on following tasks: [ME2E0013], [ME2E0014]

[deepanshushuklad11]

I would like to pick [ME2E0013], [ME2E0014] ,[ME2E0015] ,[ME2E0016] Since these tasks are related.

[tarunrajput]

I would like to pick [ME2E0013], [ME2E0014] ,[ME2E0015] ,[ME2E0016] Since these tasks are related.

@deepanshushuklad11, pls pick only two tasks at a time

[kunalchavhan]

I would like to contribute to

• ME2E0008 - Add a new file in the packages/rn-tester/.maestro folder called flatlist.yml. Copy over the setup from the new-arch-examples.yml to set up a test that starts the app, scrolls until Flatlist is visible and Clicks on FlatList
• ME2E0010 - Update the packages/rn-tester/.maestro/flatlist.yml to click on the switch Empty and verify that the text ”The list is empty :o” is visible

[cipolleschi]

so happy that so many of you already took some tasks! @ahmedbilal008 @l2hyunwoo @shubhamguptadream11 @muskan27797 @sarthak-d11 @tarunrajput @yasir6jan

@deepanshushuklad11 I assigned you ME2E0015 and ME2E0016 as the other two were already claimed and we have a rule to claim 2 tasks at the same time. @kunalchavhan I could only assign you ME2E0010 as the other task was taken already!

Feel free to tag me once the PR are ready to be reviewed

[poonamjain96]

I want to work on following tasks: [ME2E0004], [ME2E0007]

[cipolleschi]

@poonamjain96 there has been an issue and @tarunrajput opened https://github.com/facebook/react-native/pull/46784 to address ME2E0006 and ME2E0007, instead of 0005 and 0006.

Let me know if you want to work on another task.

[poonamjain96]

No issues @cipolleschi So now I am picking ME2E0004 and ME2E0005

[MobinAkhter]

Hi, @cipolleschi. I’d like to take up task [ME2E0005] to modify the new-arch-example.yml. Please assign it to me!

[poonamjain96]

@cipolleschi

[ME2E0004] Depends on ME2E0003. Modify the test new-arch-example.yml and add steps to click on "Set Opacity" and verify that the HSBA values change accordingly.

Problem: When I click on the SetOpacity function, the opacity changes visually, but the HSBA values are not updating. As a result, I can't assert whether the HSBA values are changing in the test cases.

Solution: Instead of validating the HSBA values directly, we could add another text display next to HSBA: h: 0, s: 0, b: 0, a: 0 that shows Opacity: <value>. This would allow us to verify changes in opacity more easily.

Let me know your thoughts on this.

[dglalperen]

Issue Summary:

While working on the following items: [ME2E0008], [ME2E0009], [ME2E0010], I encountered an issue where Maestro is unable to identify or locate the text "Item 600" or "600" within the FlatList. Despite several attempts, I was unable to resolve the problem, and I am hoping someone else can successfully test or reproduce this behavior.

Reproduction Steps

Below is the current flatlist.yml Maestro test configuration I was using:

appId: ${APP_ID}  # iOS: com.meta.RNTester.localDevelopment | Android: com.facebook.react.uiapp
---
- launchApp
- assertVisible: "Components"
- scrollUntilVisible:
    element:
      id: "FlatList"
    direction: DOWN
    speed: 40
- tapOn:
    id: "FlatList"
- tapOn:
    id: "Basic"
- scrollUntilVisible:
    element:
      text: "Item 600"
    timeout: 500000
    direction: DOWN
    speed: 30
- tapOn:
    id: "Empty"
- assertVisible:
    id: "The list is empty :o"

Observed Behavior

Maestro is unable to find the text "Item 600" or "600" during the test. The scroll seems to be working as expected for other items, but it doesn't detect or tap on "Item 600" specifically.

Expected Behavior

Maestro should be able to scroll to and detect "Item 600" as visible in the FlatList, similar to other items in the list.

Request for Assistance

Can anyone confirm if they are able to replicate this issue? Any insights on why this might be happening or workarounds would be greatly appreciated.

[shubhamguptadream11]

@dglalperen Did you checked this PR? It is ready for review so the things probably working here.

[dglalperen]

@shubhamguptadream11 checked it out now, works ! Thank you :)

[VishalGawade1]

Is there any pending task I can contribute to? I don't see any closure for ME2E0005. can I take it?

[shubhamguptadream11]

@VishalGawade1 ME2E0005 is already solved and merged by @poonamjain96

[VishalGawade1]

@VishalGawade1 ME2E0005 is already solved and merged by @poonamjain96

Oh my bad, I missed it. Is there any other task I can work on?

[cipolleschi]

@VishalGawade1 thanks for your interest! We are not planning to add other test cases for now. We want to first verify that this initial set of tests works well for us. If that's the case, you can expect more tasks to be open in the next future! 😄

[cipolleschi]

@dglalperen sorry, I have not seen your message. I just landed the test for flatlist. It was assigned to @l2hyunwoo. Yes, you are right that it was not visible right away: we had to pass the testID to the <Text> component to make it visible.

[cipolleschi]

@kunalchavhan if you are still interested, now you can have a look at the Empty List test case.

[KrastanD]

I'd like to throw my hat in the ring if anything frees up

[kunalchavhan]

yes @cipolleschi, will send PR.

[cipolleschi]

Thank you all for participating in this effort! :D