Improve e2e times and flakiness by using the search

[mateoguzmana]

Summary:

I'm writing some tests for some APIs/components, and I see some opportunity for improvement in the way how we can write e2e cases for the RNTester.

This diff improves two things:

1. e2e execution times: right now as we are using the scrollUntilVisible functionality, it takes quite some time for some cases, as some of the items are not very up in the lists and getting to them it's not always very fast.
2. Flakiness: I ran the tests multiple times locally, and in multiple occasions, the scrollUntilVisible did not find anything as the scroll was too fast so the test ended up failing.

Instead of using scrollUntilVisible for all cases, we can simply use the search bar which we have in both Components and APIs tabs. This runs faster and we can also share the search flow across multiple test cases, so writing the tests becomes a bit simpler as well.

Initially, I did this for all cases but not all cases benefit from this change as some of them are easier to find than others – the improvement was most notable in iOS (keyboard visibility seems to make a big difference), but I still think it is a good baseline to use the search as if more test cases are added, likely, many of them are not going to be so easy to find.

Changelog:

[INTERNAL] - Improve e2e times and flakiness by using the search

Test Plan:

These are the differences in the time it takes to pass the tests locally.

iOS:

Before	After

Android (no significant difference):

Before	After

[cipolleschi]

Hey @mateoguzmana, that's amazing, thank you so much!

We currently have a small issue with tests running on APIs: we also run the E2E tests in debug mode, to catch issues with metro, and debug is currently showing some warnings that are preventing the navigation to that tab. We already have a person looking into them to fix those warnings.

To shed some lights on the E2E tests: we use them for Releases. They are non supposed to be exhaustive, but more of a smoke test to inform the release crew if something really important stopped working. For this reason, we want them to run fast - so this PR is amazing! - not to slow down the release.

We should find the right balance between testing RNTester and release speed.
If the test number starts to slow down the release, we might have to split them in different flows and run them in parallel.

[facebook-github-bot]

@cipolleschi has imported this pull request. If you are a Meta employee, you can view this diff on Phabricator.

[mateoguzmana]

Thanks for all the info @cipolleschi!

We currently have a small issue with tests running on APIs

I noticed this locally as well, I was going to suggest to trigger a command to attempt hide the warnings only if they are visible for tests going to that tab, but great to know you are looking into it.

We should find the right balance between testing RNTester and release speed.
If the test number starts to slow down the release, we might have to split them in different flows and run them in parallel.

Makes a lot of sense.

Would it still be ok for me to send a few PRs for some test cases? They are non-exhaustive and I think it could they could help for smoke testing. For some of the Kotlin migrations I do quite a lot of checks, and wanted to automate that further so it's faster to test locally and with a bit more structure. (maybe I should join the testing release crew 😄)

[cipolleschi]

Would it still be ok for me to send a few PRs for some test cases?

It depends how much time they add to the jobs! 😬

[facebook-github-bot]

@cipolleschi merged this pull request in df6eff4.

[react-native-bot]

This pull request was successfully merged by @mateoguzmana in df6eff4

^{When will my fix make it into a release? | How to file a pick request?}