Flaky E2E Metrics App Start Time

[krystofwoldrich]

Description

Can we improve the app start time measurement tests?

Often the test fails (looking at the last 30 commits on main cca 1/3 where only metrics failed) because the App With Sentry started faster.

https://github.com/getsentry/sentry-react-native/actions/runs/6710776019/job/18236705951

[krystofwoldrich]

@vaind I've seen there are some conditions in place to avoid this, is there something else that we could improve?

[krystofwoldrich]

Recently I also noticed this

https://github.com/getsentry/sentry-react-native/actions/runs/6954144218/job/18920431622

StartupTimeTest > startup time() FAILED
    org.openqa.selenium.WebDriverException: An unknown error has occurred
    Build info: version: '4.2.1', revision: 'ac4d0fdd4a'
    System info: host: 'Mac-1700637830941.local', ip: '10.213.5.236', os.name: 'Mac OS X', os.arch: 'x86_64', os.version: '12.7.1', java.version: '11.0.21'
    Driver info: io.appium.java_client.ios.IOSDriver
    Command: [744b0c72-5b9d-4a04-b3a2-c9f0a4490f4c, terminateApp {bundleId=io.sentry.rn.perf-test-app-sentry}]
    Capabilities {appium:appiumData: {appiumVersion: 2.0.0, automationName: XCUITest}, appium:automationName: XCUITest, appium:chromeOptions: {w3c: false}, appium:commandTimeouts: {default: 60000}, appium:databaseEnabled: false, appium:deviceContextId: b77f8d23-2f05-4e32-81a8-d89..., appium:deviceName: 00008101-000559D20203001E, appium:disableWindowAnimation: true, appium:javascriptEnabled: true, appium:jobUuid: b68759cda36f451eb12b6484163..., appium:locationContextEnabled: false, appium:networkConnectionEnabled: false, appium:orientation: PORTRAIT, appium:platformVersion: 14.8, appium:takesScreenshot: true, appium:testobject_device: iPhone_12_Pro_Max_real_us, appium:testobject_device_name: iPhone 12 Pro Max, appium:testobject_device_session_id: 926ee55c-d870-4465-b171-dc5..., appium:testobject_test_report_api_url: https://api.us-west-1.sauce.../, appium:testobject_test_report_url: https://app.saucelabs.com/t..., appium:testobject_user_id: sentryio, appium:udid: 00008101-000559D20203001E, appium:usedCachedDevice: false, appium:webDriverAgentUrl: http://127.0.0.1:5720/, appium:webStorageEnabled: false, browserName: safari, platformName: IOS, sauce:options: {appiumVersion: 2.0.0, build: CI getsentry/sentry-react-n..., name: Performance tests, realDevice: true, tags: [ios, ci]}}
    Session ID: 744b0c72-5b9d-4a04-b3a2-c9f0a4490f4c
        at java.base@11.0.21/jdk.internal.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
        at java.base@11.0.21/jdk.internal.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62)
        at java.base@11.0.21/jdk.internal.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
        at java.base@11.0.21/java.lang.reflect.Constructor.newInstance(Constructor.java:490)
        at app//org.openqa.selenium.remote.codec.w3c.W3CHttpResponseCodec.createException(W3CHttpResponseCodec.java:200)
        at app//org.openqa.selenium.remote.codec.w3c.W3CHttpResponseCodec.decode(W3CHttpResponseCodec.java:133)
        at app//org.openqa.selenium.remote.codec.w3c.W3CHttpResponseCodec.decode(W3CHttpResponseCodec.java:53)
        at app//org.openqa.selenium.remote.HttpCommandExecutor.execute(HttpCommandExecutor.java:184)
        at app//io.appium.java_client.remote.AppiumCommandExecutor.execute(AppiumCommandExecutor.java:180)
        at app//org.openqa.selenium.remote.RemoteWebDriver.execute(RemoteWebDriver.java:551)
        at app//org.openqa.selenium.remote.RemoteWebDriver.execute(RemoteWebDriver.java:606)
        at app//io.appium.java_client.AppiumDriver.execute(AppiumDriver.java:[262](https://github.com/getsentry/sentry-react-native/actions/runs/6954144218/job/18920431622#step:14:267))
        at app//io.appium.java_client.CommandExecutionHelper.execute(CommandExecutionHelper.java:27)
        at app//io.appium.java_client.InteractsWithApps.terminateApp(InteractsWithApps.java:179)
        at app//io.appium.java_client.InteractsWithApps.terminateApp(InteractsWithApps.java:163)
        at app//StartupTimeTest.collectAppStartupTimes(StartupTimeTest.kt:194)
        at app//StartupTimeTest.access$collectAppStartupTimes(StartupTimeTest.kt:16)
        at app//StartupTimeTest$collectStartupTimes$1.invoke(StartupTimeTest.kt:114)
        at app//StartupTimeTest$collectStartupTimes$1.invoke(StartupTimeTest.kt:100)
        at app//TestBase.withDriver(TestBase.kt:26)
        at app//StartupTimeTest.collectStartupTimes(StartupTimeTest.kt:100)
        at app//StartupTimeTest.startup time(StartupTimeTest.kt:44)

[krystofwoldrich]

We could ignore the invalid results. And not mark the job as failed.

[krystofwoldrich]

We are not measuring the app start correctly.

https://github.com/getsentry/action-app-sdk-overhead-metrics/blob/9615bad7552f3f34b12b433b5c31ea08acf7c301/src/test/kotlin/StartupTimeTest.kt#L223

The main activity of the application will be displayed before RN start executing and thus before the JS SDK inits.