Skip to content

Conversation

krystofwoldrich
Copy link
Contributor

@krystofwoldrich krystofwoldrich commented Mar 4, 2025

📢 Type of change

  • Enhancement

📜 Description

This PR adds E2E tests for the sample application.

The tests are the same as introduced in #4535 and #4584

I switched from Detox to Maestro in combination with Jest because Detox uses AndroidTest.apk which starts ContentProviders in the background which breaks our app start logic which checks for background starts.

Since app starts are main goal of adding these tests we need to use tool which uses Android UIAutomator which is closer to how actual users operate the app.

At the moment the android test checks for warm start. This will get updated after the fix from getsentry/sentry-java#4216 is released.

💚 How did you test it?

ci and running locally

📝 Checklist

  • I added tests to verify changes
  • No new PII added or SDK only sends newly added PII if sendDefaultPII is enabled
  • I updated the docs if needed.
  • I updated the wizard if needed.
  • All tests passing
  • No breaking changes

🔮 Next steps

#skip-changelog

@krystofwoldrich krystofwoldrich changed the title Add detox test test(sample): Add E2E tests for app start transactions Mar 4, 2025
Copy link
Contributor

github-actions bot commented Mar 4, 2025

Android (new) Performance metrics 🚀

  Plain With Sentry Diff
Startup time 394.37 ms 387.67 ms -6.69 ms
Size 7.15 MiB 8.39 MiB 1.23 MiB

Baseline results on branch: main

Startup times

Revision Plain With Sentry Diff
07e58c9+dirty 391.00 ms 409.66 ms 18.66 ms
1332acb+dirty 385.00 ms 404.80 ms 19.80 ms
52c0562+dirty 401.23 ms 435.65 ms 34.42 ms
3f05680+dirty 399.86 ms 405.20 ms 5.34 ms
2ec71da+dirty 375.64 ms 431.59 ms 55.95 ms
f06c879+dirty 361.27 ms 407.88 ms 46.61 ms
cc8a6ca+dirty 397.63 ms 393.60 ms -4.03 ms
5e5c392+dirty 360.23 ms 347.88 ms -12.35 ms
b6f8ea2+dirty 397.51 ms 457.40 ms 59.88 ms
575f9da+dirty 337.15 ms 370.47 ms 33.32 ms

App size

Revision Plain With Sentry Diff
07e58c9+dirty 7.15 MiB 8.35 MiB 1.20 MiB
1332acb+dirty 7.15 MiB 8.37 MiB 1.22 MiB
52c0562+dirty 7.15 MiB 8.39 MiB 1.24 MiB
3f05680+dirty 7.15 MiB 8.38 MiB 1.23 MiB
2ec71da+dirty 7.15 MiB 8.38 MiB 1.23 MiB
f06c879+dirty 7.15 MiB 8.12 MiB 997.78 KiB
cc8a6ca+dirty 7.15 MiB 8.39 MiB 1.23 MiB
5e5c392+dirty 7.15 MiB 8.39 MiB 1.23 MiB
b6f8ea2+dirty 7.15 MiB 8.38 MiB 1.23 MiB
575f9da+dirty 7.15 MiB 8.10 MiB 979.68 KiB

Previous results on branch: add-detox-test

Startup times

Revision Plain With Sentry Diff
61ddd2e+dirty 416.74 ms 409.22 ms -7.51 ms
922b103+dirty 387.53 ms 391.51 ms 3.98 ms
ec9549b+dirty 388.98 ms 383.76 ms -5.22 ms
d993785+dirty 359.24 ms 360.76 ms 1.51 ms
7b25cf9+dirty 483.98 ms 526.37 ms 42.39 ms
576acc7+dirty 417.75 ms 416.39 ms -1.36 ms

App size

Revision Plain With Sentry Diff
61ddd2e+dirty 7.15 MiB 8.39 MiB 1.23 MiB
922b103+dirty 7.15 MiB 8.39 MiB 1.23 MiB
ec9549b+dirty 7.15 MiB 8.39 MiB 1.23 MiB
d993785+dirty 7.15 MiB 8.39 MiB 1.23 MiB
7b25cf9+dirty 7.15 MiB 8.39 MiB 1.23 MiB
576acc7+dirty 7.15 MiB 8.39 MiB 1.23 MiB

Copy link
Contributor

github-actions bot commented Mar 4, 2025

iOS (legacy) Performance metrics 🚀

  Plain With Sentry Diff
Startup time 1240.88 ms 1248.47 ms 7.59 ms
Size 2.63 MiB 3.75 MiB 1.12 MiB

Baseline results on branch: main

Startup times

Revision Plain With Sentry Diff
7fd512a+dirty 1218.78 ms 1217.25 ms -1.53 ms
ae2f6bc+dirty 1226.13 ms 1227.71 ms 1.58 ms
c398f67+dirty 1219.67 ms 1225.66 ms 5.99 ms
9cd0e9f+dirty 1224.94 ms 1239.88 ms 14.94 ms
80e955a+dirty 1227.94 ms 1230.26 ms 2.32 ms
7143e6f+dirty 1237.04 ms 1233.88 ms -3.16 ms
1faf8e3+dirty 1214.87 ms 1222.83 ms 7.97 ms
cdf2bdf+dirty 1230.10 ms 1238.08 ms 7.98 ms
cdc3945+dirty 1202.43 ms 1223.65 ms 21.23 ms
1332acb+dirty 1230.53 ms 1234.54 ms 4.01 ms

App size

Revision Plain With Sentry Diff
7fd512a+dirty 2.36 MiB 3.10 MiB 753.35 KiB
ae2f6bc+dirty 2.63 MiB 3.70 MiB 1.06 MiB
c398f67+dirty 2.36 MiB 3.04 MiB 696.27 KiB
9cd0e9f+dirty 2.36 MiB 3.08 MiB 735.56 KiB
80e955a+dirty 2.63 MiB 3.70 MiB 1.06 MiB
7143e6f+dirty 2.63 MiB 3.70 MiB 1.06 MiB
1faf8e3+dirty 2.36 MiB 3.08 MiB 736.75 KiB
cdf2bdf+dirty 2.36 MiB 3.12 MiB 779.40 KiB
cdc3945+dirty 2.63 MiB 3.75 MiB 1.12 MiB
1332acb+dirty 2.36 MiB 3.11 MiB 759.86 KiB

Previous results on branch: add-detox-test

Startup times

Revision Plain With Sentry Diff
d993785+dirty 1226.10 ms 1232.39 ms 6.29 ms
ec9549b+dirty 1233.27 ms 1237.41 ms 4.14 ms
61ddd2e+dirty 1235.82 ms 1233.81 ms -2.00 ms
576acc7+dirty 1213.31 ms 1217.08 ms 3.78 ms
7b25cf9+dirty 1233.31 ms 1235.75 ms 2.44 ms
922b103+dirty 1233.41 ms 1226.48 ms -6.93 ms

App size

Revision Plain With Sentry Diff
d993785+dirty 2.63 MiB 3.75 MiB 1.12 MiB
ec9549b+dirty 2.63 MiB 3.75 MiB 1.12 MiB
61ddd2e+dirty 2.63 MiB 3.75 MiB 1.12 MiB
576acc7+dirty 2.63 MiB 3.75 MiB 1.12 MiB
7b25cf9+dirty 2.63 MiB 3.75 MiB 1.12 MiB
922b103+dirty 2.63 MiB 3.75 MiB 1.12 MiB

Copy link
Contributor

github-actions bot commented Mar 4, 2025

iOS (new) Performance metrics 🚀

  Plain With Sentry Diff
Startup time 1216.67 ms 1210.18 ms -6.50 ms
Size 3.19 MiB 4.32 MiB 1.13 MiB

Baseline results on branch: main

Startup times

Revision Plain With Sentry Diff
7fd512a+dirty 1239.41 ms 1241.50 ms 2.09 ms
ae2f6bc+dirty 1220.82 ms 1217.93 ms -2.88 ms
c398f67+dirty 1227.31 ms 1230.00 ms 2.69 ms
9cd0e9f+dirty 1244.61 ms 1247.43 ms 2.82 ms
80e955a+dirty 1243.63 ms 1245.58 ms 1.95 ms
7143e6f+dirty 1228.29 ms 1229.02 ms 0.73 ms
1faf8e3+dirty 1223.38 ms 1220.56 ms -2.82 ms
cdf2bdf+dirty 1237.04 ms 1237.86 ms 0.82 ms
cdc3945+dirty 1216.51 ms 1223.55 ms 7.04 ms
1332acb+dirty 1243.98 ms 1241.12 ms -2.86 ms

App size

Revision Plain With Sentry Diff
7fd512a+dirty 2.92 MiB 3.66 MiB 758.62 KiB
ae2f6bc+dirty 3.19 MiB 4.26 MiB 1.08 MiB
c398f67+dirty 2.92 MiB 3.60 MiB 701.89 KiB
9cd0e9f+dirty 2.92 MiB 3.64 MiB 741.23 KiB
80e955a+dirty 3.19 MiB 4.26 MiB 1.08 MiB
7143e6f+dirty 3.19 MiB 4.26 MiB 1.08 MiB
1faf8e3+dirty 2.92 MiB 3.64 MiB 742.61 KiB
cdf2bdf+dirty 2.92 MiB 3.69 MiB 790.57 KiB
cdc3945+dirty 3.19 MiB 4.32 MiB 1.13 MiB
1332acb+dirty 2.92 MiB 3.67 MiB 772.45 KiB

Previous results on branch: add-detox-test

Startup times

Revision Plain With Sentry Diff
d993785+dirty 1210.45 ms 1227.92 ms 17.46 ms
ec9549b+dirty 1216.31 ms 1221.02 ms 4.71 ms
61ddd2e+dirty 1232.92 ms 1230.49 ms -2.43 ms
576acc7+dirty 1221.35 ms 1234.94 ms 13.59 ms
7b25cf9+dirty 1222.13 ms 1223.43 ms 1.31 ms
922b103+dirty 1218.81 ms 1211.89 ms -6.93 ms

App size

Revision Plain With Sentry Diff
d993785+dirty 3.19 MiB 4.32 MiB 1.13 MiB
ec9549b+dirty 3.19 MiB 4.32 MiB 1.13 MiB
61ddd2e+dirty 3.19 MiB 4.32 MiB 1.13 MiB
576acc7+dirty 3.19 MiB 4.32 MiB 1.13 MiB
7b25cf9+dirty 3.19 MiB 4.32 MiB 1.13 MiB
922b103+dirty 3.19 MiB 4.32 MiB 1.13 MiB

Base automatically changed from fix-attach-app-start-to-the-first-started-root-span to main March 4, 2025 17:57
Copy link
Contributor

github-actions bot commented Mar 5, 2025

Android (legacy) Performance metrics 🚀

  Plain With Sentry Diff
Startup time 471.69 ms 469.09 ms -2.60 ms
Size 17.75 MiB 20.12 MiB 2.37 MiB

Baseline results on branch: main

Startup times

Revision Plain With Sentry Diff
22e31b6 396.48 ms 419.64 ms 23.16 ms
dadc233+dirty 333.78 ms 343.94 ms 10.16 ms
63ed251 457.74 ms 441.54 ms -16.20 ms
34aba08 328.10 ms 342.84 ms 14.74 ms
5dff5ee 462.32 ms 456.26 ms -6.06 ms
0c98663 437.75 ms 420.70 ms -17.05 ms
21a0abb 442.40 ms 422.88 ms -19.52 ms
ed3d77e 386.05 ms 370.62 ms -15.42 ms
700cbf4 425.56 ms 436.26 ms 10.70 ms
77680ec 482.22 ms 475.07 ms -7.16 ms

App size

Revision Plain With Sentry Diff
22e31b6 17.73 MiB 19.84 MiB 2.10 MiB
dadc233+dirty 17.73 MiB 19.75 MiB 2.02 MiB
63ed251 17.74 MiB 20.08 MiB 2.34 MiB
34aba08 17.73 MiB 19.80 MiB 2.07 MiB
5dff5ee 17.75 MiB 20.11 MiB 2.36 MiB
0c98663 17.75 MiB 20.12 MiB 2.37 MiB
21a0abb 17.75 MiB 20.11 MiB 2.36 MiB
ed3d77e 17.74 MiB 20.08 MiB 2.35 MiB
700cbf4 17.73 MiB 20.07 MiB 2.33 MiB
77680ec 17.74 MiB 20.08 MiB 2.34 MiB

Previous results on branch: add-detox-test

Startup times

Revision Plain With Sentry Diff
922b103 441.90 ms 436.64 ms -5.25 ms
7b25cf9 443.10 ms 501.36 ms 58.26 ms
d993785 399.15 ms 415.98 ms 16.83 ms
ec9549b 397.13 ms 417.94 ms 20.81 ms
61ddd2e 384.94 ms 420.98 ms 36.04 ms
576acc7 400.42 ms 427.40 ms 26.99 ms

App size

Revision Plain With Sentry Diff
922b103 17.75 MiB 20.12 MiB 2.37 MiB
7b25cf9 17.75 MiB 20.12 MiB 2.37 MiB
d993785 17.75 MiB 20.12 MiB 2.37 MiB
ec9549b 17.75 MiB 20.12 MiB 2.37 MiB
61ddd2e 17.75 MiB 20.12 MiB 2.37 MiB
576acc7 17.75 MiB 20.12 MiB 2.37 MiB

@krystofwoldrich krystofwoldrich marked this pull request as ready for review March 6, 2025 17:10
@krystofwoldrich
Copy link
Contributor Author

I reverted the network config removal commit, although locally it worked. In CI, the configuration is needed.

Copy link
Collaborator

@antonis antonis left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM 🎸
Thank you for adding those tests to verify the cold/warm app starts Krystof 🙇

@krystofwoldrich krystofwoldrich enabled auto-merge (squash) March 10, 2025 18:56
@krystofwoldrich krystofwoldrich merged commit e46354e into main Mar 10, 2025
62 checks passed
@krystofwoldrich krystofwoldrich deleted the add-detox-test branch March 10, 2025 18:59
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants