new_gallery__transition_perf new_gallery__transition_perf regression, now consistently red.

[kf6gpe]

99th_percentile_frame_rasterizer_time_millis also looks affected:

Looks like it may have started around #59804.

cc @yjbanov

[yjbanov]

This is an Android benchmark, changing the platform label.

[zanderso]

@liyuqian ptal

[liyuqian]

Ouch, that engine roll #59804 includes 127 engine commits (see below)... @jason-simmons : do you have a tool to bisect the performance regression with engine commits? Also CC @CareF for an actual case that we need to bisect the engine commits for a performance regression.

liyuqian-macbookpro2:flutter liyuqian$ git log 965fbbe..b5f5e63 --no-merges --oneline
b5f5e6332c [buildroot] Roll to pick up fix for misleading .lib output for Windows (#19129)
40ef2e5b14 Roll CanvasKit to 0.16.2. (#19011)
b7f8d2d110 Roll Skia from 99b047087d51 to b54946b86d33 (7 revisions) (#19128)
9fcfec39c0 Avoid creating a vector when constructing Dart typed data objects for platform messages (#18838)
1551b74213 [shell] Adds a shell test for timezone fetches (#19108)
5157a6b210 Parameterize upscaling for image decoding (#19067)
0266e60a08 Roll Skia from a85e4bf00907 to 99b047087d51 (3 revisions) (#19126)
a726141eb3 Roll Fuchsia Linux SDK from eyQ-Z... to 4Tot8... (#19125)
e7ffcb2970 Roll Skia from 81454dfaa92a to a85e4bf00907 (4 revisions) (#19124)
7c338107d5 Roll Skia from 67237c14a6db to 81454dfaa92a (2 revisions) (#19123)
d3c5fb2fe5 Roll Dart SDK from 7d7c1298e2ca to 54481776c96d (4 revisions) (#19122)
a0c2da7685 Roll Fuchsia Mac SDK from s-b1H... to eqQfp... (#19120)
b93ee31070 Roll Skia from a3a9da74308f to 67237c14a6db (1 revision) (#19116)
354ff10afb Roll Dart SDK from 764e72800f40 to 7d7c1298e2ca (1 revision) (#19117)
f6985267be Roll buildroot to 2415cd5095c186fe5054552bbb26aaa2b8877ac8 (#19106)
a86301c5e1 Roll Dart SDK from 0c2c331d64f3 to 764e72800f40 (15 revisions) (#19113)
59987bf478 Roll Skia from 72f403c343dc to a3a9da74308f (1 revision) (#19110)
2a8436f442 Roll Skia from de175abede4d to 72f403c343dc (127 revisions) (#19107)
d7d9e8b76f Fixes FlutterEngine internal retain cycle (#18595)
a5a0a6014a Roll Dart SDK from f36151e2d2a2 to 0c2c331d64f3 (13 revisions) (#19104)
f3bc5eef2d Roll Fuchsia Mac SDK from Gpg6y... to s-b1H... (#19103)
e9688f8e36 Revert "Roll Skia from de175abede4d to 32d5cfa1f35e (15 revisions) (#19005)" (#19101)
fd34f0fd00 Log Windows errors to stderr (#19095)
2c60d87c39 createOverlaySurface JNI method (#19076)
d14958ea4d Update buildroot (#19098)
e70c87d626 [iOS] handle text plugin negative range (#19068)
90c6b74cb8 Roll Skia from 1f14ca04b9b8 to 8346834d7cfc (3 revisions) (#19097)
f9eeee310c Poor video scaling quality #53080 (#18814)
29b739a514 Roll Dart SDK from c45c07592a95 to f36151e2d2a2 (9 revisions) (#19093)
8c7cbf0b29 Roll Skia from 2518f546e3f2 to 1f14ca04b9b8 (5 revisions) (#19092)
2254fd8844 Roll Fuchsia Linux SDK from aVCEp... to eyQ-Z... (#19090)
54b211ac07 Roll Skia from 4c47d0ddea16 to 2518f546e3f2 (2 revisions) (#19088)
237b5f32ef Roll Dart SDK from 1046888b1bf9 to c45c07592a95 (4 revisions) (#19086)
c9c80011bb Fix windows SkParagraph compilation (#19073)
f4a5f9f448 Roll Fuchsia Mac SDK from 6I5dv... to Gpg6y... (#19082)
ef29ce68aa Roll Dart SDK from 7e5fe50b244c to 1046888b1bf9 (5 revisions) (#19084)
9dc08393be Roll Skia from 3244745c3d4b to 4c47d0ddea16 (3 revisions) (#19083)
137320b5c7 Correct BM_ShellInitializationAndShutdown (#18829)
43ad8347e1 Roll Skia from de980231f62c to 3244745c3d4b (1 revision) (#19081)
b36c83756f Roll Dart SDK from 4b9aa2bd7ecb to 7e5fe50b244c (4 revisions) (#19080)
e3f3fecb18 Manual roll of Dart 021a49e88c...4b9aa2bd7e (#19079)
8b7b244489 Roll Skia from f08a82b52dbf to de980231f62c (1 revision) (#19078)
19da21af86 Roll Skia from 5ca9d1a3414a to f08a82b52dbf (3 revisions) (#19077)
60a9b85f06 Manual roll of Dart 6d4e7d6830...021a49e88c (#19074)
ee51d23660 Roll Skia from f1eb43e8800b to 5ca9d1a3414a (8 revisions) (#19072)
00aa0bc071 take web_ui to null safety (#19027)
71cb701080 Instantiate image codec doc fix (#19062)
4ac08bc07c Roll Fuchsia Linux SDK from YEXby... to aVCEp... (#19071)
99442c6c18 Revert add createOverlaySurface JNI #19040 (#19075)
01fb58211e Roll Skia from 762cb4ea46dd to f1eb43e8800b (8 revisions) (#19063)
c8b077fa1a Revert "Remove pipeline in favor of layer tree holder (#18901)" (#19066)
3d4bb12ba7 Manual roll of Dart f043f9e5f6...6d4e7d6830 (#19065)
01e06ed3d9 Add support for headless mode (#19038)
d0f9ef631c Allow access to raw zircon handle (#19052)
279154e874 Roll Fuchsia Mac SDK from OiCnN... to 6I5dv... (#19061)
e5f0af7b3a Roll Skia from f4ec452b7ab3 to 762cb4ea46dd (1 revision) (#19060)
3a96741247 Platform resolved locale and Android localization refactor (#18645)
469b86b051 Roll Fuchsia Linux SDK from evdT_... to YEXby... (#19058)
def37e27dd Roll Skia from 47111d18f48d to f4ec452b7ab3 (4 revisions) (#19057)
486f26fbf9 Rename trace-whitelist to trace-allowlist (#19047)
4cd36e1dfb Roll Fuchsia Mac SDK from crMVM... to OiCnN... (#19055)
de063512c5 Add createOverlaySurface JNI (#19040)
1ec2a98913 Roll Skia from e3a39f7053bd to 47111d18f48d (9 revisions) (#19053)
01712fb5cf Roll Skia from 602b4024858c to e3a39f7053bd (4 revisions) (#19048)
ca2a370578 started polling the gpu usage (#18752)
c00882a71a updated tests names (#18966)
efd10459e5 Roll Skia from 08ac84a66a5e to 602b4024858c (7 revisions) (#19046)
9d1303e838 Roll Fuchsia Linux SDK from Gm8Wf... to evdT_... (#19045)
20273ea425 Roll Fuchsia Linux SDK from ThzHh... to Gm8Wf... (#18998)
cba8990d13 Adding support for image filtering in CanvasKit backend. (#19035)
1c35046f64 Roll Fuchsia Mac SDK from XHlPK... to crMVM... (#19042)
e697863b57 Roll Skia from 45fe2e8a9914 to 08ac84a66a5e (4 revisions) (#19041)
20c1012dbd Roll Skia from ac16760df463 to 45fe2e8a9914 (4 revisions) (#19028)
0f59c7919d Roll Fuchsia Mac SDK from pVORF... to XHlPK... (#19039)
00c11ebfcc Roll Fuchsia Mac SDK from 9XDKA... to pVORF... (#19037)
5c87c31fd3 Roll Fuchsia Mac SDK from Q196k... to 9XDKA... (#19034)
28ec1f5ebc Roll Fuchsia Mac SDK from H-uAk... to Q196k... (#19032)
801559ac4e Revert to last-known-good-rev of Dart SDK (#19031)
e84d497b9d Fix hit testing logic in fuchsia a11y (#19029)
f7d241fd8a Wire up channel for restoration data (#18042)
983b6e198d Call Shell::NotifyLowMemory when backgrounded/memory pressure occurs on Android (#19026)
e3fdb23304 [fuchsia] Add ability to configure separate data and asset dirs (#18858)
7cb7003a11 onEndFrame JNI (#18867)
8c24c41a0b Roll Skia from 637838d20abd to ac16760df463 (1 revision) (#19025)
be499ab464 Roll Fuchsia Mac SDK from gR0Zc... to H-uAk... (#19022)
b0a0e0e32b Roll Skia from 3d6bf04366f6 to 637838d20abd (2 revisions) (#19021)
194acdfaed apply null safety syntax to mobile dart:ui (#18933)
0a852d8ad7 Revert "Call Shell::NotifyLowMemoryWarning on Android Trim and LowMemory events (#18979)" (#19023)
006564625f Roll Skia from 30212b7941d6 to 3d6bf04366f6 (17 revisions) (#19020)
b2fea9dcfa Roll Skia from 21bbfc6c2dfe to 30212b7941d6 (6 revisions) (#19009)
f6455fa6f2 Roll Dart SDK from 0b64f5488965 to 50836c171e91 (4 revisions) (#19017)
8dcc95dbd4 Roll Fuchsia Mac SDK from Wj0yo... to gR0Zc... (#19015)
1f3aa23e25 Roll Dart SDK from f0ea02bc51f8 to 0b64f5488965 (9 revisions) (#19013)
acf048bd0d Remove log added for local testing (#19012)
2651beba06 Exit before pushing a trace event when layer tree holder is empty (#19008)
e1c622b4aa Make SKSL caching test work on Fuchsia on arm64 (#18572)
ac809b48cb onBeginFrame JNI (#18866)
5f37029a2f Roll Dart SDK from f043f9e5f6ea to f0ea02bc51f8 (22 revisions) (#19010)
48d7091649 Roll buildroot for Windows warning update (#19000)
9f898e98c2 Don't process key events when the text input is not requested (#18990)
4bd6aea07c Always send key events, even if they're used for text input. (#18991)
5171fbdc75 Roll Skia from 32d5cfa1f35e to 21bbfc6c2dfe (5 revisions) (#19006)
87d8888048 Show warning if method response errors occur and error not handled. (#18946)
5ddc122f02 Fix inverted check in creating resource surface (#18989)
71fce02dd6 Fix shift-tab not working by adding more GTK->GLFW key mappings. (#18988)
45386229e2 Roll Skia from de175abede4d to 32d5cfa1f35e (15 revisions) (#19005)
d417772d7a Roll Fuchsia Mac SDK from gAD3P... to Wj0yo... (#19001)
cea5a9c7a0 Roll Dart SDK from b0d2b97d2cd7 to f043f9e5f6ea (6 revisions) (#18997)
a0a92d6c63 Roll Skia from 0ad37b87549b to de175abede4d (1 revision) (#18995)
369e0a9b91 Add ui_benchmarks (#18945)
2a82a08329 [dart] Account for compiler api change (#19002)
f5ca58b188 Roll Dart SDK from e62b89c56d01 to b0d2b97d2cd7 (4 revisions) (#18994)
b82769bdfb Roll Skia from 553496b66f12 to 0ad37b87549b (2 revisions) (#18993)
336d000b43 Roll Skia from f69e40841eb9 to 553496b66f12 (2 revisions) (#18992)
2739bbfcfe [web] Provide a hook to disable location strategy (#18969)
74291b73be Roll Dart SDK from 0fab083e1c2b to e62b89c56d01 (10 revisions) (#18987)
7f71f1d90f Roll Fuchsia Mac SDK from oWhyp... to gAD3P... (#18982)
1ddc6e1d3a Call destructor and fix check (#18985)
e5845af424 Put JNI functions under an interface (#18903)
efd1452a4d Roll Skia from 9c401e7e1ace to f69e40841eb9 (11 revisions) (#18983)
ca26b7558e Make Linux shell plugin constructor descriptions consistent (#18940)
b19a17d5d1 Implement an EGL resource context for the Linux shell. (#18918)
db7f226c5e Include the memory header as unique_ptr is used in ASCII Trie. (#18978)
50cae024a3 Reland: Add RAII wrapper for EGLSurface (#18977)
4dabac999f Roll Fuchsia Linux SDK from 8AiUM... to ThzHh... (#18976)
f5ab179e7a Call Shell::NotifyLowMemoryWarning on Android Trim and LowMemory events (#18979)
141ee78b83 Roll Dart SDK from 2b917f5b6a0e to 0fab083e1c2b (15 revisions) (#18974)

[jason-simmons]

The dev/devicelab/bin/run.dart script has a --local-engine flag that will run the benchmark using a locally built engine. I've used that to bisect within an engine roll.

cd dev/devicelab
../../bin/cache/dart-sdk/bin/dart bin/run.dart --local-engine android_profile -t new_gallery__transition_perf