Skip to content

add --exit flag to dev/devicelab/bin/test_runner.dart #134165

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Merged
merged 2 commits into from
Sep 7, 2023
Merged

add --exit flag to dev/devicelab/bin/test_runner.dart #134165

merged 2 commits into from
Sep 7, 2023

Conversation

andrewkolos
Copy link
Contributor

@andrewkolos andrewkolos commented Sep 6, 2023 •
edited
Loading

Resolves #134070

Adds a flag to the test_runner.dart test script that will cause the test runner to exit upon first failure (or, said another way, exit without retrying).

This is in parity with the --exit flag of dev/devicelab/bin/run.dart.

Pre-launch Checklist

  • I read the Contributor Guide and followed the process outlined there for submitting PRs.
  • I read the Tree Hygiene wiki page, which explains my responsibilities.
  • I read and followed the Flutter Style Guide, including Features we expect every widget to implement.
  • I signed the CLA.
  • I listed at least one issue that this PR fixes in the description above.
  • I updated/added relevant documentation (doc comments with ///).
  • I added new tests to check the change I am making, or this PR is test-exempt.
  • All existing and new tests are passing.

@flutter-dashboard
Copy link

It looks like this pull request may not have tests. Please make sure to add tests before merging. If you need an exemption to this rule, contact Hixie or stuartmorgan on the #hackers channel in Chat (don't just cc them here, they won't see it! Use Discord!).

If you are not sure if you need tests, consider this rule of thumb: the purpose of a test is to make sure someone doesn't accidentally revert the fix. Ask yourself, is there anything in your PR that you feel it is important we not accidentally revert back to how it was before your fix?

Reviewers: Read the Tree Hygiene page and make sure this patch meets those guidelines before LGTMing.

christopherfujino
christopherfujino reviewed Sep 7, 2023
View reviewed changes
dev/devicelab/lib/command/test.dart
argParser.addFlag(
'exit',
help: 'Exit on the first test failure. Currently flakes are intentionally (though '
'incorrectly) not considered to be failures.',
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

why incorrectly?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This is copy-pasted from the --exit flag in run.dart. This verbiage was introduced in a commit from @Hixie. I interpret this as an opinion that tasks should be considered failing if they fail on their first run, regardless of whether or not they pass after retrying.

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

flakes are by definition failing tests (that sometimes don't report their failure).

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

ahh, i see

christopherfujino
christopherfujino reviewed Sep 7, 2023
View reviewed changes
@Hixie
Copy link
Contributor

Hixie commented Sep 7, 2023

test-exempt: test tooling

christopherfujino
christopherfujino approved these changes Sep 7, 2023
View reviewed changes
Copy link
Contributor

@christopherfujino christopherfujino left a comment •
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM. BTW, about the print, what I do is I add an invalid todo comment (like print('debug'); // TODO), which I know Linux analyze will fail on (it enforces the // TODO(ldap) format), so I don't accidentally merge it. The only downside to this approach is it's annoying in editor if your editor highlights lint errors.

@andrewkolos andrewkolos added the autosubmit Merge PR when tree becomes green via auto submit App label Sep 7, 2023
@auto-submit auto-submit bot merged commit aea4552 into flutter:master Sep 7, 2023
@engine-flutter-autoroll engine-flutter-autoroll mentioned this pull request Sep 7, 2023
Merged
engine-flutter-autoroll added a commit to engine-flutter-autoroll/packages that referenced this pull request Sep 7, 2023
@engine-flutter-autoroll
aea4552 add --exit flag to dev/devicelab/bin/test_runner.dart (flutte…
d06a650 
…r/flutter#134165)
auto-submit bot pushed a commit to flutter/packages that referenced this pull request Sep 7, 2023
@engine-flutter-autoroll
Manual roll Flutter from 685ce14 to aea4552 (64 revisions) (#4870)
913c946 
Manual roll Flutter from 685ce14 to aea4552 (64 revisions)

Manual roll requested by [email protected]

flutter/flutter@685ce14...aea4552

2023-09-07 [email protected] add --exit flag to dev/devicelab/bin/test_runner.dart (flutter/flutter#134165)
2023-09-07 [email protected] fix `--exit` flag in dev/devicelab/bin/run.dart (flutter/flutter#134162)
2023-09-07 [email protected] Roll Packages from e7d812c to 22d4754 (9 revisions) (flutter/flutter#134232)
2023-09-07 [email protected] Roll Flutter Engine from 71bea01d3abe to f0b718e28779 (2 revisions) (flutter/flutter#134231)
2023-09-07 [email protected] DropdownRoutePage should dispose the created ScrollController. (flutter/flutter#133941)
2023-09-07 [email protected] Cover some test/widgets tests with leak tracking (flutter/flutter#133803)
2023-09-07 [email protected] SearchDelegate should dispose resources. (flutter/flutter#133948)
2023-09-07 [email protected] Fixed [NavigationRailDestination]'s label opacity while disabled not being coherent with the icon (flutter/flutter#132345)
2023-09-07 [email protected] Roll Flutter Engine from 558136a1ccbf to 71bea01d3abe (2 revisions) (flutter/flutter#134216)
2023-09-07 [email protected] Roll Flutter Engine from 5a45ecd24aa3 to 558136a1ccbf (1 revision) (flutter/flutter#134206)
2023-09-07 [email protected] Roll Flutter Engine from d864ae68db3c to 5a45ecd24aa3 (1 revision) (flutter/flutter#134201)
2023-09-07 [email protected] Fix `TabBar` doesn't use `labelStyle` & `unselectedLabelStyle` color (flutter/flutter#133989)
2023-09-07 [email protected] Fix `DataTable`'s `headingTextStyle` & `dataTextStyle` are not merged with default text style (flutter/flutter#134138)
2023-09-07 [email protected] Roll Flutter Engine from 187c5b3c5f71 to d864ae68db3c (2 revisions) (flutter/flutter#134199)
2023-09-07 [email protected] Reland "Fix `Chip.shape`'s side is not used when provided in Material 3" (flutter/flutter#133856)
2023-09-07 [email protected] Roll Flutter Engine from 75437a3bd002 to 187c5b3c5f71 (1 revision) (flutter/flutter#134193)
2023-09-07 [email protected] Manual roll Flutter Engine from 2c69d05dfafb to 75437a3bd002 (15 revisions) (flutter/flutter#134188)
2023-09-07 [email protected] Revert "Roll Flutter Engine from 2c69d05dfafb to fa14d337449b (6 revisions)" (flutter/flutter#134183)
2023-09-06 49699333+dependabot[bot]@users.noreply.github.com Bump actions/upload-artifact from 3.1.2 to 3.1.3 (flutter/flutter#134173)
2023-09-06 [email protected] Roll Flutter Engine from 2c69d05dfafb to fa14d337449b (6 revisions) (flutter/flutter#134169)
2023-09-06 [email protected] DraggableScrollableActuator should dispose notifier. (flutter/flutter#133917)
2023-09-06 [email protected] Roll Flutter Engine from b04c2a378302 to 2c69d05dfafb (3 revisions) (flutter/flutter#134164)
2023-09-06 [email protected] Clean the fixed TODOs. (flutter/flutter#133859)
2023-09-06 [email protected] Roll Flutter Engine from 839051596b1d to b04c2a378302 (7 revisions) (flutter/flutter#134158)
2023-09-06 [email protected] [Windows Arm64] Also use Windows 11 for Devicelab tests (flutter/flutter#134082)
2023-09-06 [email protected] Fix `subtitleTextStyle.color` isn't applied to the `ListTile.subtitle` in Material 2 (flutter/flutter#133422)
2023-09-06 [email protected] Add `CheckedPopupMenuItem.onTap` callback (flutter/flutter#134000)
2023-09-06 [email protected] MinimumTextContrastGuideline should dispose image. (flutter/flutter#133861)
2023-09-06 [email protected] [flutter_tools] Fix "FormatException: Invalid date format" during version freshness check (flutter/flutter#134088)
2023-09-06 [email protected] Fix not disposed items in Cupertino app and route. (flutter/flutter#134085)
2023-09-06 [email protected] Roll Flutter Engine from a5e7fa6bf81a to 839051596b1d (2 revisions) (flutter/flutter#134140)
2023-09-06 [email protected] _DropdownMenuState should dispose TextEditingController. (flutter/flutter#133914)
2023-09-06 [email protected] Roll Flutter Engine from 5253a33096d1 to a5e7fa6bf81a (1 revision) (flutter/flutter#134137)
2023-09-06 [email protected] Roll Flutter Engine from c7fd088291e2 to 5253a33096d1 (1 revision) (flutter/flutter#134135)
2023-09-06 [email protected] Roll Flutter Engine from 3d9989f1e155 to c7fd088291e2 (1 revision) (flutter/flutter#134132)
2023-09-06 [email protected] Roll Flutter Engine from bace539bb654 to 3d9989f1e155 (3 revisions) (flutter/flutter#134128)
2023-09-06 [email protected] Roll Flutter Engine from 9344685efbc3 to bace539bb654 (1 revision) (flutter/flutter#134104)
2023-09-06 [email protected] Roll Flutter Engine from 0c8c1647dcd0 to 9344685efbc3 (1 revision) (flutter/flutter#134103)
2023-09-06 [email protected] Roll Flutter Engine from 0c663258fd09 to 0c8c1647dcd0 (1 revision) (flutter/flutter#134100)
2023-09-06 [email protected] Roll Flutter Engine from 8bacc3b38707 to 0c663258fd09 (3 revisions) (flutter/flutter#134096)
2023-09-06 [email protected] Roll Flutter Engine from 590349006d23 to 8bacc3b38707 (5 revisions) (flutter/flutter#134089)
2023-09-06 [email protected] Roll Flutter Engine from 5b2cc9d9b8fe to 590349006d23 (2 revisions) (flutter/flutter#134081)
2023-09-05 [email protected] Roll Flutter Engine from 98b036ae708e to 5b2cc9d9b8fe (2 revisions) (flutter/flutter#134080)
2023-09-05 [email protected] Roll Flutter Engine from f4975e04f35e to 98b036ae708e (3 revisions) (flutter/flutter#134077)
...
engine-flutter-autoroll added a commit to engine-flutter-autoroll/packages that referenced this pull request Nov 15, 2023
@engine-flutter-autoroll
aea4552 add --exit flag to dev/devicelab/bin/test_runner.dart (flutte…
549b631 
…r/flutter#134165)
@andrewkolos andrewkolos deleted the add-exit-flag-to-test_runner branch April 29, 2024 18:17
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@Hixie Hixie Hixie left review comments

@christopherfujino christopherfujino christopherfujino approved these changes

Assignees
No one assigned
Labels
autosubmit Merge PR when tree becomes green via auto submit App
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.

Consider adding a --no-retry flag to the devicelab test runner command
3 participants
@andrewkolos @Hixie @christopherfujino