Add scheduled quarantine runs #19352
Conversation
BrennanConroy
commented
Feb 25, 2020
BrennanConroy
commented
- Run quarantined tests every 4 hours
- Fix running tests on internal
BrennanConroy
added
the
area-infrastructure
|
Somewhat unrelated question, but should we consider running quarantined tests only on master? Is there a point to running them on all prs? |
In limited cases it's useful to run on PRs. I was recently making changes around the watch tests to fix the flakiness, so I was watching the pipeline on my PR. |
|
Yeah but you can just remove the flaky attribute on the test as part of your PR that is fixing it for it to show up in the normal runs |
I don't think we're supposed to be doing that? |
| displayName: Every 4 hours test run | ||
| branches: | ||
| include: | ||
| - master |
There was a problem hiding this comment.
@Pilchie @mkArtakMSFT do we have enough Blazor tests running in Helix to make including schedule runs for the 'blazor-wasm' branch worthwhile?
There was a problem hiding this comment.
I don't think we have any running #18636
There was a problem hiding this comment.
@BrennanConroy you're probably right but I wasn't positive on the current status
Don't think such a change is something we'd want to check in until and unless Quarantine Review (aka @Pilchie @mkArtakMSFT @ajcvickers and @anurse) deem the test(s) fixed. Temporarily, while debugging a PR, sure. Unfortunately, this may require additional PR updates and validation runs after you're done debugging. |
|
Well, if the only use of running quarantined tests on PRs, is when you are working on the quarantined tests, that seems like a lot of wasted helix runs. It doesn't seem too painful to comment out the flaky attribute while you are working on a test's reliability |
There was a problem hiding this comment.
Ignoring the larger policy questions, changes look fine.
Not sure what's up w/ the macOS build but https://dev.azure.com/dnceng/public/_build/results?buildId=536581&view=logs&j=9895f7d8-b062-540b-461c-07f8695bfcc6&t=35dc6bdb-4c39-5586-e98d-711dcc0c4661&l=20 doesn't look good. Searching for "curl: (56) SSLRead() return error -9806" leads to a fair number of suggestions to update the affected application but no root cause. May be enough to retry that job.
|
I found that "error -9806" could be related to using the wrong curl on mac. |
