Try using xml logger

[HaoK]

Try using xml vstest logger instead of trx to try and get xunit reporter working. Also add support for logs to windows command and various minor cleanup

[HaoK]

@MattGal can you help me figure out where I need to copy the xml test results file for xml reporter to upload them correctly?

We are trying to see if moving away from trx would get this to work faster, so we installed https://github.com/spekt/xunit.testlogger which is generating an xml file now, but its not being found by the reporter, I tried copying it to root, but maybe it needs to go somewhere else?

I copied TestResults.xml

Copying TestResults/* to Root/

Searching '/datadisks/disk1/work/BD980A0A/w/9A710897/e' for test results files
No results file found in any of the following formats: xunit, junit, trx
+ date -u +%FT%TZ
2020-01-23T22:39:45Z
+ /usr/bin/python3 /datadisks/disk1/work/BD980A0A/p/xunit-reporter.py
2020-01-23T22:39:45.919Z	ERROR  	xunit-reporter.py	xunit-reporter(80)	main	Unable to report xunit results: no test results xml file found.
+ exit 1

[MattGal]

@MattGal can you help me figure out where I need to copy the xml test results file for xml reporter to upload them correctly?

We are trying to see if moving away from trx would get this to work faster, so we installed https://github.com/spekt/xunit.testlogger which is generating an xml file now, but its not being found by the reporter, I tried copying it to root, but maybe it needs to go somewhere else?

I copied TestResults.xml

Copying TestResults/* to Root/

Searching '/datadisks/disk1/work/BD980A0A/w/9A710897/e' for test results files
No results file found in any of the following formats: xunit, junit, trx
+ date -u +%FT%TZ
2020-01-23T22:39:45Z
+ /usr/bin/python3 /datadisks/disk1/work/BD980A0A/p/xunit-reporter.py
2020-01-23T22:39:45.919Z	ERROR  	xunit-reporter.py	xunit-reporter(80)	main	Unable to report xunit results: no test results xml file found.
+ exit 1

Some points:

• If you want XUnit reporter to find your XML, it should just be in $HELIX_WORKITEM_ROOT . This line and the related function do a non-recursive walk of this folder. Since you want to ingest your xunit to Kusto this is the one you want.
• Often this won't hurt folks, but please avoid copying files into root (on docker work items it really doesn't matter) because they will never be cleaned up til the machine is deleted.
• If you just want a file uploaded, put a copy of it (or ideally create it initially) under $HELIX_WORKITEM_UPLOAD_ROOT.
• XUnit output names arbitraily need to match (case-sensitive because Linux):

acceptableXUnitFileNames = [
    "testResults.xml",
    "test-results.xml",
    "test_results.xml"
]

[analogrelay]

I see results in Kusto! 🎉

[analogrelay]

Of course, this means we're only going to get data from un-quarantined tests. We should come up with a plan for how to get this data out. Maybe we have to rethink our quarantine process.

If we ran a single run of all tests, we'd still be able to tell if they were quarantined or not by the traits. Unfortunately, the run would fail even when only quarantined tests fail.

Another option is to change how the quarantine process works and just skip tests that are quarantined. It would mean we can't monitor them for when they get non-flaky, but we'd still be able to track which tests were skipped because they were flaky and track the number of skipped flaky tests over time so that teams can drive that number down.

Food for thought at the next DoI meeting :).

[HaoK]

I think we should probably just add a flaky-test.yml and just run all tests in that run every day on master, and get rid of the flaky test pass in our normal PRs

[HaoK]

Then we'd have all the data in kusto

[analogrelay]

Yep, I think a separate run is the solution. I think we can (and should) continue running them on PRs if we can though. The main downside is that for projects with no quarantined tests, we’ll submit a Helix work item with almost no action, it will run 0 tests. We could start with that though. I did some spelunking and it’s definitely feasible to run a little code to check a test assembly (once built) for any quarantined tests using the vstest APIs. That way we could save submitting a no-op job to helix.

[HaoK]

Cool maybe we can just repurpose the old helix-test pipeline to be the flaky helix test run

[HaoK]

Windows and linux look like they are working now, marking as ready for review

[Pilchie]

Hooray!