Skip to content

System.Text.Json.SourceGeneration intermittent failures #72101

@greenEkatherine

Description

@greenEkatherine

Frequency:

  • 4/1-7/22 ... 618 failures, first one starting on 7/5 ... 36 failures per day. Many of them in Official runs
let failedTests = (testNameSubstring : string, methodName : string, includePR : bool, messageSubstr: string, includePassedOnRerun : bool) {
cluster('engsrvprod.kusto.windows.net').database('engineeringdata').AzureDevOpsTests
    | where TestName contains testNameSubstring
    | where includePassedOnRerun or (Outcome == 'Failed')
    | extend startOfTestName = indexof_regex(TestName, @"[^.]+$")
    | extend Method = substring(TestName, startOfTestName)
    | extend Type = substring(TestName, 0, startOfTestName - 1)
    | project-away startOfTestName
    | where (methodName == '') or (Method == methodName)
    | where Message contains messageSubstr
    | distinct JobId, WorkItemId, Message, StackTrace, Method, Type, Arguments, Outcome
    | join kind=inner (cluster('engsrvprod.kusto.windows.net').database('engineeringdata').Jobs
        | where ((Branch == 'refs/heads/main') or (Branch == 'refs/heads/master') or (includePR and (Source startswith "pr/")))
        | where Type startswith "test/functional/cli/"
            and not(Properties contains "runtime-staging")
        | summarize arg_max(Finished, Properties, Type, Branch, Source, Started, QueueName) by JobId
        | project-rename JobType = Type) on JobId
    | extend PropertiesJson = parse_json(Properties)
    | extend OS = replace_regex(tostring(PropertiesJson.operatingSystem), @'\((.*)\).*|([^\(].*)', @'\1\2')
    | extend Runtime = iif(PropertiesJson.runtimeFlavor == "mono", "Mono", iif(PropertiesJson.DefinitionName contains "coreclr", "CoreCLR", ""))
    | extend TargetBranch = extractjson("$.['System.PullRequest.TargetBranch']", Properties)
    | extend Architecture = PropertiesJson.architecture
    | extend Scenario = iif(isempty(PropertiesJson.scenario), "--", PropertiesJson.scenario)
    //| extend DefinitionName = PropertiesJson.DefinitionName
    | project-away PropertiesJson
};
failedTests('System.Text.Json.SourceGeneration', '', true, 'NaN', true);

Assertions failed in several builds for tests:

Example failure:
Assert.Equal() Failure Expected: 1.10000002 Actual: NaN log results

cc @jtschuster

Runfo Tracking Issue: System.Text.Json failures on Mono_Release

Build Definition Kind Run Name Console Core Dump Test Results Run Client
1900886 runtime-extra-platforms Rolling net7.0-Android-Release-x86-Mono_Release-Ubuntu.1804.Amd64.Android.29.Open console.log test results runclient.py
1900886 runtime-extra-platforms Rolling net7.0-Android-Release-x86-Mono_Release-Ubuntu.1804.Amd64.Android.29.Open console.log test results runclient.py
1900886 runtime-extra-platforms Rolling net7.0-Android-Release-x86-Mono_Release-Ubuntu.1804.Amd64.Android.29.Open console.log test results runclient.py
1900886 runtime-extra-platforms Rolling net7.0-Android-Release-x86-Mono_Release-Ubuntu.1804.Amd64.Android.29.Open console.log test results runclient.py
1900886 runtime-extra-platforms Rolling net7.0-Android-Release-x86-Mono_Release-Ubuntu.1804.Amd64.Android.29.Open console.log test results runclient.py
1900886 runtime-extra-platforms Rolling net7.0-Android-Release-x86-Mono_Release-Ubuntu.1804.Amd64.Android.29.Open console.log test results runclient.py
1900886 runtime-extra-platforms Rolling net7.0-Android-Release-x86-Mono_Release-Ubuntu.1804.Amd64.Android.29.Open console.log test results runclient.py
1900886 runtime-extra-platforms Rolling net7.0-Android-Release-x86-Mono_Release-Ubuntu.1804.Amd64.Android.29.Open console.log test results runclient.py
1900886 runtime-extra-platforms Rolling net7.0-Android-Release-x86-Mono_Release-Ubuntu.1804.Amd64.Android.29.Open console.log test results runclient.py
1900886 runtime-extra-platforms Rolling net7.0-Android-Release-x86-Mono_Release-Ubuntu.1804.Amd64.Android.29.Open console.log test results runclient.py
1900886 runtime-extra-platforms Rolling net7.0-Android-Release-x86-Mono_Release-Ubuntu.1804.Amd64.Android.29.Open console.log test results runclient.py
1900886 runtime-extra-platforms Rolling net7.0-Android-Release-x86-Mono_Release-Ubuntu.1804.Amd64.Android.29.Open console.log test results runclient.py
1900886 runtime-extra-platforms Rolling net7.0-Android-Release-x86-Mono_Release-Ubuntu.1804.Amd64.Android.29.Open console.log test results runclient.py
1900886 runtime-extra-platforms Rolling net7.0-Android-Release-x86-Mono_Release-Ubuntu.1804.Amd64.Android.29.Open console.log test results runclient.py
1900886 runtime-extra-platforms Rolling net7.0-Android-Release-x86-Mono_Release-Ubuntu.1804.Amd64.Android.29.Open console.log test results runclient.py
1900886 runtime-extra-platforms Rolling net7.0-Android-Release-x86-Mono_Release-Ubuntu.1804.Amd64.Android.29.Open console.log test results runclient.py
1900886 runtime-extra-platforms Rolling net7.0-Android-Release-x86-Mono_Release-Ubuntu.1804.Amd64.Android.29.Open console.log test results runclient.py
1900886 runtime-extra-platforms Rolling net7.0-Android-Release-x86-Mono_Release-Ubuntu.1804.Amd64.Android.29.Open console.log test results runclient.py
1900886 runtime-extra-platforms Rolling net7.0-Android-Release-x86-Mono_Release-Ubuntu.1804.Amd64.Android.29.Open console.log test results runclient.py
1900783 runtime-extra-platforms PR 67861 net7.0-tvOS-Release-arm64-Mono_Release-OSX.1100.Amd64.AppleTV.Open console.log runclient.py
1900783 runtime-extra-platforms PR 67861 net7.0-iOSSimulator-Release-x64-Mono_Release-OSX.1200.Amd64.Open console.log runclient.py
1900774 runtime-extra-platforms PR 72545 net7.0-tvOS-Release-arm64-Mono_Release-OSX.1100.Amd64.AppleTV.Open console.log runclient.py
1900774 runtime-extra-platforms PR 72545 net7.0-tvOSSimulator-Release-x64-Mono_Release-OSX.1200.Amd64.Open console.log runclient.py
1900571 runtime-extra-platforms Rolling net7.0-tvOS-Release-arm64-Mono_Release-OSX.1100.Amd64.AppleTV.Open console.log runclient.py
1900571 runtime-extra-platforms Rolling net7.0-tvOSSimulator-Release-x64-Mono_Release-OSX.1200.Amd64.Open console.log runclient.py
1900479 runtime-extra-platforms Rolling net7.0-Android-Release-x86-Mono_Release-Ubuntu.1804.Amd64.Android.29.Open console.log test results runclient.py
1900479 runtime-extra-platforms Rolling net7.0-Android-Release-x86-Mono_Release-Ubuntu.1804.Amd64.Android.29.Open console.log test results runclient.py
1900479 runtime-extra-platforms Rolling net7.0-Android-Release-x86-Mono_Release-Ubuntu.1804.Amd64.Android.29.Open console.log test results runclient.py
1900479 runtime-extra-platforms Rolling net7.0-Android-Release-x86-Mono_Release-Ubuntu.1804.Amd64.Android.29.Open console.log test results runclient.py
1900479 runtime-extra-platforms Rolling net7.0-Android-Release-x86-Mono_Release-Ubuntu.1804.Amd64.Android.29.Open console.log test results runclient.py
1900479 runtime-extra-platforms Rolling net7.0-Android-Release-x86-Mono_Release-Ubuntu.1804.Amd64.Android.29.Open console.log test results runclient.py
1900479 runtime-extra-platforms Rolling net7.0-Android-Release-x86-Mono_Release-Ubuntu.1804.Amd64.Android.29.Open console.log test results runclient.py
1900479 runtime-extra-platforms Rolling net7.0-Android-Release-x86-Mono_Release-Ubuntu.1804.Amd64.Android.29.Open console.log test results runclient.py
1900479 runtime-extra-platforms Rolling net7.0-Android-Release-x86-Mono_Release-Ubuntu.1804.Amd64.Android.29.Open console.log test results runclient.py
1900134 runtime-extra-platforms Rolling net7.0-Android-Release-x86-Mono_Release-Ubuntu.1804.Amd64.Android.29.Open console.log test results runclient.py
1900062 runtime-extra-platforms Rolling net7.0-tvOS-Release-arm64-Mono_Release-OSX.1015.Amd64.AppleTV.Open console.log test results runclient.py
1900062 runtime-extra-platforms Rolling net7.0-Android-Release-x86-Mono_Release-Ubuntu.1804.Amd64.Android.29.Open console.log test results runclient.py
1900062 runtime-extra-platforms Rolling net7.0-Android-Release-x86-Mono_Release-Ubuntu.1804.Amd64.Android.29.Open console.log test results runclient.py
1900062 runtime-extra-platforms Rolling net7.0-Android-Release-x86-Mono_Release-Ubuntu.1804.Amd64.Android.29.Open console.log test results runclient.py
1899657 runtime-extra-platforms Rolling net7.0-Android-Release-x86-Mono_Release-Ubuntu.1804.Amd64.Android.29.Open console.log test results runclient.py
1899657 runtime-extra-platforms Rolling net7.0-Android-Release-x86-Mono_Release-Ubuntu.1804.Amd64.Android.29.Open console.log test results runclient.py
1899657 runtime-extra-platforms Rolling net7.0-Android-Release-x86-Mono_Release-Ubuntu.1804.Amd64.Android.29.Open console.log test results runclient.py
1899657 runtime-extra-platforms Rolling net7.0-Android-Release-x86-Mono_Release-Ubuntu.1804.Amd64.Android.29.Open console.log test results runclient.py
1899657 runtime-extra-platforms Rolling net7.0-Android-Release-x86-Mono_Release-Ubuntu.1804.Amd64.Android.29.Open console.log test results runclient.py
1899657 runtime-extra-platforms Rolling net7.0-Android-Release-x86-Mono_Release-Ubuntu.1804.Amd64.Android.29.Open console.log test results runclient.py
1899657 runtime-extra-platforms Rolling net7.0-Android-Release-x86-Mono_Release-Ubuntu.1804.Amd64.Android.29.Open console.log test results runclient.py
1899657 runtime-extra-platforms Rolling net7.0-Android-Release-x86-Mono_Release-Ubuntu.1804.Amd64.Android.29.Open console.log test results runclient.py
1899657 runtime-extra-platforms Rolling net7.0-Android-Release-x86-Mono_Release-Ubuntu.1804.Amd64.Android.29.Open console.log test results runclient.py
1899657 runtime-extra-platforms Rolling net7.0-Android-Release-x86-Mono_Release-Ubuntu.1804.Amd64.Android.29.Open console.log test results runclient.py
1899657 runtime-extra-platforms Rolling net7.0-Android-Release-x86-Mono_Release-Ubuntu.1804.Amd64.Android.29.Open console.log test results runclient.py
1899657 runtime-extra-platforms Rolling net7.0-Android-Release-x86-Mono_Release-Ubuntu.1804.Amd64.Android.29.Open console.log test results runclient.py
1899657 runtime-extra-platforms Rolling net7.0-Android-Release-x86-Mono_Release-Ubuntu.1804.Amd64.Android.29.Open console.log test results runclient.py
1899657 runtime-extra-platforms Rolling net7.0-Android-Release-x86-Mono_Release-Ubuntu.1804.Amd64.Android.29.Open console.log test results runclient.py
1899657 runtime-extra-platforms Rolling net7.0-Android-Release-x86-Mono_Release-Ubuntu.1804.Amd64.Android.29.Open console.log test results runclient.py
1899657 runtime-extra-platforms Rolling net7.0-Android-Release-x86-Mono_Release-Ubuntu.1804.Amd64.Android.29.Open console.log test results runclient.py
1899657 runtime-extra-platforms Rolling net7.0-Android-Release-x86-Mono_Release-Ubuntu.1804.Amd64.Android.29.Open console.log test results runclient.py
1899657 runtime-extra-platforms Rolling net7.0-Android-Release-x86-Mono_Release-Ubuntu.1804.Amd64.Android.29.Open console.log test results runclient.py
1899117 runtime-extra-platforms Rolling net7.0-Android-Release-x86-Mono_Release-Ubuntu.1804.Amd64.Android.29.Open console.log test results runclient.py
1899117 runtime-extra-platforms Rolling net7.0-Android-Release-x86-Mono_Release-Ubuntu.1804.Amd64.Android.29.Open console.log test results runclient.py
1899115 runtime-extra-platforms Rolling net7.0-Android-Release-x86-Mono_Release-Ubuntu.1804.Amd64.Android.29.Open console.log test results runclient.py
1899115 runtime-extra-platforms Rolling net7.0-Android-Release-x86-Mono_Release-Ubuntu.1804.Amd64.Android.29.Open console.log test results runclient.py
1899115 runtime-extra-platforms Rolling net7.0-Android-Release-x86-Mono_Release-Ubuntu.1804.Amd64.Android.29.Open console.log test results runclient.py
1899115 runtime-extra-platforms Rolling net7.0-Android-Release-x86-Mono_Release-Ubuntu.1804.Amd64.Android.29.Open console.log test results runclient.py
1899115 runtime-extra-platforms Rolling net7.0-Android-Release-x86-Mono_Release-Ubuntu.1804.Amd64.Android.29.Open console.log test results runclient.py
1899115 runtime-extra-platforms Rolling net7.0-Android-Release-x86-Mono_Release-Ubuntu.1804.Amd64.Android.29.Open console.log test results runclient.py
1899115 runtime-extra-platforms Rolling net7.0-Android-Release-x86-Mono_Release-Ubuntu.1804.Amd64.Android.29.Open console.log test results runclient.py
1899115 runtime-extra-platforms Rolling net7.0-Android-Release-x86-Mono_Release-Ubuntu.1804.Amd64.Android.29.Open console.log test results runclient.py
1898268 runtime-extra-platforms PR 67861 net7.0-Browser-Release-wasm-Mono_Release-WasmTestOnBrowser-Ubuntu.1804.Amd64.Open console.log runclient.py
1898268 runtime-extra-platforms PR 67861 net7.0-Browser-Release-wasm-Mono_Release-WasmTestOnBrowser-Ubuntu.1804.Amd64.Open console.log runclient.py
1898268 runtime-extra-platforms PR 67861 net7.0-Browser-Release-wasm-Mono_Release-WasmTestOnBrowser-Ubuntu.1804.Amd64.Open console.log runclient.py
1898268 runtime-extra-platforms PR 67861 net7.0-Android-Release-x86-Mono_Release-Ubuntu.1804.Amd64.Android.29.Open console.log test results runclient.py
1898268 runtime-extra-platforms PR 67861 net7.0-Android-Release-x86-Mono_Release-Ubuntu.1804.Amd64.Android.29.Open console.log test results runclient.py
1898268 runtime-extra-platforms PR 67861 net7.0-Android-Release-x86-Mono_Release-Ubuntu.1804.Amd64.Android.29.Open console.log test results runclient.py
1898268 runtime-extra-platforms PR 67861 net7.0-Android-Release-x86-Mono_Release-Ubuntu.1804.Amd64.Android.29.Open console.log test results runclient.py
1898268 runtime-extra-platforms PR 67861 net7.0-Android-Release-x86-Mono_Release-Ubuntu.1804.Amd64.Android.29.Open console.log test results runclient.py
1898268 runtime-extra-platforms PR 67861 net7.0-Android-Release-x86-Mono_Release-Ubuntu.1804.Amd64.Android.29.Open console.log test results runclient.py
1898268 runtime-extra-platforms PR 67861 net7.0-Android-Release-x86-Mono_Release-Ubuntu.1804.Amd64.Android.29.Open console.log test results runclient.py
1898268 runtime-extra-platforms PR 67861 net7.0-Android-Release-x86-Mono_Release-Ubuntu.1804.Amd64.Android.29.Open console.log test results runclient.py
1898268 runtime-extra-platforms PR 67861 net7.0-Android-Release-x86-Mono_Release-Ubuntu.1804.Amd64.Android.29.Open console.log test results runclient.py
1898170 runtime-extra-platforms Rolling net7.0-Android-Release-x86-Mono_Release-Ubuntu.1804.Amd64.Android.29.Open console.log test results runclient.py
1898170 runtime-extra-platforms Rolling net7.0-Android-Release-x86-Mono_Release-Ubuntu.1804.Amd64.Android.29.Open console.log test results runclient.py
1898170 runtime-extra-platforms Rolling net7.0-MacCatalyst-Release-arm64-Mono_Release-OSX.1100.Arm64.Open console.log runclient.py
1898169 runtime-extra-platforms Rolling net7.0-Android-Release-x86-Mono_Release-Ubuntu.1804.Amd64.Android.29.Open console.log test results runclient.py
1898169 runtime-extra-platforms Rolling net7.0-Android-Release-x86-Mono_Release-Ubuntu.1804.Amd64.Android.29.Open console.log test results runclient.py
1898169 runtime-extra-platforms Rolling net7.0-Android-Release-x86-Mono_Release-Ubuntu.1804.Amd64.Android.29.Open console.log test results runclient.py
1898169 runtime-extra-platforms Rolling net7.0-Android-Release-x86-Mono_Release-Ubuntu.1804.Amd64.Android.29.Open console.log test results runclient.py
1898169 runtime-extra-platforms Rolling net7.0-Android-Release-x86-Mono_Release-Ubuntu.1804.Amd64.Android.29.Open console.log test results runclient.py
1898169 runtime-extra-platforms Rolling net7.0-Android-Release-x86-Mono_Release-Ubuntu.1804.Amd64.Android.29.Open console.log test results runclient.py
1898169 runtime-extra-platforms Rolling net7.0-Android-Release-x86-Mono_Release-Ubuntu.1804.Amd64.Android.29.Open console.log test results runclient.py
1898169 runtime-extra-platforms Rolling net7.0-Android-Release-x86-Mono_Release-Ubuntu.1804.Amd64.Android.29.Open console.log test results runclient.py
1897635 runtime-extra-platforms PR 72124 net7.0-Browser-Release-wasm-Mono_Release-WasmTestOnBrowser-Ubuntu.1804.Amd64.Open console.log runclient.py
1897635 runtime-extra-platforms PR 72124 net7.0-Browser-Release-wasm-Mono_Release-WasmTestOnBrowser-Ubuntu.1804.Amd64.Open console.log runclient.py
1897635 runtime-extra-platforms PR 72124 net7.0-Browser-Release-wasm-Mono_Release-WasmTestOnBrowser-Ubuntu.1804.Amd64.Open console.log runclient.py
1897635 runtime-extra-platforms PR 72124 net7.0-Android-Release-x86-Mono_Release-Ubuntu.1804.Amd64.Android.29.Open console.log test results runclient.py
1897635 runtime-extra-platforms PR 72124 net7.0-Android-Release-x86-Mono_Release-Ubuntu.1804.Amd64.Android.29.Open console.log test results runclient.py
1897635 runtime-extra-platforms PR 72124 net7.0-Android-Release-x86-Mono_Release-Ubuntu.1804.Amd64.Android.29.Open console.log test results runclient.py
1897635 runtime-extra-platforms PR 72124 net7.0-Android-Release-x86-Mono_Release-Ubuntu.1804.Amd64.Android.29.Open console.log test results runclient.py
1897635 runtime-extra-platforms PR 72124 net7.0-Android-Release-x86-Mono_Release-Ubuntu.1804.Amd64.Android.29.Open console.log test results runclient.py
1897635 runtime-extra-platforms PR 72124 net7.0-Android-Release-x86-Mono_Release-Ubuntu.1804.Amd64.Android.29.Open console.log test results runclient.py
1897635 runtime-extra-platforms PR 72124 net7.0-Android-Release-x86-Mono_Release-Ubuntu.1804.Amd64.Android.29.Open console.log test results runclient.py

Displaying 100 of 1291 results

Build Result Summary

Day Hit Count Week Hit Count Month Hit Count
5 59 97

Metadata

Metadata

Assignees

No one assigned

    Type

    No type

    Projects

    No projects

    Milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions