MicroBenchmarks.dll fails with "The specified network password is not correct."

[kunalspathak]

We are seeing failures in our superpmi collection as soon as we start executing MicroBenchmarks. Wondering if perf lab is also seeing these failures and what is the fix? This is blocking our ability to do superpmi collection on benchmarks.

[18:22:58] C:\h\w\97FE0875\w\9B61093D\u\performance>C:\h\w\97FE0875\w\9B61093D\u\performance\tools\dotnet\x64\dotnet C:\h\w\97FE0875\w\9B61093D\u\performance\artifacts\MicroBenchmarks.dll  --filter "*" --corerun C:\h\w\97FE0875\p\superpmi\CoreRun.exe --partition-count 30 --partition-index 11 --envVars COMPlus_JitName:superpmi-shim-collector.dll  COMPlus_ZapDisable:1  COMPlus_ReadyToRun:0 --iterationCount 1 --warmupCount 0 --invocationCount 1 --unrollFactor 1 --strategy ColdStart 
[18:22:58] Unhandled exception. System.TypeInitializationException: The type initializer for 'System.Net.Security.Tests.SslStreamTests' threw an exception.
[18:22:58]  ---> Internal.Cryptography.CryptoThrowHelper+WindowsCryptographicException: The specified network password is not correct.
[18:22:58]    at System.Security.Cryptography.X509Certificates.CertificatePal.FilterPFXStore(ReadOnlySpan`1 rawData, SafePasswordHandle password, PfxCertStoreFlags pfxCertStoreFlags)
[18:22:58]    at System.Security.Cryptography.X509Certificates.CertificatePal.FromBlobOrFile(ReadOnlySpan`1 rawData, String fileName, SafePasswordHandle password, X509KeyStorageFlags keyStorageFlags)
[18:22:58]    at System.Security.Cryptography.X509Certificates.X509Certificate..ctor(Byte[] rawData, String password, X509KeyStorageFlags keyStorageFlags)
[18:22:58]    at System.Net.Test.Common.Configuration.Certificates.GetCertificate(String certificateFileName) in C:\h\w\97FE0875\w\9B61093D\u\performance\src\benchmarks\micro\libraries\System.Net.Http\Configuration.Certificates.cs:line 27
[18:22:58]    at System.Net.Security.Tests.SslStreamTests..cctor() in C:\h\w\97FE0875\w\9B61093D\u\performance\src\benchmarks\micro\libraries\System.Net.Security\SslStreamTests.cs:line 24
[18:22:58]    --- End of inner exception stack trace ---
[18:22:58]    at System.Net.Security.Tests.SslStreamTests.TlsProtocols()+MoveNext() in C:\h\w\97FE0875\w\9B61093D\u\performance\src\benchmarks\micro\libraries\System.Net.Security\SslStreamTests.Protocols.cs:line 22
[18:22:58]    at System.Collections.Generic.LargeArrayBuilder`1.AddRange(IEnumerable`1 items)
[18:22:58]    at System.Collections.Generic.EnumerableHelpers.ToArray[T](IEnumerable`1 source)
[18:22:58]    at BenchmarkDotNet.Running.BenchmarkConverter.GetValidValuesForParamsSource(Type parentType, String sourceName)
[18:22:58]    at BenchmarkDotNet.Running.BenchmarkConverter.GetArgumentsDefinitions(MethodInfo benchmark, Type target, SummaryStyle summaryStyle)+MoveNext()
[18:22:58]    at System.Collections.Generic.LargeArrayBuilder`1.AddRange(IEnumerable`1 items)
[18:22:58]    at System.Collections.Generic.EnumerableHelpers.ToArray[T](IEnumerable`1 source)
[18:22:58]    at BenchmarkDotNet.Running.BenchmarkConverter.MethodsToBenchmarksWithFullConfig(Type type, MethodInfo[] benchmarkMethods, IConfig config)
[18:22:58]    at System.Linq.Enumerable.SelectListIterator`2.MoveNext()
[18:22:58]    at System.Linq.Enumerable.WhereEnumerableIterator`1.ToArray()
[18:22:58]    at BenchmarkDotNet.Running.BenchmarkSwitcher.RunWithDirtyAssemblyResolveHelper(String[] args, IConfig config, Boolean askUserForInput)
[18:22:58]    at BenchmarkDotNet.Running.BenchmarkSwitcher.Run(String[] args, IConfig config)
[18:22:58]    at MicroBenchmarks.Program.Main(String[] args) in C:\h\w\97FE0875\w\9B61093D\u\performance\src\benchmarks\micro\Program.cs:line 47

I am not sure if #1750 is related but we started seeing the failures around that time.

[kunalspathak]

@DrewScoggins @wfurt

[kunalspathak]

@BruceForstall FYI.

[BruceForstall]

e.g., here's a link to a failed run.

https://dev.azure.com/dnceng/internal/_build/results?buildId=1783706&view=logs&j=780b3331-d7b2-5848-543a-ef5d1bbff1c9&t=efe49bc7-a53a-5503-4fec-d1c38bd80ad8

It's in the "internal" pipelines group, so you can't just click to find the console log, you need some Kusto magic, but I wanted to drop a link here anyway.

[BruceForstall]

@DrewScoggins @wfurt Note in particular:

Internal.Cryptography.CryptoThrowHelper+WindowsCryptographicException: The specified network password is not correct.

Is this being seen in the normal performance runs? Was anything done to ensure this passes in normal perf runs? (We might be running our superpmi collection runs on a different set of Helix machines, for instance)

[wfurt]

What platform this is running on? I did run on my Linux box and I don't see any failure.
From the stack above, it looks like it is failing to load ECDH key.
I'm wondering if the error is lame and the root case is in ECDH support...
cc: @rzikm

#1750 is more than year old. It would be surprising if that is the culprit.

[kunalspathak]

What platform this is running on?

We are seeing failures on windows. Linux runs are fine.

#1750 is more than year old.

That's true...I misread the year :)

[wfurt]

I just did run on my Win11 and everything works ok. So I'm reasonably confident the password is not wrong.
cc: @bartonjs for any more thoughts as it seems to fail load certificate from PFX.

[bartonjs]

If it's being run on Windows make sure it's being run in a context where the user has a loaded user profile.

[kunalspathak]

We didn't change anything in the way it is run.

[BruceForstall]

We didn't change anything in the way it is run.

To expand on this, our pipeline has been running the same way for a long time, and we haven't recently changed anything about how we run the pipeline. Changes we'd worry about would be (1) changes to the dotnet/performance repo, (2) changes to Helix/AzDO infrastructure, (3) changes to the daily dotnet SDK build (i.e., the product).

[wfurt]

That would leave #2405 as suspect IMHO. We can try to revert it and give it try. However, it did pass all the CI tests.
It is not clear to me if superpmi was passing at that time or not. And why that would be any different from the PR tests.
cc: @rzikm.

[BruceForstall]

I'm not sure if this matters, but note that the superpmi collect pipeline runs in the "internal" (not "public") AzDO TeamProject, which has a separate set of Helix machines.

[wfurt]

I assume performance CI does not run from Helix. Maybe @MattGal may have some insight about #2454 (comment).

[MattGal]

I assume performance CI does not run from Helix. Maybe @MattGal may have some insight about #2454 (comment).

• To @jsbarton's point, I can confirm that anything running on windows.10.amd64.x86.rt is running in an interactive user session with a local, randomly generated account name "runner", UAC disabled.
• The image used on this queue has not changed since 5/11/2022 (though it is rolling out changes today). While this happened around the same time, there's runs using the image rolled out on 5/11 (WorkItems| where JobName == "96330f1e-bad9-4c6f-a2bd-56288b632df2") that do not show this problem.
• I suggest investigating to see if you can get a repro using a VM from https://ms.portal.azure.com/#@microsoft.onmicrosoft.com/resource/subscriptions/d60c2e3b-00e6-48f3-a069-542c17981e1f/resourceGroups/helix-repro-vms/providers/Microsoft.DevTestLab/labs/helix-repro-vms/overview. If you find we have something wrong with the machines in question, file an issue in dotnet/arcade.
• In the short term to differentiate if this is something in helix or something in the work items themselves, I'd suggest rerunning the last good commit (dc7728f1f2ef1d1a78286e9ae6c5d7dfe25e673c) and see if it repros; if it doesn't the problem is definitely in the 50+ commits between the last good and first bad runs.

[kunalspathak]

I'd suggest rerunning the last good commit (dc7728f1f2ef1d1a78286e9ae6c5d7dfe25e673c)

That's a good point. Started the run.