Update to 3.1.109

[dagood]

#1768

[dagood]

60dbbc6 Generate tarball smoke-test nuget.config

The bootstrap online-smoke-test was failing because it wasn't able to find the runtime packs in the public feeds: public feeds aren't included in support/tarball/smoke-testNuGet.Config, which is copied to the root of the tarball. To get the public feeds in, I have the tarball generation code create a copy of /nuget.config and use RemoveInternetSourcesFromNuGetConfig to include public feeds, but still remove internal feeds.

Fixes #1669. (Although we didn't have this exact scenario in mind at the time.)

[dagood]

To get a Microsoft-built SDK to compare against, I went here via URL manipulation:
https://maestro-prod.westus2.cloudapp.azure.com/560/https:%2F%2Fgithub.com%2Fdotnet%2Finstaller/65737/graph
-> https://dev.azure.com/dnceng/internal/_build/results?view=results&buildId=829159
-> https://dev.azure.com/dnceng/_apis/resources/Containers/5168270/BlobArtifacts?itemPath=BlobArtifacts%2Fdotnet-sdk-3.1.109-linux-x64.tar.gz

I got this: https://gist.github.com/dagood/5b523381aa7200fdfe5fd7e9635853f9

I scanned through and saw that the Microsoft-built SDK doesn't have a few DLLs that source-build does have:

--- /work/testing-smoke/builtCli
+++ /work/microsoft-built-dotnet-sdk-3.1.109.tar.gz
@@ -1781,17 +2487,13 @@
 ./shared/Microsoft.AspNetCore.App/3.1.9/Microsoft.Extensions.WebEncoders.dll
 ./shared/Microsoft.AspNetCore.App/3.1.9/Microsoft.JSInterop.dll
 ./shared/Microsoft.AspNetCore.App/3.1.9/Microsoft.Net.Http.Headers.dll
-./shared/Microsoft.AspNetCore.App/3.1.9/Microsoft.Win32.Registry.dll
 ./shared/Microsoft.AspNetCore.App/3.1.9/Microsoft.Win32.SystemEvents.dll
 ./shared/Microsoft.AspNetCore.App/3.1.9/System.Diagnostics.EventLog.dll
 ./shared/Microsoft.AspNetCore.App/3.1.9/System.Drawing.Common.dll
 ./shared/Microsoft.AspNetCore.App/3.1.9/System.IO.Pipelines.dll
-./shared/Microsoft.AspNetCore.App/3.1.9/System.Security.AccessControl.dll
-./shared/Microsoft.AspNetCore.App/3.1.9/System.Security.Cryptography.Cng.dll
 ./shared/Microsoft.AspNetCore.App/3.1.9/System.Security.Cryptography.Pkcs.dll
 ./shared/Microsoft.AspNetCore.App/3.1.9/System.Security.Cryptography.Xml.dll
 ./shared/Microsoft.AspNetCore.App/3.1.9/System.Security.Permissions.dll
-./shared/Microsoft.AspNetCore.App/3.1.9/System.Security.Principal.Windows.dll
 ./shared/Microsoft.AspNetCore.App/3.1.9/System.Windows.Extensions.dll

This lines up with the names mentioned in dotnet/aspnetcore#17973. But that's a 3.1.0 bug, and I'm not sure yet why we would be affected by it now.

[dagood]

ResolvePackageFileConflicts is kicking in while building the ASP.NET Core sharedfx. In the binlog for Microsoft.AspNetCore.App.Runtime.csproj:

Encountered conflict between
  'Platform:Microsoft.Win32.Registry.dll' and
  'CopyLocal:/work/packages/restored/microsoft.win32.registry/4.7.0/runtimes/unix/lib/netcoreapp3.1/Microsoft.Win32.Registry.dll'.
  Choosing 'CopyLocal:/work/packages/restored/microsoft.win32.registry/4.7.0/runtimes/unix/lib/netcoreapp3.1/Microsoft.Win32.Registry.dll'
  because file version '4.700.20.47203' is greater than '4.700.19.56404'.
Encountered conflict between
  'Platform:System.Security.AccessControl.dll' and
  'CopyLocal:/work/packages/restored/system.security.accesscontrol/4.7.0/lib/netstandard2.0/System.Security.AccessControl.dll'.
  Choosing 'CopyLocal:/work/packages/restored/system.security.accesscontrol/4.7.0/lib/netstandard2.0/System.Security.AccessControl.dll'
  because file version '4.700.20.47203' is greater than '4.700.19.56404'.
...

4.700.20.47203 is the version we just source-built in dotnet/corefx. The ASP.NET sharedfx and base sharedfx both carry a copy.
4.700.19.56404 is the version in Microsoft.NETCore.App.Ref/3.1.0.

I think that in theory, these DLLs should be excluded from the ASP.NET Core sharedfx because they're in the base sharedfx. Not sure yet why that happens in the Microsoft build but not source-build.

[dagood]

I checked 3.1.108, and the extra DLLs are there, too. So, this is not a regression and I don't think it should block the build.

---

I looked at a Microsoft build binlog for aspnet, and it looks like the only difference is that the Microsoft.Win32.Registry/4.7.0 it downloads (built back around 3.1.0) has the lower version of the DLL, 4.700.19.56404. The build uses Microsoft.NETCore.App.Ref/3.1.0's platform manifest, listing the old version, 4.700.19.56404. When they are an exact match, ResolvePackageFileConflicts chooses to use the Platform version and doesn't copy it into the sharedfx.

On our side, we rebuild Microsoft.Win32.Registry/4.7.0 with every single source-build, so the DLLs inside get a newer version corresponding with 3.1.9, 4.700.20.47203. The targeting pack isn't rebuilt, and it still has 4.700.19.56404 in the platform manifest. ASP.NET uses the platform manifest in the targeting pack, so it copies the DLLs from Microsoft.Win32.Registry/4.7.0 rather than using Platform's "old" DLLs.

This is yet another way that the Microsoft build reuse of old nupkgs is making a difference in source-build. My current granularity plan/proposal could help with this by testing that the file versions are the same between the old, reused Microsoft-built nupkg and the fresh bits that source-build produces. We already assume and rely on the DLLs being compatible, so making the file version the same so it's treated as part of the platform seems reasonable to me.

Filed #1779

[dagood]

Now seeing this in the final bootstrap build, in dotnet/sdk:

PrepareForReadyToRunCompilation.cs(362,117): error CS0433: The type 'Architecture' exists in both 'System.Runtime.InteropServices.RuntimeInformation, Version=4.0.2.0, Culture=neutral, PublicKeyToken=b03f5f7f11d50a3a' and 'mscorlib, Version=4.0.0.0, Culture=neutral, PublicKeyToken=b77a5c561934e089' [/bst/bootstrap_dir/final-sdk/src/sdk.afdde814dc8dedb325e63b39cf72249b43918a23/src/Tasks/Microsoft.NET.Build.Tasks/Microsoft.NET.Build.Tasks.csproj]

The reference being from:

/bst/bootstrap_dir/final-sdk/packages/restored/microsoft.bcl.asyncinterfaces/1.1.0/ref/net461/System.Runtime.InteropServices.RuntimeInformation.dll

I think this might be along the lines of this problem in dotnet/source-build-reference-packages#164 (comment), @dseefeld can you confirm?

I also reverted the changes to add System.Text.Json and related packages because these have not been tested in a tarball, yet. Microsoft.Bcl.AsyncInterfaces is currently causing problems with my tarball build locally. By removing these, it will show up as a prebuilt, but I'll re-address that with prebuilt removal later after I get the tarball building through with these other reference packages.

The build didn't hit that problem before, ~4 days ago. I think this is because of the floating SBRP branch reference, and some changes, in particular dotnet/source-build-reference-packages#165 which merged ~3 days ago. That PR put back the change that had the problems listed above. @dseefeld, what was done to resolve those issues in 5.0? (PR description is blank and there are no backlinks.)

If this isn't quickly reconcilable between the branches, I could make the 3.1 build use a pinned reference to the SBRP repo for now to keep using an older version.

[dseefeld]

@dseefeld Dan Seefeldt FTE, what was done to resolve those issues in 5.0? (PR description is blank and there are no backlinks.)

The issues that I was seeing with the updates to SBRP was as described in the arcade patch in #1777. This is a temporary update to Arcade until RC1, when Microsoft.Bcl.AsyncInterfaces will be re-added to the product. This won't work to fix the issue that you're seeing above, but I'm in favor of having a release/3.1 branch of SBRP just for servicing. I don't think there will be many changes on that branch and it will prevent against issues like this in the future.

[dagood]

Ah, right.

Sure, I'll go with pre-arcade-release/3.1 to start with and keep distinct from arcade-powered source-build "normal" branches, and set it back to just after the revert to see if that fixes it.

[dagood]

I found a deeper Msft/source-build diff with the package that I think is more clearly linked with what I ran into, filed at #1781. But, no need to fix that at this second--if the re-revert alone fixes things, that's fine.