[release/7.0.2xx] Runtime fails to generate native System.Private.CoreLib in source-build bootstrap

[lbussell]

I have been looking at this issue for a little bit and can't seem to make any progress, so I'm hoping someone here can point me in the right direction.

The runtime-portable build is failing in the 7.0.2xx source build bootstrap scenario only with the following error:

      Generating native image of System.Private.CoreLib for Linux.x64.Release. Logging to 
      /tarball/src/runtime/artifacts/source-build/self/src/dotnet.sh /tarball/src/runtime/artifacts/source-build/self/src/artifacts/bin/coreclr/Linux.x64.Release/crossgen2/crossgen2.dll -o:/tarball/src/runtime/artifacts/source-build/self/src/artifacts/bin/coreclr/Linux.x64.Release/System.Private.CoreLib.dll -r:/tarball/src/runtime/artifacts/source-build/self/src/artifacts/bin/coreclr/Linux.x64.Release/IL/*.dll --targetarch:x64 -O /tarball/src/runtime/artifacts/source-build/self/src/artifacts/bin/coreclr/Linux.x64.Release/IL/System.Private.CoreLib.dll --perfmap-format-version:1 --perfmap --perfmap-path:/tarball/src/runtime/artifacts/source-build/self/src/artifacts/bin/coreclr/Linux.x64.Release/
      Specify which project file to use because this '/tarball/src/runtime/artifacts/source-build/self/src/src/coreclr' contains more than one project file.
    /tarball/src/runtime/artifacts/source-build/self/src/src/coreclr/crossgen-corelib.proj(104,5): error MSB3073: The command "/tarball/src/runtime/artifacts/source-build/self/src/dotnet.sh /tarball/src/runtime/artifacts/source-build/self/src/artifacts/bin/coreclr/Linux.x64.Release/crossgen2/crossgen2.dll -o:/tarball/src/runtime/artifacts/source-build/self/src/artifacts/bin/coreclr/Linux.x64.Release/System.Private.CoreLib.dll -r:/tarball/src/runtime/artifacts/source-build/self/src/artifacts/bin/coreclr/Linux.x64.Release/IL/*.dll --targetarch:x64 -O /tarball/src/runtime/artifacts/source-build/self/src/artifacts/bin/coreclr/Linux.x64.Release/IL/System.Private.CoreLib.dll --perfmap-format-version:1 --perfmap --perfmap-path:/tarball/src/runtime/artifacts/source-build/self/src/artifacts/bin/coreclr/Linux.x64.Release/" exited with code 1.
    
    Build FAILED.
    
    /tarball/src/runtime/artifacts/source-build/self/src/src/coreclr/crossgen-corelib.proj(104,5): error MSB3073: The command "/tarball/src/runtime/artifacts/source-build/self/src/dotnet.sh /tarball/src/runtime/artifacts/source-build/self/src/artifacts/bin/coreclr/Linux.x64.Release/crossgen2/crossgen2.dll -o:/tarball/src/runtime/artifacts/source-build/self/src/artifacts/bin/coreclr/Linux.x64.Release/System.Private.CoreLib.dll -r:/tarball/src/runtime/artifacts/source-build/self/src/artifacts/bin/coreclr/Linux.x64.Release/IL/*.dll --targetarch:x64 -O /tarball/src/runtime/artifacts/source-build/self/src/artifacts/bin/coreclr/Linux.x64.Release/IL/System.Private.CoreLib.dll --perfmap-format-version:1 --perfmap --perfmap-path:/tarball/src/runtime/artifacts/source-build/self/src/artifacts/bin/coreclr/Linux.x64.Release/" exited with code 1.
        0 Warning(s)
        1 Error(s)

Inside src/coreclr are multiple project files, crossgen-corelib.proj, runtime.proj, and runtime-prereqs.proj. I checked that these are also present in the runtime that's in .NET 7.0.1xx.
I also checked that the error from the .NET CLI isn't new to .NET 7.0.2xx. I'm not sure where else to look. Maybe @ViktorHofer knows who to tag?

Tagging subscribers to this area: @hoyosjs
See info in area-owners.md if you want to be subscribed.

Issue Details

---

I have been looking at this issue for a little bit and can't seem to make any progress, so I'm hoping someone here can point me in the right direction.

The runtime-portable build is failing in the 7.0.2xx source build bootstrap scenario only with the following error:

      Generating native image of System.Private.CoreLib for Linux.x64.Release. Logging to 
      /tarball/src/runtime/artifacts/source-build/self/src/dotnet.sh /tarball/src/runtime/artifacts/source-build/self/src/artifacts/bin/coreclr/Linux.x64.Release/crossgen2/crossgen2.dll -o:/tarball/src/runtime/artifacts/source-build/self/src/artifacts/bin/coreclr/Linux.x64.Release/System.Private.CoreLib.dll -r:/tarball/src/runtime/artifacts/source-build/self/src/artifacts/bin/coreclr/Linux.x64.Release/IL/*.dll --targetarch:x64 -O /tarball/src/runtime/artifacts/source-build/self/src/artifacts/bin/coreclr/Linux.x64.Release/IL/System.Private.CoreLib.dll --perfmap-format-version:1 --perfmap --perfmap-path:/tarball/src/runtime/artifacts/source-build/self/src/artifacts/bin/coreclr/Linux.x64.Release/
      Specify which project file to use because this '/tarball/src/runtime/artifacts/source-build/self/src/src/coreclr' contains more than one project file.
    /tarball/src/runtime/artifacts/source-build/self/src/src/coreclr/crossgen-corelib.proj(104,5): error MSB3073: The command "/tarball/src/runtime/artifacts/source-build/self/src/dotnet.sh /tarball/src/runtime/artifacts/source-build/self/src/artifacts/bin/coreclr/Linux.x64.Release/crossgen2/crossgen2.dll -o:/tarball/src/runtime/artifacts/source-build/self/src/artifacts/bin/coreclr/Linux.x64.Release/System.Private.CoreLib.dll -r:/tarball/src/runtime/artifacts/source-build/self/src/artifacts/bin/coreclr/Linux.x64.Release/IL/*.dll --targetarch:x64 -O /tarball/src/runtime/artifacts/source-build/self/src/artifacts/bin/coreclr/Linux.x64.Release/IL/System.Private.CoreLib.dll --perfmap-format-version:1 --perfmap --perfmap-path:/tarball/src/runtime/artifacts/source-build/self/src/artifacts/bin/coreclr/Linux.x64.Release/" exited with code 1.
    
    Build FAILED.
    
    /tarball/src/runtime/artifacts/source-build/self/src/src/coreclr/crossgen-corelib.proj(104,5): error MSB3073: The command "/tarball/src/runtime/artifacts/source-build/self/src/dotnet.sh /tarball/src/runtime/artifacts/source-build/self/src/artifacts/bin/coreclr/Linux.x64.Release/crossgen2/crossgen2.dll -o:/tarball/src/runtime/artifacts/source-build/self/src/artifacts/bin/coreclr/Linux.x64.Release/System.Private.CoreLib.dll -r:/tarball/src/runtime/artifacts/source-build/self/src/artifacts/bin/coreclr/Linux.x64.Release/IL/*.dll --targetarch:x64 -O /tarball/src/runtime/artifacts/source-build/self/src/artifacts/bin/coreclr/Linux.x64.Release/IL/System.Private.CoreLib.dll --perfmap-format-version:1 --perfmap --perfmap-path:/tarball/src/runtime/artifacts/source-build/self/src/artifacts/bin/coreclr/Linux.x64.Release/" exited with code 1.
        0 Warning(s)
        1 Error(s)

Inside src/coreclr are multiple project files, crossgen-corelib.proj, runtime.proj, and runtime-prereqs.proj. I checked that these are also present in the runtime that's in .NET 7.0.1xx.
I also checked that the error from the .NET CLI isn't new to .NET 7.0.2xx. I'm not sure where else to look. Maybe @ViktorHofer knows who to tag?

Author:	lbussell
Assignees:	-
Labels:	area-Infrastructure-coreclr
Milestone:	-

[ViktorHofer]

cc @MichaelSimons @jkotas

[lbussell]

Also, here is a link to the failure in CI: https://dev.azure.com/dnceng/internal/_build/results?buildId=2048260&view=logs&j=05a957c8-6537-5d6d-69e3-d72786d35c31&t=8f2f8c4f-4a49-5a74-0521-8049d6bde132&l=25268

And the binlog:
runtime.zip

[jkotas]

This is one of the first places where the runtime that was just built runs an app. This failure likely means that the runtime that was just built is badly broken.

What are the steps to reproduce this locally?

[lbussell]

This can be reproduced in a source-build bootstrap context. The fastest way is to

1. Download dotnet-sdk-source-7.0.200.tar.gz from here
2. Download dotnet-sdk-7.0.200-fedora.36-x64.tar.gz, Private.SourceBuilt.Artifacts.7.0.200.tar.gz from here.
3. Extract the source tarball
4. Extract the SDK to <extracted source tarball>/.dotnet
5. Copy the Private.SourceBuilt.Artifacts archive to <extracted source tarball>/packages/archive
6. In the container mcr.microsoft.com/dotnet-buildtools/prereqs:fedora-36, run ./build.sh --with-sdk .dotnet/ from the root of the extracted source tarball.

[jkotas]

When I re-run the failing command, I see that it is trying to download .NET SDK 7.0.200 and failing:

[root@a1d109dfbb94 ~]# /root/src/runtime/artifacts/source-build/self/src/dotnet.sh /root/src/runtime/artifacts/source-build/self/src/artifacts/bin/coreclr/Linux.x64.Release/crossgen2/crossgen2.dll -o:/root/src/runtime/artifacts/source-build/self/src/artifacts/bin/coreclr/Linux.x64.Release/System.Private.CoreLib.dll -r:/root/src/runtime/artifacts/source-build/self/src/artifacts/bin/coreclr/Linux.x64.Release/IL/*.dll --targetarch:x64 -O /root/src/runtime/artifacts/source-build/self/src/artifacts/bin/coreclr/Linux.x64.Release/IL/System.Private.CoreLib.dll --perfmap-format-version:1 --perfmap --perfmap-path:/root/src/runtime/artifacts/source-build/self/src/artifacts/bin/coreclr/Linux.x64.Release/
Downloading 'https://dotnet.microsoft.com/download/dotnet/scripts/v1/dotnet-install.sh'
Attempting to install dotnet from public_location.
dotnet-install: Note that the intended use of this script is for Continuous Integration (CI) scenarios, where:
dotnet-install: - The SDK needs to be installed without user interaction and without admin rights.
dotnet-install: - The SDK installation doesn't need to persist across multiple CI runs.
dotnet-install: To set up a development environment or to run apps, use installers rather than this script. Visit https://dotnet.microsoft.com/download to get the installer.

dotnet-install: Attempting to download using primary link https://dotnetcli.azureedge.net/dotnet/Sdk/7.0.200/dotnet-sdk-7.0.200-linux-x64.tar.gz
curl: (22) The requested URL returned error: 404
dotnet-install: The resource at primary link 'https://dotnetcli.azureedge.net/dotnet/Sdk/7.0.200/dotnet-sdk-7.0.200-linux-x64.tar.gz' is not available.
dotnet-install: Attempting to download using primary link https://dotnetbuilds.azureedge.net/public/Sdk/7.0[root@a1d109dfbb94 ~]# cd ..
curl: (22) The requested URL returned error: 404
dotnet-install: The resource at primary link 'https://dotnetbuilds.azureedge.net/public/Sdk/7.0.200/dotnet-sdk-7.0.200-linux-x64.tar.gz' is not available.
dotnet_install: Error: Could not find `.NET Core SDK` with version = 7.0.200
dotnet_install: Error: Refer to: https://aka.ms/dotnet-os-lifecycle for information on .NET Core support
Failed to install dotnet from public_location.
Attempting to install dotnet from dotnetbuilds.
dotnet-install: Note that the intended use of this script is for Continuous Integration (CI) scenarios, where:
dotnet-install: - The SDK needs to be installed without user interaction and without admin rights.
dotnet-install: - The SDK installation doesn't need to persist across multiple CI runs.
dotnet-install: To set up a development environment or to run apps, use installers rather than this script. Visit https://dotnet.microsoft.com/download to get the installer.

dotnet-install: Attempting to download using primary link https://dotnetbuilds.azureedge.net/public/Sdk/7.0.200/dotnet-sdk-7.0.200-linux-x64.tar.gz
curl: (22) The requested URL returned error: 404
dotnet-install: The resource at primary link 'https://dotnetbuilds.azureedge.net/public/Sdk/7.0.200/dotnet-sdk-7.0.200-linux-x64.tar.gz' is not available.
dotnet_install: Error: Could not find `.NET Core SDK` with version = 7.0.200
dotnet_install: Error: Refer to: https://aka.ms/dotnet-os-lifecycle for information on .NET Core support
Failed to install dotnet from dotnetbuilds.
Failed to install dotnet SDK from any of the specified locations.
[root@a1d109dfbb94 ~]#

What is the runtime that you expect /root/src/runtime/artifacts/source-build/self/src/dotnet.sh script to use?

[lbussell]

It should be using the SDK from .dotnet/, which I believe it does when run from build.sh. I've seen the same thing on my end when trying to run just the failing command by itself . my guess is running the command alone like that skips some of the scaffolding needed to run the build correctly. @crummel or @MichaelSimons any debugging tips here?

[crummel]

These variables are what we set to get everything using the correct SDK, I'm not sure which in particular is the right one for runtime but just setting all of them should work fine.

[jkotas]

I have set these variables to point to .dotnet directory created by step "4. Extract the SDK to /.dotnet" above.

The problem is that the path to crossgen2 that this is trying to execute is .../src/runtime/artifacts/source-build/self/src/artifacts/bin/coreclr/Linux.x64.Release/crossgen2/crossgen2.dll . This file does not exist. The actual path to this file is .../src/runtime/artifacts/source-build/self/src/artifacts/bin/coreclr/Linux.x64.Release/crossgen2/linux-x64/crossgen2.dll. Notice extra linux-x64.

Is looks like there is a breaking change in the .NET SDK 7.0.xxx that added linux-x64 to the build output path. What's the right way to disable the new behavior? The crossgen2 build output path is RID specific already, it does not make sense to nest it further.

cc @dsplaisted

[jkotas]

The problem was introduced by changes in sdk\7.0.200\Sdks\Microsoft.NET.Sdk\targets\Microsoft.NET.RuntimeIdentifierInference.targets . I have confirmed that the issue goes away if I copy Microsoft.NET.RuntimeIdentifierInference.targets from .NET SDK 7.0.100. @lbussell If you need to unblock the source build, you should be able to add temporary source build patch that reverts Microsoft.NET.RuntimeIdentifierInference.targets to .NET SDK 7.0.100 version.

@nagilson Could you please investigate whether the behavior change is a bug in .NET SDK, or whether this behavior is by design and we need to workaround it in dotnet/runtime build?

[nagilson]

Thanks @jkotas for the ping and investigation. It was probably this PR: #28484

Before I can give a definitive answer, I need a bit more information ( see Questions). But if I understand the situation correctly, this is by design. We can certainly talk about if it should or shouldn't be by design if 'ya want.

Workarounds

First: @lbussell as a work-around, setting UseCurrentRuntimeIdentifier = false should disable this behavior. Please let me know how that goes.

Another option may be: AppendRuntimeIdentifierToOutputPath = false which you can set, but that will just change the pathing; you'll still get the RID, so it's a hacky workaround.

Explanation

I'm guessing this is caused by the following: There is a point in time in which crossgen2 uses RuntimeIdentifiers but does not set a RuntimeIdentifier. If crossgen2 depends on any of the following: SelfContained, or any Publish* property like PublishSingleFile, the SDK detects this and gives it a RuntimeIdentifier. Now it has a RuntimeIdentifiers and RuntimeIdentifier.

Questions

Is the above information correct, may someone from crossgen comment on the claim that it sets RuntimeIdentifiers but no RuntimeIdentifier? And do you use one of the following SelfContained or Publish* properties? If so, I'd imagine you had some special logic before to use them, because those properties require a RuntimeIdentifier to be set?