"sdk" project takes indefinite time to complete, likely stuck.

[pauldotknopf]

It has taken 8 hours so far. I think something is wrong.

Some background though, I'm using some custom patches to build .NET Core within the Yocto environment.

qmlnet@59fb4a3

The commits sit all on top of the latest 2.1.401 revs.

All the native cmake components compile correctly (coreclr/corefx/core-setup).

I'm using this command to run the build.

export SOURCE_BUILD_SKIP_SUBMODULE_CHECK=1
# This is our Yocto cross sysroot
export ROOTFS_DIR=${STAGING_DIR_HOST}
# This where our toolchain.cmake and tryrun.cmake exist that contain the build needed cross-compilation cmake projects.
export CONFIG_DIR=${CMAKE_CONFIG_DIR}
./build.sh /p:Platform=x64 /p:Configuration=Release /p:SkipGenerateRootFs=true

The commands all run, up until the sdk project, which seemingly hangs indefinitely.

Refer to logs.tar.gz to see all the logs generated by the source-build repo.

Refer to sdk-logs.tar.gz for the verbose logs generated by the sdk project. Note, I had to ctrl-c kill the build.

I've run .build.sh with /p:RootRepo=sdk so that you can see the command output and environment variables.

Tools are already initialized
+ /home/pknopf/git/x3/abrarecipes/build/tmp/work/corei7-64-poky-linux/dotnet-source-build/2.1.401-r0/git/Tools/dotnetcli/dotnet /home/pknopf/git/x3/abrarecipes/build/tmp/work/corei7-64-poky-linux/dotnet-source-build/2.1.401-r0/git/Tools/dotnetcli/sdk/2.1.301/MSBuild.dll /home/pknopf/git/x3/abrarecipes/build/tmp/work/corei7-64-poky-linux/dotnet-source-build/2.1.401-r0/git/build.proj /flp:v=diag /clp:v=m /p:Platform=x64 /p:Configuration=Release /p:SkipGenerateRootFs=true /p:RootRepo=sdk /p:SkipRepoReferences=true
Microsoft (R) Build Engine version 15.7.179.6572 for .NET Core
Copyright (C) Microsoft Corporation. All rights reserved.

  sdk is using versions /home/pknopf/git/x3/abrarecipes/build/tmp/work/corei7-64-poky-linux/dotnet-source-build/2.1.401-r0/git/bin/obj/x64/Release/PackageVersions.props
  <?xml version="1.0" encoding="utf-8"?>
  <Project ToolsVersion="14.0" xmlns="http://schemas.microsoft.com/developer/msbuild/2003">
  <PropertyGroup>
  <FSharpNETSdkPackageVersion>1.0.4-bundled-0100</FSharpNETSdkPackageVersion>
  <MicrosoftApplicationInsightsPackageVersion>2.0.0</MicrosoftApplicationInsightsPackageVersion>
  <MicrosoftBuildPackageVersion>15.7.179</MicrosoftBuildPackageVersion>
  <MicrosoftBuildFrameworkPackageVersion>15.7.179</MicrosoftBuildFrameworkPackageVersion>
  <MicrosoftBuildNuGetSdkResolverPackageVersion>4.8.0-preview3.5156</MicrosoftBuildNuGetSdkResolverPackageVersion>
  <MicrosoftBuildRuntimePackageVersion>15.7.179</MicrosoftBuildRuntimePackageVersion>
  <MicrosoftBuildTasksCorePackageVersion>15.7.179</MicrosoftBuildTasksCorePackageVersion>
  <MicrosoftBuildUtilitiesCorePackageVersion>15.7.179</MicrosoftBuildUtilitiesCorePackageVersion>
  <MicrosoftCodeAnalysisCommonPackageVersion>2.8.0</MicrosoftCodeAnalysisCommonPackageVersion>
  <MicrosoftCodeAnalysisCompilersPackageVersion>2.8.0</MicrosoftCodeAnalysisCompilersPackageVersion>
  <MicrosoftCodeAnalysisCSharpPackageVersion>2.8.0</MicrosoftCodeAnalysisCSharpPackageVersion>
  <MicrosoftCodeAnalysisVisualBasicPackageVersion>2.8.0</MicrosoftCodeAnalysisVisualBasicPackageVersion>
  <MicrosoftDotNetCliCommandLinePackageVersion>0.1.1-dev</MicrosoftDotNetCliCommandLinePackageVersion>
  <MicrosoftDotNetCommonItemTemplatesPackageVersion>1.0.2-beta2-20170727-301</MicrosoftDotNetCommonItemTemplatesPackageVersion>
  <MicrosoftDotNetCommonProjectTemplates1XPackageVersion>1.0.2-beta2-20170727-301</MicrosoftDotNetCommonProjectTemplates1XPackageVersion>
  <MicrosoftDotNetCommonProjectTemplates20PackageVersion>1.0.2-beta2-20170727-301</MicrosoftDotNetCommonProjectTemplates20PackageVersion>
  <MicrosoftDotNetCommonProjectTemplates21PackageVersion>1.0.2-beta2-20170727-301</MicrosoftDotNetCommonProjectTemplates21PackageVersion>
  <MicrosoftDotNetPlatformAbstractionsPackageVersion>2.1.3</MicrosoftDotNetPlatformAbstractionsPackageVersion>
  <MicrosoftDotNetTestProjectTemplates1XPackageVersion>1.0.2-beta2-20170727-301</MicrosoftDotNetTestProjectTemplates1XPackageVersion>
  <MicrosoftDotNetTestProjectTemplates20PackageVersion>1.0.2-beta2-20170727-301</MicrosoftDotNetTestProjectTemplates20PackageVersion>
  <MicrosoftDotNetTestProjectTemplates21PackageVersion>1.0.2-beta2-20170727-301</MicrosoftDotNetTestProjectTemplates21PackageVersion>
  <MicrosoftDotNetWebSpaProjectTemplatesPackageVersion>1.0.417</MicrosoftDotNetWebSpaProjectTemplatesPackageVersion>
  <MicrosoftExtensionsCommandLineUtilsPackageVersion>1.0.1</MicrosoftExtensionsCommandLineUtilsPackageVersion>
  <MicrosoftExtensionsDependencyModelPackageVersion>2.1.3</MicrosoftExtensionsDependencyModelPackageVersion>
  <MicrosoftNETSdkPublishPackageVersion>2.1.402</MicrosoftNETSdkPublishPackageVersion>
  <MicrosoftNETSdkWebPackageVersion>2.1.402</MicrosoftNETSdkWebPackageVersion>
  <MicrosoftNETSdkWebProjectSystemPackageVersion>2.1.402</MicrosoftNETSdkWebProjectSystemPackageVersion>
  <MicrosoftNETCoreAppPackageVersion>2.1.3</MicrosoftNETCoreAppPackageVersion>
  <MicrosoftNETCoreCompilersPackageVersion>2.8.0</MicrosoftNETCoreCompilersPackageVersion>
  <MicrosoftNETCoreDotNetAppHostPackageVersion>2.1.3</MicrosoftNETCoreDotNetAppHostPackageVersion>
  <MicrosoftNETCoreDotNetHostPackageVersion>2.1.3</MicrosoftNETCoreDotNetHostPackageVersion>
  <MicrosoftNETCoreDotNetHostPolicyPackageVersion>2.1.3</MicrosoftNETCoreDotNetHostPolicyPackageVersion>
  <MicrosoftNETCoreDotNetHostResolverPackageVersion>2.1.3</MicrosoftNETCoreDotNetHostResolverPackageVersion>
  <MicrosoftNETCorePlatformsPackageVersion>2.1.1</MicrosoftNETCorePlatformsPackageVersion>
  <MicrosoftPrivateCoreFxNETCoreAppPackageVersion>4.5.0-servicing-26725-05</MicrosoftPrivateCoreFxNETCoreAppPackageVersion>
  <MicrosoftTemplateEngineAbstractionsPackageVersion>1.0.2-beta2-20170727-301</MicrosoftTemplateEngineAbstractionsPackageVersion>
  <MicrosoftTemplateEngineCliPackageVersion>1.0.2-beta2-20170727-301</MicrosoftTemplateEngineCliPackageVersion>
  <MicrosoftTemplateEngineCorePackageVersion>1.0.2-beta2-20170727-301</MicrosoftTemplateEngineCorePackageVersion>
  <MicrosoftTemplateEngineCoreContractsPackageVersion>1.0.2-beta2-20170727-301</MicrosoftTemplateEngineCoreContractsPackageVersion>
  <MicrosoftTemplateEngineEdgePackageVersion>1.0.2-beta2-20170727-301</MicrosoftTemplateEngineEdgePackageVersion>
  <MicrosoftTemplateEngineIDEPackageVersion>1.0.2-beta2-20170727-301</MicrosoftTemplateEngineIDEPackageVersion>
  <MicrosoftTemplateEngineMocksPackageVersion>1.0.2-beta2-20170727-301</MicrosoftTemplateEngineMocksPackageVersion>
  <MicrosoftTemplateEngineOrchestratorRunnableProjectsPackageVersion>1.0.2-beta2-20170727-301</MicrosoftTemplateEngineOrchestratorRunnableProjectsPackageVersion>
  <MicrosoftTemplateEngineUtilsPackageVersion>1.0.2-beta2-20170727-301</MicrosoftTemplateEngineUtilsPackageVersion>
  <NETStandardLibraryPackageVersion>2.0.2-servicing-26321-0</NETStandardLibraryPackageVersion>
  <NewtonsoftJsonPackageVersion>9.0.1</NewtonsoftJsonPackageVersion>
  <NuGetBuildTasksPackageVersion>4.8.0-preview3.5156</NuGetBuildTasksPackageVersion>
  <NuGetBuildTasksPackPackageVersion>4.8.0-preview3.5156</NuGetBuildTasksPackPackageVersion>
  <NuGetCommandLineXPlatPackageVersion>4.8.0-preview3.5156</NuGetCommandLineXPlatPackageVersion>
  <NuGetCommandsPackageVersion>4.8.0-preview3.5156</NuGetCommandsPackageVersion>
  <NuGetCommonPackageVersion>4.8.0-preview3.5156</NuGetCommonPackageVersion>
  <NuGetConfigurationPackageVersion>4.8.0-preview3.5156</NuGetConfigurationPackageVersion>
  <NuGetCredentialsPackageVersion>4.8.0-preview3.5156</NuGetCredentialsPackageVersion>
  <NuGetDependencyResolverCorePackageVersion>4.8.0-preview3.5156</NuGetDependencyResolverCorePackageVersion>
  <NuGetFrameworksPackageVersion>4.8.0-preview3.5156</NuGetFrameworksPackageVersion>
  <NuGetLibraryModelPackageVersion>4.8.0-preview3.5156</NuGetLibraryModelPackageVersion>
  <NuGetPackagingPackageVersion>4.8.0-preview3.5156</NuGetPackagingPackageVersion>
  <NuGetPackagingCorePackageVersion>4.8.0-preview3.5156</NuGetPackagingCorePackageVersion>
  <NuGetProjectModelPackageVersion>4.8.0-preview3.5156</NuGetProjectModelPackageVersion>
  <NuGetProtocolPackageVersion>4.8.0-preview3.5156</NuGetProtocolPackageVersion>
  <NuGetResolverPackageVersion>4.8.0-preview3.5156</NuGetResolverPackageVersion>
  <NuGetVersioningPackageVersion>4.8.0-preview3.5156</NuGetVersioningPackageVersion>
  <RoslynToolsBuildTasksPackageVersion>1.0.0-dev</RoslynToolsBuildTasksPackageVersion>
  <RoslynToolsRepoToolsetPackageVersion>1.0.0-dev</RoslynToolsRepoToolsetPackageVersion>
  <RuntimeUbuntu1604X64MicrosoftNETCoreAppPackageVersion>2.1.3</RuntimeUbuntu1604X64MicrosoftNETCoreAppPackageVersion>
  <RuntimeUbuntu1604X64MicrosoftNETCoreDotNetAppHostPackageVersion>2.1.3</RuntimeUbuntu1604X64MicrosoftNETCoreDotNetAppHostPackageVersion>
  <RuntimeUbuntu1604X64MicrosoftNETCoreDotNetHostPackageVersion>2.1.3</RuntimeUbuntu1604X64MicrosoftNETCoreDotNetHostPackageVersion>
  <RuntimeUbuntu1604X64MicrosoftNETCoreDotNetHostPolicyPackageVersion>2.1.3</RuntimeUbuntu1604X64MicrosoftNETCoreDotNetHostPolicyPackageVersion>
  <RuntimeUbuntu1604X64MicrosoftNETCoreDotNetHostResolverPackageVersion>2.1.3</RuntimeUbuntu1604X64MicrosoftNETCoreDotNetHostResolverPackageVersion>
  <RuntimeUbuntu1604X64MicrosoftPrivateCoreFxNETCoreAppPackageVersion>4.5.0-servicing-26725-05</RuntimeUbuntu1604X64MicrosoftPrivateCoreFxNETCoreAppPackageVersion>
  <XliffTasksPackageVersion>0.2.0-beta-000048</XliffTasksPackageVersion>
  </PropertyGroup>
  </Project>
  [19:35:52.90] Building 'sdk'
  Running command:
    /home/pknopf/git/x3/abrarecipes/build/tmp/work/corei7-64-poky-linux/dotnet-source-build/2.1.401-r0/git/src/sdk/build.sh --pack --verbosity diagnostic --configuration Release /p:BUILD_BUILDNUMBER=20180802.3 /p:PB_PackageVersionPropsUrl=file:///home/pknopf/git/x3/abrarecipes/build/tmp/work/corei7-64-poky-linux/dotnet-source-build/2.1.401-r0/git/bin/obj/x64/Release/PackageVersions.props  /p:DotNetRestoreSourcePropsPath=/home/pknopf/git/x3/abrarecipes/build/tmp/work/corei7-64-poky-linux/dotnet-source-build/2.1.401-r0/git/bin/obj/x64/Release/RestoreSources.props /p:DotNetOutputBlobFeedDir=/home/pknopf/git/x3/abrarecipes/build/tmp/work/corei7-64-poky-linux/dotnet-source-build/2.1.401-r0/git/bin/obj/x64/Release/blob-feed/
    Log: /home/pknopf/git/x3/abrarecipes/build/tmp/work/corei7-64-poky-linux/dotnet-source-build/2.1.401-r0/git/bin/logs/sdk.log
    With Enivornment Varibles:
      DOTNET_RUNTIME_ID=ubuntu.16.04-x64
      DOTNET_TOOL_DIR=/home/pknopf/git/x3/abrarecipes/build/tmp/work/corei7-64-poky-linux/dotnet-source-build/2.1.401-r0/git/Tools/dotnetcli/
      BUILD_TOOLS_TOOL_DIR=/home/pknopf/git/x3/abrarecipes/build/tmp/work/corei7-64-poky-linux/dotnet-source-build/2.1.401-r0/git/Tools/
      BUILDTOOLS_SKIP_CROSSGEN=1
      DotNetBuildFromSource=true
      DotNetRestorePackagesPath=/home/pknopf/git/x3/abrarecipes/build/tmp/work/corei7-64-poky-linux/dotnet-source-build/2.1.401-r0/git/packages/
      DotNetCoreSdkDir=/home/pknopf/git/x3/abrarecipes/build/tmp/work/corei7-64-poky-linux/dotnet-source-build/2.1.401-r0/git/Tools/dotnetcli/
      DotNetBuildToolsDir=/home/pknopf/git/x3/abrarecipes/build/tmp/work/corei7-64-poky-linux/dotnet-source-build/2.1.401-r0/git/Tools/
      StabilizePackageVersion=true
      PB_IsStable=true
      MSBUILDDISABLENODEREUSE=1
      GIT_COMMIT=e1240e4547a992d9e900293aea88ee92f93a45cc
      RESOLVE_REPO_TOOLSET_PACKAGE_DIR=/home/pknopf/git/x3/abrarecipes/build/tmp/work/corei7-64-poky-linux/dotnet-source-build/2.1.401-r0/git/Tools/source-built/RoslynTools.RepoToolset/
      RESOLVE_REPO_TOOLSET_SDK_DIR=/home/pknopf/git/x3/abrarecipes/build/tmp/work/corei7-64-poky-linux/dotnet-source-build/2.1.401-r0/git/Tools/source-built/RoslynTools.RepoToolset/sdk/
      RESOLVE_REPO_TOOLSET_SDK_VERSION=1.0.0-source-built

How long is the sdk project expected to take to build?

I know it can be hard to debug such a custom setup (with patches applied as well!), but any help would be greatly appreciated. This effort is to bring .NET Core support to Yocto (relevant #655). The current effort so far has been for arm only, I'm working on bringing it to x86/x64 as well.

[pauldotknopf]

/cc @Tragetaschen, any thoughts?

[pauldotknopf]

Ok, I just had to wait a little longer for it to complete. In total, it took a little over 8 hours for the entire build to complete. A large chunk of it was building the sdk project.

Is this normal? Is there anyway to speed that up?

Here are the logs for the entire completed build: logs.tar.gz

[dagood]

Our PR CI generally takes 1h 30m, with about 20m of that being sdk. On my local machine the build takes ~50m end-to-end. 8h definitely seems weird, and I haven't seen a difference like that just based on distro.

Do you know if your machine has something unusual about its perf vs. "normal" machines, or if the distro might be doing something special? Network speed, HDD, SSD, RAM (swapping)? Would it be reasonable to compare Yocto perf to a different distro running on the same hardware?

(Also, one thing about the bin/logs files: they always append on subsequent runs, so it's useful to clear them out to make sure it's easy to see only the latest build. Incremental builds also aren't known to work very well--I don't think it'd cause this, but a heads up for the future.)

[pauldotknopf]

Do you know if your machine has something unusual about its perf vs. "normal" machines

The build happens on my Ubuntu 16 (i7 quad core, 16GB ram, SSD) machine. Yocto isn't in-play here, although I am cross-compiling for Yocto.

Thanks for the info.

I'm going to close this issue, since it is working. I've got bigger fish to fry (other coreclr/core-setup bugs, unrelated).