Add CLI to known-good #418

dagood · 2018-04-11T22:22:36Z

No description provided.

dagood · 2018-04-12T15:55:45Z

NuGet.Protocol 4.7.0-preview3.4324 depends on NuGet.Configuration (>= 4.7.0-preview3.4324) but NuGet.Configuration 4.7.0-preview3.4324 was not found. An approximate best match of NuGet.Configuration 4.7.0-preview3.5032 was resolved.
Detected package downgrade: NuGet.Common from 4.7.0-preview4.5065 to 4.7.0-preview3.4324. Reference the package directly from the project to select a different version. \n dotnet -> Microsoft.DotNet.Configurer -> NuGet.Configuration 4.7.0-preview4.5065 -> NuGet.Common (>= 4.7.0-preview4.5065) \n dotnet -> Microsoft.DotNet.Configurer -> NuGet.Common (>= 4.7.0-preview3.4324)
Detected package downgrade: NuGet.Frameworks from 4.7.0-preview3.5032 to 4.7.0-preview3.4324. Reference the package directly from the project to select a different version. \n dotnet -> Microsoft.DotNet.Cli.Utils -> NuGet.ProjectModel 4.7.0-preview3.4324 -> NuGet.DependencyResolver.Core 4.7.0-preview3.4324 -> NuGet.Protocol 4.7.0-preview3.4324 -> NuGet.Configuration 4.7.0-preview3.5032 -> NuGet.Common 4.7.0-preview3.5032 -> NuGet.Frameworks (>= 4.7.0-preview3.5032) \n dotnet -> Microsoft.DotNet.Cli.Utils -> NuGet.Frameworks (>= 4.7.0-preview3.4324)

NuGet.Configuration isn't in the blob feed, which would explain all these. Either it wasn't built, or it isn't copied to the blob feed correctly. Looking into it.

dagood · 2018-04-12T17:23:17Z

Enabling the NuGet.Configuration package build worked, next up:

Detected package downgrade: System.Collections.Immutable from 1.5.0-preview2-26401-03 to 1.3.1. Reference the package directly from the project to select a different version.
tool_fsc -> Microsoft.FSharp.Compiler 10.1.1-rc-171107-0 -> Microsoft.DiaSymReader.PortablePdb 1.5.0-dev -> System.Collections.Immutable (>= 1.5.0-preview2-26401-03)
tool_fsc -> Microsoft.FSharp.Compiler 10.1.1-rc-171107-0 -> System.Collections.Immutable (>= 1.3.1)

Detected package downgrade: System.Reflection.Metadata from 1.6.0-preview2-26401-03 to 1.4.2. Reference the package directly from the project to select a different version.
tool_fsc -> Microsoft.FSharp.Compiler 10.1.1-rc-171107-0 -> Microsoft.DiaSymReader.PortablePdb 1.5.0-dev -> System.Reflection.Metadata (>= 1.6.0-preview2-26401-03)
tool_fsc -> Microsoft.FSharp.Compiler 10.1.1-rc-171107-0 -> System.Reflection.Metadata (>= 1.4.2)

I'm going to try adding a package reference in CLI to force the source-built versions to be used.
@dseefeld can you look at why it looks like fsharp didn't uptake System.Reflection.Metadata and System.Collections.Immutable?

dagood · 2018-04-12T17:40:15Z

Looking again, that's an internet version, not source-built--I think these packages aren't being produced by CoreFX when they should be.

dagood · 2018-04-12T17:50:09Z

Wes pointed me to #210, which tracks adding the type of CoreFX build that produces those packages. (Apparently not trivial.)

Tomas pointed me to some commits I can revert to bring symreader and symreader-portable back to depending on 1.3.1 and 1.4.2, so the mismatch will go away. I'm going to add patches to revert: dotnet/symreader@ac20b8b dotnet/symreader-portable@58d468a

dagood · 2018-04-12T20:55:52Z

Info on the MSBuild fix this PR needed: the source-built Microsoft.Build.Runtime package didn't mark the included contentFiles/any/netcoreapp2.0/* as Content, so they were being passed to CSC as Compile. Building MSBuild.dll fails because it's not text, and the MSBuild xml files cause even more errors after that. The fix marks these properly as Content in the package.

dagood · 2018-04-12T21:47:17Z

The last commit has successful first-stage CI legs, but @dseefeld locally saw the tarball fail in MSBuild because I had the MSBuild commit change without the repos/msbuild.proj change. Added that to this PR temporarily. (I expect to rebase on dev/release/2.1 at some point after the MSBuild fix is merged.)

(I forgot to change the fork back to Microsoft/MSBuild as part of my last commit, which would match the PR better: #410. The MSBuild commits are identical, though.)

dagood · 2018-04-13T14:51:53Z

@dotnet-bot test ci please
Offline tarball build regularly hits the 2-hour timeout, doubling it.

https://ci.dot.net/job/dotnet_source-build/job/dev_release_2.1/job/GenPRTest/job/RHEL7.2_Unshared_Release_prtest/1/

mmitche · 2018-04-13T16:39:54Z

@dotnet-bot test ci please
@dotnet-bot test this please

Fix cli/run-build.sh's argsnotargets. Fix CLI's NuGet.Config generation with source-build directory inputs. Disable crossgen until we figure out the dll load error. Disable bundled tools until we figure out the best solution for dotnet-dev-certs, dotnet-ef, etc. Use existing GitInfo* properties if provided.

peterhuene · 2018-04-13T18:20:16Z

patches/cli/0007-Fix-argsnotargets.patch

+
+ if [ $BUILD -eq 1 ]; then
+-    (set -x; dotnet msbuild build.proj /bl:msbuild.generatepropsfile.binlog /p:Architecture=$ARCHITECTURE $CUSTOM_BUILD_ARGS /p:GeneratePropsFile=true /t:WriteDynamicPropsToStaticPropsFiles $argsnotargets)
+    (set -x; dotnet msbuild build.proj /bl:msbuild.generatepropsfile.binlog /p:Architecture=$ARCHITECTURE $CUSTOM_BUILD_ARGS /p:GeneratePropsFile=true /t:WriteDynamicPropsToStaticPropsFiles ${argsnotargets[@]})


I'm going to guess the intention of using $argsnotargets here is to pass the entire array through.

Perhaps this is meant to be $(IFS=' '; echo "${argsnotargets[*]}")?

@johnbeisner is that right? This is a patch in source-build that we'll eventually take into CLI, so I'd like to get it right.

That would make sense to me. I borrowed the syntax I used from for arg in ${args[@]}. (That for loop strongly indicates to me that all the non-targets args are meant to be passed, but this usage was bugged.)

peterhuene · 2018-04-13T18:37:47Z

patches/cli/0002-Make-BuildFromSource-disable-ASP.NET-dev-certs.patch

@@ -0,0 +1,100 @@
+From cd96ec40e18bf5cdce6cb3aad8b699051da7bf34 Mon Sep 17 00:00:00 2001


I think we should also patch this:

private bool ShouldGenerateAspNetCertificate() { #if DOTNET_BUILD_FROM_SOURCE return false; #else var generateAspNetCertificate = _environmentProvider.GetEnvironmentVariableAsBool("DOTNET_GENERATE_ASPNET_CERTIFICATE", true); return ShouldRunFirstRunExperience() && generateAspNetCertificate && !_aspNetCertificateSentinel.Exists(); #endif }

to prevent the first run experience from telling users a certificate was created when one wasn't.

Filed #434 to get this in.

peterhuene · 2018-04-13T18:45:40Z

Over than the additional patch, I think this looks good. Looking over the CLI patches, I think I should be able to get those all in upstream.

standardJobSetup sets the 120-minute default that we need to override.

dagood · 2018-04-13T19:26:27Z

I queued this unshared RHEL with the extended timeout: https://ci.dot.net/job/dotnet_source-build/job/dev_release_2.1/job/GenPRTest/job/RHEL7.2_Unshared_Release_prtest/6/. (Expect the automatically triggered one to fail due to timeout.)

dagood · 2018-04-13T22:08:03Z

OSX failed to "download" a local file:

Target "DownloadHostAndSharedFxArtifacts" in file "/Users/dotnet-bot/j/workspace/dotnet_source-build/dev_release_2.1/OSX10.12_Release_prtest/src/cli/build/Prepare.targets" from project "/Users/dotnet-bot/j/workspace/dotnet_source-build/dev_release_2.1/OSX10.12_Release_prtest/src/cli/build.proj" (target "Prepare" depends on it):
Set Property: OverwriteExtractionDestination=
Set Property: OverwriteExtractionDestination=True
Using "DownloadFile" task from assembly "/Users/dotnet-bot/j/workspace/dotnet_source-build/dev_release_2.1/OSX10.12_Release_prtest/src/cli/build_projects/dotnet-cli-build/bin/dotnet-cli-build.dll".
Task "DownloadFile"
  Task Parameter:Uri=file:///Users/dotnet-bot/j/workspace/dotnet_source-build/dev_release_2.1/OSX10.12_Release_prtest/bin/obj/x64/Release/blobs/Runtime/2.1.0-preview2-26411-0/dotnet-runtime-2.1.0-preview2-26411-0-osx-x64.tar.gz
  Task Parameter:DestinationPath=/Users/dotnet-bot/j/workspace/dotnet_source-build/dev_release_2.1/OSX10.12_Release_prtest/src/cli/bin/2/osx-x64/intermediate/coreSetupDownload/2.1.0-preview2-26411-0/combinedSharedHostAndFrameworkArchive.tar.gz
  Copying '/Users/dotnet-bot/j/workspace/dotnet_source-build/dev_release_2.1/OSX10.12_Release_prtest/bin/obj/x64/Release/blobs/Runtime/2.1.0-preview2-26411-0/dotnet-runtime-2.1.0-preview2-26411-0-osx-x64.tar.gz' to '/Users/dotnet-bot/j/workspace/dotnet_source-build/dev_release_2.1/OSX10.12_Release_prtest/src/cli/bin/2/osx-x64/intermediate/coreSetupDownload/2.1.0-preview2-26411-0/combinedSharedHostAndFrameworkArchive.tar.gz'

/Users/dotnet-bot/j/workspace/dotnet_source-build/dev_release_2.1/OSX10.12_Release_prtest/src/cli/build/Prepare.targets(43,5): error MSB4018: The "DownloadFile" task failed unexpectedly. [/Users/dotnet-bot/j/workspace/dotnet_source-build/dev_release_2.1/OSX10.12_Release_prtest/src/cli/build.proj] [/Users/dotnet-bot/j/workspace/dotnet_source-build/dev_release_2.1/OSX10.12_Release_prtest/repos/cli.proj]

Building locally for repro. My first thought is that the filename Core-Setup produced could be wrong.

Fixed with next commit, see #438.

This makes the Core-Setup filename match what CLI expects.

dagood · 2018-04-15T19:53:16Z

Filed https://github.com/dotnet/core-eng/issues/3291 for the CI infra failures on the legs that use Ubuntu.

dagood · 2018-04-16T14:52:27Z

@dotnet-bot
test RHEL7.2 Unshared Release
test Ubuntu16.04 Release
(In case we get lucky with the choice of machines. (Looks like we did.))

dagood · 2018-04-16T15:47:36Z

Merging: RHEL7.2 leg is expected to time out, and with everything else green it seems unlikely the OSX change broke it since I tested with the extended timeout on the previous commit.

dagood mentioned this pull request Apr 12, 2018

Remove patches from nuget-client #420

Closed

dagood force-pushed the add-cli branch from e6d9f5f to a217a0d Compare April 13, 2018 14:47

dagood added 7 commits April 13, 2018 11:40

Add CLI to known-good, fix args

eb9ca59

nuget-client workaround: build NuGet.Configuration

a61a2f0

symreader[-portable] workaround: revert pkg update

e8318d6

Double offline build timeout

de9c6bd

Build MSBuild 15.1.9999 to ensure it's used in PVP

2cfe24b

Convert forks to patches

a184e0b

dagood force-pushed the add-cli branch from 4896b71 to a184e0b Compare April 13, 2018 16:45

dagood changed the title ~~(WIP) Add CLI to known-good~~ Add CLI to known-good Apr 13, 2018

This was referenced Apr 13, 2018

Remove patches from cli #426

Closed

Enable cli crossgen step #427

Closed

Enable (some?) cli bundled tools #428

Closed

peterhuene reviewed Apr 13, 2018

View reviewed changes

dagood mentioned this pull request Apr 13, 2018

For 2.1-rc1, examine the set of prebuilt packages for dependencies that need to be cut #433

Closed

dagood mentioned this pull request Apr 13, 2018

Patch CLI ShouldGenerateAspNetCertificate to accurately tell user if a cert was generated #434

Closed

Move setJobTimeout after standardJobSetup

043a6ab

standardJobSetup sets the 120-minute default that we need to override.

Always build portable runtime on OSX

009da49

This makes the Core-Setup filename match what CLI expects.

dagood merged commit b964970 into dotnet:dev/release/2.1 Apr 16, 2018

dseefeld mentioned this pull request Apr 16, 2018

Get CI working e2e with offline build #304

Closed

26 tasks

dagood mentioned this pull request Apr 16, 2018

Make the cli repo build in dev/release/2.1 #414

Closed

dagood mentioned this pull request May 2, 2018

Removing some patches for source build. dotnet/cli#9188

Merged

dagood deleted the add-cli branch May 4, 2018 15:25

		@@ -0,0 +1,100 @@
		From cd96ec40e18bf5cdce6cb3aad8b699051da7bf34 Mon Sep 17 00:00:00 2001

Add CLI to known-good #418

Add CLI to known-good #418

Uh oh!

Conversation

dagood commented Apr 11, 2018

Uh oh!

dagood commented Apr 12, 2018

Uh oh!

dagood commented Apr 12, 2018

Uh oh!

dagood commented Apr 12, 2018

Uh oh!

dagood commented Apr 12, 2018

Uh oh!

dagood commented Apr 12, 2018

Uh oh!

dagood commented Apr 12, 2018 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

dagood commented Apr 13, 2018 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

mmitche commented Apr 13, 2018

Uh oh!

peterhuene Apr 13, 2018 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

dagood Apr 13, 2018

Choose a reason for hiding this comment

Uh oh!

peterhuene Apr 13, 2018 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

dagood Apr 13, 2018

Choose a reason for hiding this comment

Uh oh!

peterhuene commented Apr 13, 2018

Uh oh!

dagood commented Apr 13, 2018

Uh oh!

dagood commented Apr 13, 2018 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

dagood commented Apr 15, 2018 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

dagood commented Apr 16, 2018 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

dagood commented Apr 16, 2018

Uh oh!

Uh oh!

dagood commented Apr 12, 2018 •

edited

Loading

dagood commented Apr 13, 2018 •

edited

Loading

peterhuene Apr 13, 2018 •

edited

Loading

peterhuene Apr 13, 2018 •

edited

Loading

dagood commented Apr 13, 2018 •

edited

Loading

dagood commented Apr 15, 2018 •

edited

Loading

dagood commented Apr 16, 2018 •

edited

Loading