Add auto dependency flow design doc #181

dagood · 2017-09-25T21:01:45Z

Issue: #183. This builds on PR #152 (repo API), adding arguments for automatic dependency flow.

@dotnet/source-build-contrib

eerhardt · 2017-09-26T14:50:51Z

Documentation/auto-dependency-flow/README.md

+## Vision: product build end-to-end
+This narrative describes an end-to-end Core-Setup (Runtime) product build using CoreFX, CoreCLR, and Standard. More repos will be involved in real product builds.
+
+The orchestrator starts by calculating a repo dependency graph. Edges are determined by comparing each repo's *repo output manifest* and *repo input manifest*. The graph is a DAG, so it's buildable. In this case, we have:


repo's repo output manifest and repo input manifest

I question how dynamic this graph needs to be. In my mind, I don't see the dependencies between repos changing often enough to warrant a dynamic system where the orchestrator is completely agnostic of the dependencies. I think the system we have in source-build today works now, and will work for the foreseeable future.

I think the amount of work to build and maintain these input/output manifests for each repo will not be worth the value gained.

I see this is mentioned in the "Incremental steps toward the vision" section below. Do we see this as necessary even in the "long term" vision?

I suppose that depends on how hard it is to change the dependencies later. If the graph is easily updated by checking into the orchestrator itself, it might not be worth the effort. Also, it might be easier to see problems if there's one source of truth for the graph rather than having it sprinkled around the repos.

I agree that keeping the "orchestrator" as simple as possible is important. Clear declarative contracts are good, and having a ridiculously straight forward orchestrator initially will allow the build to stand up, and then we'll really know what needed and where the highest priority problems are

I'm going to add a commit removing the "far enough forward we probably won't ever do it" stuff so we have a concrete idea what that looks like. (I also prefer removing it at this point.)

@markwilkie, what do you mean by "Clear declarative contracts are good"? That seems to argue the opposite of this thread, which is proposing to remove a potential declarative contract and instead (continue to) store the metadata in the source-build repository.

It still sounds to me like you're arguing for these contracts, but in this context, repo output/input manifest == produces/consumes, and I'm getting rid of them. The only use case for repo output/input is removing the "which repo builds first" metadata from source-build, and instead associating repos based on what packages they output/input.

The orchestrator has to build a graph either way, and IMO it's only marginally more challenging with repo input/output manifests. The problem is asking repos to produce this information without first performing a build: this is a rather large implementation/maintenance cost for an undetermined--almost definitely small--gain. (Even after performing a build, this information is not necessarily easy to gather.)

One of the things we learned with desktop is that it's becomes very expensive to manage a product/build when it's unclear what the dependencies are. (it's becomes tough to change anything) The core "requirement" here (for me anyway) is that there's a reasonable way to manage the build over time. A proven strategy is to leverage contracts to help do this. (no magic bullets of course....)

I suspect there are many implementations that will work. For example, where the contract (input/output list?) lives is probably not important. What's important (I think) is that it's explicit what the dependencies are and not simply implied as a "side effect" of the build scripts themselves. Knowing the implications what one's change in that world is a tough place to live.

I see--you could say that each repo produces itself, and each repo in source-build has a list of repos that it consumes. The contract for produces is identity, and for consumes it's having a RepositoryReference item of another repo's name.

Perhaps, although referencing packages is probably the most concrete (and known) way to take these dependencies. So long as we're not taking crazy dependencies on specific little packages, I presume this is a reasonable thing to do. (but I don't know that for sure)

I presume this is a reasonable thing to do.

If "this" is "have repos list the packages they depend on and produce before any building happens", the consensus so far is that it's not reasonable and we shouldn't ask repos to do it. Instead, we expect that repo-level dependencies defined in the source-build repo are clear and easy enough to maintain that we should stick with that. (What I described above is the current state of the repo's "contracts", which can also be used by the product build orchestrator.)

eerhardt · 2017-09-26T14:54:34Z

Documentation/auto-dependency-flow/README.md

+
+The process repeats for the next leaf, Core-Setup: the orchestrator creates the build output props and passes it into a Core-Setup repo build. The Core-Setup build finishes, outputting to the blob feed, and the Core-Setup leaf is removed.
+
+There are no more nodes: the repo builds are complete! The blob feed contains all the product outputs. The orchestrator performs finalization steps (signing, testing, publishing) and then the product build is complete.


This may be picking on the details, but signing at this stage isn't going to work.

Take for example the Core-Setup build today - it needs to produce .zips, .msis, etc. that have assemblies embedded in them. In order to sign all these different artifacts, we would need to write systems that knows how to take these things apart, sign all the things inside them that need to be signed, and then put them back together again. Signing is probably best done during the "repo builds" themselves.

Same as above, this is something that was discussed earlier on as a way to save time, but I agree I don't see it being feasible (and maybe not even possible, depending on exactly how the layers go together--e.g. signed things put in signed containers would still require multiple round trips).

Also giving removing this a shot.

eerhardt · 2017-09-26T15:07:21Z

Documentation/auto-dependency-flow/README.md

+
+For product builds, we want to allow the repos continue to depend on the oldest possible TFM to give the product greater reach. Specifically, Visual Studio targets old netstandard TFMs.
+
+For source builds, upgrading TFMs allows the repos to avoid depending on old package versions. They would also need to be built from source.


Do we want to spec here how this is going to be specified to the repos? ex. /p:DotNetSourceBuildTargetFramework=netcoreapp2.1.

Hmm, I'd kind of assumed that the TFM could somehow be picked up from the source-built NETStandard.Library/Microsoft.NETCore.App nupkg using a props file in the package or something. Looking a bit closer, I don't see anything like that, and it looks like CoreFX needs netcoreappX.X before Core-Setup builds, so it probably can't even work in theory.

An arg makes sense to me. It sounds like we'll need one for netstandard and one for netcoreapp.

We can't pick up the TFM from a NuGet package because NuGet needs the TFM to know which assets to pick out of the NuGet packages. TFMs need to be specified first - before 'restore' can take place.

It sounds like we'll need one for netstandard and one for netcoreapp.

Do we need both? I assumed the idea was that the whole source build would be for netcoreapp? /cc @weshaggard @ericstj

I don't have a good grasp of what this impacts during the product build itself. I imagine people using the source-build end product would want to be able to build apps against netstandard to have the greater targeting area, but I don't know what's necessary to make that happen.

A really good/important question we need to figure out is: "What is the expected usage of these source-built assets?"

Do we expected people to reference the .nupkgs that are produced by this build? If so, then they won't be able to build any netstandard1.x projects. (Or even netstandard2.0 when we move to a netstandard2.1 or 3.0.) Is this intended?

Created #184 to track/discuss--maybe we need some documentation around that point in particular.

eerhardt · 2017-09-26T15:42:01Z

/cc @livarcocc @DustinCampbell

eerhardt · 2017-09-26T18:15:42Z

Documentation/auto-dependency-flow/api.md

+
+# Output placement arguments
+
+## `/p:DotNetOutputBlobFeedDir=<target directory>`


What does the word blob mean in this property? Is it linked to Azure blobs in some way?

This refers to "blob feed", which is the dir structure this API tells repos to put outputs into. It's not a free-for-all dumping ground, so I think it makes sense to have the structure's name in the property rather than something more generic. Blob feeds are briefly described in the terms.

For what "blob" means in "blob feed", I don't know. Sometime soon I'm hoping there will be docs in BuildTools I can link to that will explain.

dagood · 2017-09-27T17:30:54Z

I'm merging this in to establish a location for this design's current state. Please feel free to create issues or continue to comment here to discuss it more.

natemcmaster · 2017-09-28T23:15:39Z

May have been mentioned above, but what was the reasoning for changing from /p: to -p: in 3f1f37e? I would prefer /p: so I don't have to implement a parsers to change all these args before passing to msbuild

dagood · 2017-09-28T23:24:36Z

MSBuild supports -p:! This change flowed from the repo API doc: #152.

natemcmaster · 2017-09-28T23:27:42Z

Oh, awesome 🎆. TIL a new msbuild syntax. Disregard previous comment

natemcmaster · 2017-09-29T17:01:55Z

Started trying to use this. I found at least one problem with this: PowerShell. All aspnet repos have a build.cmd script that forwards all arguments to build.ps1. The parameter binding behavior of PowerShell makes it difficult to use the posix format args.

For example, consider this:

REM build.cmd
@ECHO OFF
PowerShell -NoProfile -NoLogo -ExecutionPolicy unrestricted -Command "[System.Threading.Thread]::CurrentThread.CurrentCulture = ''; [System.Threading.Thread]::CurrentThread.CurrentUICulture = '';& '%~dp0build.ps1' %*; exit $LASTEXITCODE"

# build.ps1
[CmdletBinding(PositionalBinding = $false)]
param(
    [string]$Path = $PSScriptRoot,
    [Parameter(ValueFromRemainingArguments = $true)]
    [string[]]$MSBuildArgs
)

Write-Host "`$Path = $Path"
Write-Host "`$MSBuildArgs = $MSBuildArgs"
Write-Host "`$MSBuildArgs length = $($MSBuildArgs.Length)"
# & dotnet msbuild @MSBuildArgs

You get the following behavior:

C:\tmp> build.cmd -t:Build
$Path = C:\tmp
$MSBuildArgs = -t: Test
$MSBuildArgs length = 2

C:\tmp> build.cmd /t:Build
$Path = C:\tmp
$MSBuildArgs = /t:Test
$MSBuildArgs length = 1

C:\tmp> build.cmd -p:SomeProp=true
$Path = SomeProp=true
$MSBuildArgs =
$MSBuildArgs length = 0

C:\tmp> build.cmd -p:SomeProp=true -p:SomeOtherProp=true
ERROR: Cannot bind parameter because parameter 'p' is specified more than once. 

C:\tmp> build.cmd /p:SomeProp=true /p:SomeOtherProp=true
$Path = C:\tmp
$MSBuildArgs = /p:SomeProp=true /p:SomeOtherProp=true
$MSBuildArgs length = 2

dagood · 2017-09-29T17:16:25Z

@natemcmaster Opened #194 to talk about that in more detail.

natemcmaster · 2017-09-29T17:22:12Z

Thanks

dagood added 2 commits September 25, 2017 15:42

Initial commit of auto dependency flow design

6ec81fd

Remove incorrect Import Project single quoting

17734c7

eerhardt reviewed Sep 26, 2017

View reviewed changes

dagood mentioned this pull request Sep 26, 2017

Define auto dependency flow repo API #183

Closed

eerhardt reviewed Sep 26, 2017

View reviewed changes

dagood mentioned this pull request Sep 26, 2017

Determine desired TFM behavior in a product build (+document?) #184

Closed

dagood added 5 commits September 26, 2017 16:47

Simplify: remove bits that are distant + uncertain

e3ffcb0

Add TFM flow arg

ab81bde

Replace /p: with -p:, match framework doc

3f1f37e

Fix api.md anchor references

bced5d1

Link and describe files in README

a45413b

dagood merged commit b88f36e into dotnet:dev/release/2.0 Sep 27, 2017

dagood mentioned this pull request Sep 29, 2017

Determine Repo API framework recommendation for -p:Foo=Bar PowerShell passthrough #194

Closed

dagood deleted the auto-flow-doc branch September 29, 2017 17:20

crummel pushed a commit to crummel/dotnet_source-build that referenced this pull request Oct 11, 2017

Add auto dependency flow design doc (dotnet#181)

282532a


		The process repeats for the next leaf, Core-Setup: the orchestrator creates the build output props and passes it into a Core-Setup repo build. The Core-Setup build finishes, outputting to the blob feed, and the Core-Setup leaf is removed.

		There are no more nodes: the repo builds are complete! The blob feed contains all the product outputs. The orchestrator performs finalization steps (signing, testing, publishing) and then the product build is complete.


		For product builds, we want to allow the repos continue to depend on the oldest possible TFM to give the product greater reach. Specifically, Visual Studio targets old netstandard TFMs.

		For source builds, upgrading TFMs allows the repos to avoid depending on old package versions. They would also need to be built from source.


		# Output placement arguments

		## `/p:DotNetOutputBlobFeedDir=<target directory>`

Add auto dependency flow design doc #181

Add auto dependency flow design doc #181

Uh oh!

Conversation

dagood commented Sep 25, 2017 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

eerhardt commented Sep 26, 2017

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

dagood commented Sep 27, 2017

Uh oh!

natemcmaster commented Sep 28, 2017

Uh oh!

dagood commented Sep 28, 2017

Uh oh!

natemcmaster commented Sep 28, 2017

Uh oh!

natemcmaster commented Sep 29, 2017

Uh oh!

dagood commented Sep 29, 2017

Uh oh!

natemcmaster commented Sep 29, 2017

Uh oh!

Uh oh!

dagood commented Sep 25, 2017 •

edited

Loading