Skip to content

Commit b88f36e

Browse files
dagooddagood
authored
Add auto dependency flow design doc (#181)
1 parent 24d7d77 commit b88f36e

File tree

3 files changed

+284
-0
lines changed

3 files changed

+284
-0
lines changed

‎Documentation/auto-dependency-flow/README.md

Lines changed: 93 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,93 @@
1+
# Automatic dependency flow
2+
3+
During a build, the orchestrator must provide each repo build with outputs and information from its upstream builds so that the end result is a full product. This is automatic dependency flow.
4+
5+
* [`api.md`](api.md): Lists Repo API arguments a repo must accept to participate in auto dependency flow.
6+
* [`example-prototype.md`](example-prototype.md): Describes a prototype implementation of the auto dependency flow arguments on several repos.
7+
8+
## Terms
9+
*[Product] Dependency*: an artifact generated by one repo build that is required for a downstream repo build. "Product" is assumed, but may be used to clarify vs. a toolset dependency.
10+
11+
*Full product*: a set of repository build outputs that are coherent. As opposed to "isolated products" which have uncoordinated dependency versions that may conflict.
12+
13+
*Isolated build*: a build that a repo performs independently. The repo chooses its own dependencies, usually pinning to specific versions to isolate itself from breaking changes until they can be taken. This type of build is performed for short-term productivity, and it doesn't affect the product build or source build. These builds are the "old way" of shipping the .NET Core product: the dependencies "roll up" in a series of isolated builds to form the full product.
14+
15+
*Product build*: a build that produced a *full product* for all platforms Microsoft ships.
16+
17+
*Source build*: a build that doesn't depend on any external pre-built dependencies to produce a *full product* for a specific platform. AKA "build from source".
18+
19+
*Orchestrator*: the system that keeps track of all repos involved in a build and performs the steps necessary to coordinate the build process. This doc assumes the orchestrator has some control over the machine(s) being built on: at minimum, downloading files to disk and customizing build command args. The orchestrator is involved in source builds and product builds, but not the isolated builds.
20+
21+
*Repo build*: a particular repository's build that the orchestrator orchestrates. If building the repo involves multiple build legs (e.g. to build native bits on different platforms), all legs together are called *a* repo build.
22+
23+
*Blob feed*: a structured directory of build outputs. AKA "transport feed". Summary of important details for auto dependency flow:
24+
25+
* There is one blob feed per product build. Repo builds add to the single blob feed.
26+
* The orchestrator makes the blob feed available to all repo builds.
27+
* The blob feed is on disk or in blob storage virtual directories, but in general it has the same structure either way.
28+
29+
*Upstream/Downstream*: when dicussing repo A, repo B is "upstream" if A has a dependency on product outputs from B. Only repos built by the orchestrator are included. For example, in this doc, external toolsets aren't considered upstream from any repository, nor are any redistributed libraries that we don't build ourselves.
30+
31+
32+
# Inserting dependency changes
33+
34+
The dependency change flow is shared by all builds that involve the orchestrator.
35+
36+
TL;DR: the orchestrator derives package version props from the blob feed after each repo build and feeds the data into all subsequent repo builds.
37+
38+
## Vision: product build end-to-end
39+
This narrative describes an end-to-end Core-Setup (Runtime) product build using CoreFX, CoreCLR, and Standard. More repos will be involved in real product builds.
40+
41+
The orchestrator starts by calculating a repo dependency graph. Edges are determined using repo-to-repo dependency metadata stored and maintained in the source-build repository. The graph is a DAG, so it's buildable. In this case, we have:
42+
43+
* Core-Setup
44+
* CoreFX
45+
* CoreCLR
46+
* Standard
47+
48+
The orchestrator sees that CoreCLR and Standard are leaves and kicks off both repo builds. Packages produced [end up on the blob feed](#blob-feed) when they're finished, and CoreCLR and Standard are removed from the graph.
49+
50+
* Core-Setup
51+
* CoreFX
52+
53+
CoreFX is the next leaf, so it will be built next. The orchestrator [creates the package version props based on the blob feed and passes it](#-passing-dependency-versions) into a repo build of CoreFX. CoreFX uses the file to determine which package versions to use, and consumes them from the blob feed. The repo build completes and the CoreFX leaf is removed.
54+
55+
* Core-Setup
56+
57+
The process repeats for the next leaf, Core-Setup: the orchestrator creates the build output props and passes it into a Core-Setup repo build. The Core-Setup build finishes, outputting to the blob feed, and the Core-Setup leaf is removed.
58+
59+
There are no more nodes: the repo builds are complete! The blob feed contains all the product outputs. The orchestrator performs finalization steps (testing, publishing) and then the product build is complete.
60+
61+
### Incremental steps toward the vision
62+
The narrative represents the ideal flow, and some parts won't be available in our first steps:
63+
64+
* The CoreCLR and Standard builds may not be run in parallel.
65+
* The orchestrator might not have any finalization steps. Instead, they happen during the repo builds.
66+
67+
## Blob feed
68+
The orchestrator indicates a directory on the build machine to place output files using [`-p:DotNetOutputBlobFeedDir`](api.md#-pdotnetoutputblobfeeddirtarget-directory). The orchestrator makes the contents available to subsequent repo builds using [`-p:DotNetRestoreSourcePropsPath`](api.md#-pdotnetrestoresourcepropspathpath) and [`-p:DotNetAssetRootUrl`](api.md#-pdotnetassetrooturlurl).
69+
70+
## Passing dependency versions
71+
To pass the upstream NuGet package versions to use, the orchestrator creates a "package version props" file based on the current blob feed contents and passes it with [`-p:DotNetPackageVersionPropsPath=<path>`](api.md#-pdotnetpackageversionpropspathpath).
72+
73+
Additional API surface area for dependency passing may be added in the future, based on need. For example:
74+
75+
* Native-only repo builds may require a simpler format to reasonably parse.
76+
* Additional MSBuild props files may be added to flow non-package dependencies.
77+
78+
79+
# Source build vs. product build dependency flow
80+
81+
Source build has some requirements that don't necessarily apply to the product build: it must avoid dependencies on prebuilt dependencies and [rebuild without using the prebuilt version ("bootstrap")](https://fedoraproject.org/wiki/Packaging:Guidelines#Bootstrapping) where that's not possible. This separates the product build and source build in a few ways relevant to auto dependency flow.
82+
83+
## Latest Target Framework Monikers (TFMs)
84+
E.g. upgrading to netstandard2.0, netcoreapp2.1.
85+
86+
For product builds, we want to allow the repos continue to depend on the oldest possible TFM to give the product greater reach. Specifically, Visual Studio targets old netstandard TFMs.
87+
88+
For source builds, upgrading TFMs allows the repos to avoid depending on old package versions. They would also need to be built from source.
89+
90+
The [`-p:DotNetSourceBuildTargetFrameworkPropsPath`](api.md#-pdotnetsourcebuildtargetframeworkpropspathpath) API is used during source builds to pass the TFM being built.
91+
92+
## Upgrading all tooling dependencies we build to the latest
93+
This is *required* for build from source to satisfy bootstrapping requirements, and *desired* (mildly) for product builds in order to dogfood.

‎Documentation/auto-dependency-flow/api.md

Lines changed: 177 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,177 @@
1+
# Automatic dependency flow Repo API
2+
3+
Repos must implement these arguments to participate in the orchestrated dependency flow.
4+
5+
6+
# Dependency version input arguments
7+
8+
## `-p:DotNetPackageVersionPropsPath=<path>`
9+
Directs the repo build to use a "package version props" file at `path`. The versions in the file must be used instead of any defaults the repo would ordinarily depend on.
10+
11+
`path` is outside of the repository directory.
12+
13+
### File format: package version props
14+
The file specifies one property for each package. The name is the package identity after being suffixed with `PackageVersion`, chars uppercased to get to PascalCase, and non-alphanumeric characters (assumed to be delimiters) removed. The property value is the full version of the package. Example with a few CoreCLR packages:
15+
16+
```xml
17+
<?xml version="1.0" encoding="utf-8"?>
18+
<Project ToolsVersion="14.0" xmlns="http://schemas.microsoft.com/developer/msbuild/2003">
19+
<PropertyGroup>
20+
<MicrosoftNETCoreRuntimeCoreCLRPackageVersion>2.1.0-preview2-25701-02</MicrosoftNETCoreRuntimeCoreCLRPackageVersion>
21+
<RuntimeOsxX64MicrosoftNETCoreRuntimeCoreCLRPackageVersion>2.1.0-preview2-25701-02</RuntimeOsxX64MicrosoftNETCoreRuntimeCoreCLRPackageVersion>
22+
<TransportMicrosoftNETCoreRuntimeCoreCLRPackageVersion>2.1.0-preview2-25701-02</TransportMicrosoftNETCoreRuntimeCoreCLRPackageVersion>
23+
</PropertyGroup>
24+
</Project>
25+
```
26+
27+
To collect package identities and versions for the package version props, the orchestrator reads the `nuspec` of every package on the blob feed in `packages/*.nupkg`. The file is placed in a directory on the build machine outside of the repo's git repository.
28+
29+
### Recommended implementation
30+
Use the build dependency props format for all `PackageReference`s that may be satisfied by another repo that is being orchestrated:
31+
32+
```xml
33+
<PackageReference Include="System.Collections.Immutable"
34+
Version="$(SystemCollectionsImmutablePackageVersion)" />
35+
```
36+
37+
Store all defaults in a centralized `.props` file, and import `DotNetPackageVersionPropsPath` if specified to override the defaults:
38+
39+
```xml
40+
<?xml version="1.0" encoding="utf-8"?>
41+
<Project ToolsVersion="15.0" xmlns="http://schemas.microsoft.com/developer/msbuild/2003">
42+
<PropertyGroup>
43+
<SystemCollectionsImmutablePackageVersion>1.5.0-preview1-25712-02</SystemCollectionsImmutablePackageVersion>
44+
<!-- ...All NuGet package dependency versions. --->
45+
</PropertyGroup>
46+
47+
<!-- Override isolated build dependency versions with product build's. -->
48+
<Import Project="$(DotNetPackageVersionPropsPath)"
49+
Condition="'$(DotNetPackageVersionPropsPath)' != ''" />
50+
</Project>
51+
```
52+
53+
## `-p:DotNetSourceBuildTargetFrameworkPropsPath=<path>`
54+
**NOTE: This API may be simplified to use an argument instead of a file if only one TFM (target framework moniker) is required.** See [source-build#184](https://github.com/dotnet/source-build/issues/184).
55+
56+
Directs the repo to use the "source build target framework props" file at `path`. The repo must use the version of each TFM listed in the file instead of any default.
57+
58+
The TFMs must be used when source build wouldn't work with the repo's default values. If a product build is being orchestrated, for example, other TFMs are permitted and this property is not passed.
59+
60+
### File format: restore target framework props
61+
Each TFM being built has a property specifying the name of that TFM as built in the current source build. Example:
62+
63+
```xml
64+
<?xml version="1.0" encoding="utf-8"?>
65+
<Project ToolsVersion="14.0" xmlns="http://schemas.microsoft.com/developer/msbuild/2003">
66+
<PropertyGroup>
67+
<DotNetNETCoreAppTargetFramework>netcoreapp2.1</DotNetNETCoreAppTargetFramework>
68+
<DotNetNETStandardTargetFramework>netstandard2.1</DotNetNETStandardTargetFramework>
69+
</PropertyGroup>
70+
</Project>
71+
```
72+
73+
### Recommended implementation
74+
Import the "restore target framework props" file in a high-level MSBuild file in the repository, and set up an overridable property for each TFM the project uses. Use those properties in project files instead of hard-coded values.
75+
76+
77+
# Source override arguments
78+
79+
## `-p:DotNetBuildOffline=<bool>`
80+
Directs the repo to skip online sources if `bool` is `true`.
81+
82+
## `-p:DotNetRestoreSourcePropsPath=<path>`
83+
Directs the repo to use the semicolon-delimited sources in the `DotNetRestoreSources` property specified by the "restore source props" file at `path`. The sources should be the repo's highest priority NuGet package sources.
84+
85+
Unless otherwise specified by `DotNetBuildOffline`, the repo may also use its default sources.
86+
87+
### File format: restore source props
88+
An MSBuild file defining a single property `DotNetRestoreSources`. This file is necessary to avoid issues with escaping: there are various issues trying to pass `;` on the command line to delimit sources. Example:
89+
90+
```xml
91+
<?xml version="1.0" encoding="utf-8"?>
92+
<Project ToolsVersion="14.0" xmlns="http://schemas.microsoft.com/developer/msbuild/2003">
93+
<PropertyGroup>
94+
<DotNetRestoreSources>
95+
/git/source-build/bin/obj/x64/Release/blob-feed/packages/;
96+
/git/source-build/bin/prebuilt/nuget-packages/;
97+
</DotNetRestoreSources>
98+
</PropertyGroup>
99+
</Project>
100+
```
101+
102+
### Recommended implementation
103+
Import `DotNetRestoreSourcePropsPath` as an MSBuild props file, and use properties similar to the following to define restore sources:
104+
105+
```xml
106+
<Project>
107+
<Import Project="$(DotNetRestoreSourcePropsPath)"
108+
Condition="'$(DotNetRestoreSourcePropsPath)' != ''"/>
109+
110+
<PropertyGroup>
111+
<RestoreSources>$(DotNetRestoreSources)</RestoreSources>
112+
<RestoreSources Condition="'$(DotNetBuildOffline)' != 'true'">
113+
$(RestoreSources);
114+
https://dotnet.myget.org/F/dotnet-buildtools/api/v3/index.json;
115+
https://dotnet.myget.org/F/dotnet-core/api/v3/index.json;
116+
https://api.nuget.org/v3/index.json
117+
</RestoreSources>
118+
</PropertyGroup>
119+
</Project>
120+
```
121+
122+
If the repo doesn't use [NuGet MSBuild targets](https://docs.microsoft.com/en-us/nuget/schema/msbuild-targets#restore-target) to restore, there's more work to use the `RestoreSources` property. For example, to pass `--source` args, `RestoreSources` can be `Include`d in an Item element to split it, then formatted.
123+
124+
There is no known reasonable way to use NuGet.Config files, satisfy this API, *and* avoid duplicating the default source information. The recommendation is to not use NuGet.Config.
125+
126+
## `-p:DotNetAssetRootUrl=<url>`
127+
Directs the repo to get binary assets from a given base `url`. For example, installers and lzma files. The scheme can be `http[s]` or `file`. The url ends with a `/`.
128+
129+
### Conventions
130+
Repos must cooperate to establish conventions. Core-Setup needs to implement `DotNetOutputBlobFeedDir` in a way that allows CLI to find its binaries in `DotNetAssetRootUrl`.
131+
132+
### Recommended implementation
133+
Use the AssetRoot properties if they are defined. Example (adapted from [CLI's build/BundledRuntimes.props](https://github.com/dotnet/cli/blob/4fe4c4d28a5171946311ca3ebf65af95180eb11f/build/BundledRuntimes.props)):
134+
135+
```xml
136+
<Project ToolsVersion="15.0" xmlns="http://schemas.microsoft.com/developer/msbuild/2003">
137+
<PropertyGroup>
138+
<!-- SharedFrameworkVersion is defined elsewhere by a Core-Setup package version. -->
139+
<CombinedFrameworkHostCompressedFileName>dotnet-runtime-$(SharedFrameworkVersion)-$(SharedFrameworkRid)$(ArchiveExtension)</CombinedFrameworkHostCompressedFileName>
140+
</PropertyGroup>
141+
142+
<PropertyGroup>
143+
<CoreSetupBlobRootUrl Condition="'$(DotNetAssetRootUrl)' != ''">$(DotNetAssetRootUrl)</CoreSetupBlobRootUrl>
144+
<CoreSetupBlobRootUrl Condition="'$(CoreSetupBlobRootUrl)' == ''">https://dotnetcli.azureedge.net/dotnet/</CoreSetupBlobRootUrl>
145+
146+
<CoreSetupRootUrl>$(CoreSetupBlobRootUrl)Runtime/</CoreSetupRootUrl>
147+
148+
<CoreSetupBlobAccessTokenParam Condition="'$(DotNetAssetRootAccessTokenSuffix)' != ''">$(DotNetAssetRootAccessTokenSuffix)</CoreSetupBlobAccessTokenParam>
149+
</PropertyGroup>
150+
151+
<ItemGroup>
152+
<_DownloadAndExtractItem Include="CombinedSharedHostAndFrameworkArchive">
153+
<Url>$(CoreSetupRootUrl)$(SharedFrameworkVersion)/$(CombinedFrameworkHostCompressedFileName)$(CoreSetupBlobAccessTokenParam)</Url>
154+
</_DownloadAndExtractItem>
155+
</ItemGroup>
156+
</Project>
157+
```
158+
159+
The above assumes that Core-Setup publishes outputs to `$(DotNetOutputBlobFeedDir)assets/Runtime/<SharedFrameworkVersion>/`.
160+
161+
## `-p:DotNetAssetRootAccessTokenSuffix=<query string>`
162+
Directs the repo to append `query string` to any URL constructed using the `DotNetAssetRootUrl`.
163+
164+
Recommended implementation is included in the `DotNetAssetRootUrl` section.
165+
166+
167+
# Output placement arguments
168+
169+
## `-p:DotNetOutputBlobFeedDir=<target directory>`
170+
Directs the repo to place all outputs in the blob feed located at `target directory`, following blob feed structure.
171+
172+
### Recommended implementation
173+
Place NuGet library packages (`*.nupkg`) directly in `$(DotNetOutputBlobFeedDir)packages/`.
174+
175+
Place all other assets, such as installers, lzma files, and NuGet symbol packages (`*.symbols.nupkg`), in `$(DotNetOutputBlobFeedDir)assets/`.
176+
177+
BuildTools can be configured to do this by setting the `PackageOutputPath` and `SymbolPackageOutputPath` MSBuild properties.

‎Documentation/auto-dependency-flow/example-prototype.md

Lines changed: 14 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,14 @@
1+
# Automatic dependency contracts examples/prototypes
2+
3+
**Status: WIP.** Implementing changes based on design reviews.
4+
5+
The auto dependency flow repo API has a prototype implementation in CoreCLR, CoreFX, Core-Setup, and Standard. The prototypes can be used to guide further implementations, and for those repos in particular, they might be directly usable.
6+
7+
In general, the `flow` branch on the `dagood` fork of each repo contains the prototype changes. They are based on `release/2.0.0`.
8+
9+
* CoreCLR [`flow` diff](https://github.com/dotnet/coreclr/compare/release/2.0.0...dagood:flow)
10+
* CoreFX [`flow` diff](https://github.com/dotnet/corefx/compare/release/2.0.0...dagood:flow)
11+
* Standard [`flow` diff](https://github.com/dotnet/standard/compare/release/2.0.0...dagood:flow)
12+
* (Initial implementation: not a clean build) Core-Setup [`flow` diff](https://github.com/dotnet/core-setup/compare/release/2.0.0...dagood:flow)
13+
14+
Prototype source-build orchestration, including up to date submodules: Source-Build [`flow` diff](https://github.com/dotnet/source-build/compare/dev/release/2.0...dagood:flow).

0 commit comments

Comments
 (0)