Resolve dual 'source of truth' for publishing info

[mmitche]

Right now we have two sources of truth for publishing information. We have the standard arcade publishing routines that involve Publishing.props, shipping/non-shipping info, relative paths for blobs, etc. Then we have the source-build method, which involves a couple targets (GetCategorizedIntermediateNupkgContents). This method isn't the source of truth for the product at the moment, and also doesn't preserve a lot of interesting information like relative blob path layouts, what assets are produced by a repo leg (it assumes zips files and nupkgs). We could add that kind of info in there, but we would just end up duplicating existing info. It also doesn't generate manifests that are usable by downstream infra like the staging pipelines.

This needs to get resolved. Ideally, we would use the same infra for both VMR and individual repo builds. We would go through the same standard arcade paths for publishing, generating the same manifest data, between VMR and individual repo builds.

So basically, when arcade repos build, they runs the normal Publish.proj, creating a manifest using PushToAzureDevOpsArtifacts with the outputs that arcade says it produced. Same for runtime, aspnetcore, etc. this would include the manifest. When we get to the end of the VMR build, we now have 1 manifest for each repo built in each vertical. These get merged together just like any other build.

The important bits here are:

• The VMR needs to be able to build what MSFT builds today, and so it should probably lean on the existing sources of truth to do that.
• The VMR needs to produce outputs that are deployable through the same mechanisms we use to deploy assets today (repo publishing and .NET staging+release).
• The publishing process must remain source-only compatible.

Work Items

• Come up with a shared infrastructure design dotnet#1018

[mmitche]

@NikolaMilosavljevic

[mmitche]

Other notes. The way we do things today leads to differences between VMR and individual builds around how repos are finding upstream input artifacts. For instance: https://github.com/dotnet/installer/blob/main/src/redist/targets/GenerateLayout.targets#L264

Because VMR builds don't really "publish" the outputs of the repo build based on the layout the repo specifies, they just copy them over, a repo downstream will need to check the flag for VMR build, then alter the relative path that it's looking in.

Here's how I generally envision this works:

1. Inner build runs the normal publish step. This gathers up the artifacts in ItemToPushToBlobFeed and whatnot, and calls PushToAzureDevOpsArtifacts.

• We change this task to PushToBuildStorage to reflect its purpose better. It takes a flag that indicates whether it should do an AzDO push (note that this is a vso comment based upload today, not an actual upload), or whether it's running in a VMR build.
• If in a VMR build, we push to a specified shared location on disk (blob-feed). This location preserves desired relative blob path, manifest files, etc..

2. Downstream consumer builds use the shared location, like they do today, but in the same way they use the external shared locations today (via correct relative paths)
3. The end of each VMR vertical gathers all manifest files from all builds and merges them, producing the output of the vertical. This uses the standard AzDO push like we do today (if running in azdo).
4. The end of the full VMR build does a BAR upload with all vertical manifests as input.

The main difference between how repos move artifacts between themselves and this is just that the former has one extra step we don't need:

Repo automation:

Publish To AzDO artifacts -> Goes to Shipping, NonShipping and AssetManifests artifacts -> Call maestro publishing based on default channels -> Gather Build artifacts and push to AzDO feeds + blob storage --> Downstream repos consume based on dep updates

In the VMR case, if you wanted to match this exactly, you'd do:

Publish To Build artifacts -> Goes to Shipping, NonShipping and AssetManifests artifact folders on disk -> Call VMR publishing -> Gather Build artifacts and push to another location on disk --> Downstream repos consume based on those locations

But we don't really need the additional intermediate storage location and so a couple steps are degenerate. We can have Publish To Build artifacts do more work:

Publish To Build artifacts -> Goes to assets folders and feeds on disk, with generate manifests in assets --> Downstream repos consume based on those locations

[ViktorHofer]

But we don't really need the additional intermediate storage location and so a couple steps are degenerate. We can have Publish To Build artifacts do more work:

Couple weeks ago when Nikolai implemented the removal of the intermediate packages in the VMR, we decided that the inner repo's responsibility shouldn't be to lay the files out to a specific location inside the VMR. Instead we created the manifest file that the VMR orchestrator then reads to copy the files to the shared location.

Just to provide context... I think that you propose makes sense and that it should be fine if the Publish step in the inner repo copies the artifacts directly into that shared location.

And I'm all for merging these two different publishing paths. Maintaining both is cumbersome.