Consider adding a shipping package baseline test

[HaoK]

As part of the postmortem for a 5.0.1 issue where we needed to mark Identity.Specficiation.Tests as shipping. It seems worth adding a test with baselines to track the total set of packages we ship, similar to how we track the list of assemblies in the shared fx here: https://github.com/dotnet/aspnetcore/blob/master/src/Framework/test/TestData.cs.

The analogy would be something that just enumerates all the nupkgs in the shipping artifacts and fails if there's a mismatch from the list so we can track/be explicitly intentional about changes in the packages we ship.

Thanks for contacting us.
We're moving this issue to the Next sprint planning milestone for future evaluation / consideration. We will evaluate the request when we are planning the work for the next milestone. To learn more about what to expect next and how this issue will be handled you can read more about our triage process here.

[JunTaoLuo]

Post-build hook in arcade would be useful in creating a check for this. https://github.com/dotnet/aspnetcore/blob/master/eng/AfterSolutionBuild.targets

[dougbu]

I'm not sure what would be needed here nor how additional automation would help. If we need to check prior to the release, manifest files e.g. \vsufile\patches\sign\NET\CORE_BUILDS\2.1.X\2.1.31\20210916-02\assets\orchestration-metadata\manifests\aspnet.xml, \vsufile\patches\sign\NET\CORE_BUILDS\3.1.X\3.1.20\3.1.120-servicing-015384\manifest.json, or \vsufile\patches\sign\NET\NET_CORE\5.0.11\configs\manifests\manifest.json are available when internal builds are announced.

Looking back, the eng/Baseline.xml file e.g. https://github.com/dotnet/aspnetcore/blob/release/2.1/eng/Baseline.xml and (for 2.1) https://github.com/dotnet/versions/blob/main/build-info/dotnet/product/cli/release/2.1/build.xml and similar files are available after every release.

/cc @mkArtakMSFT because this relates to validating fixes are in a release

[dougbu]

/btw John's suggestion won't help because we always ship everything in every release, with the minor exception of targeting packs. Where we do incremental shipment (in release/2.1), Arcade isn't involved.

[dougbu]

In any case, I have no way to know when we'd do this, if do something, and cleared the milestone for now. Suggestions for "maybe" items @adityamandaleeka @rafikiassumani-msft @mkArtakMSFT @Pilchie❔