Publish crossgen2 as .zip/tar.gz file

[davidwrighton]

• For consumption in dotnet-docker repo

[ghost]

Tagging @ViktorHofer as an area owner

[davidwrighton]

@MichaelSimons @richlander this should make producing the docker images easier for crossgen2 scenarios.

@jkoritzinsky Could you review this or nominate the right reviewer?

[AaronRobinsonMSFT]

I don't think we want to check in binary files. I believe we should instead leverage a NuGet or Azure blob feed for binary data.

[AaronRobinsonMSFT]

@dotnet/arcade-contrib Does Arcade have a solution for binary data?

/cc @jashook

[vatsan-madhavan]

Have you seen this? https://github.com/dotnet/arcade/blob/master/Documentation/NativeToolBootstrapping.md

[hoyosjs]

We've used native tool bootstrapping, and had several issues on non-windows platforms before. I am not aware if they were ever fixed. Also, we are not consuming a zipped tool, but rather trying to create one for consumption. Not sure how it's intended to be used at the other end, but native bootstrapping doesn't sound like the best solution it we're trying to consume latest bits over there. From what I see this logic is going what we do with the runtime: pack build outputs into a zip and try to push it along the dotnet runtime zips (i.e. dotnet-runtime-5.0.100-preview.x.xxxxxx.x-win-x64.zip) to the https://dotnetfeed.blob.core.windows.net/dotnet-core blob. If this is the case I guess including the zip and wixpdb was a mistake of dev inner loop, unless they were also binplaced at the root of the repo for some reason.

As a side note, I see the path of the asset being produced ends up like this:
artifacts/packages/Debug/Shipping/dotnet-runtime-internal-5.0.0-ci-linux-x64.tar.gz. Is this intended to be a shipping asset? And given we already have a wixpdb, is this also generating an msi?

[hoyosjs]

Also, is https://dotnetfeed.blob.core.windows.net/dotnet-core/Runtime/<VERSION>/dotnet-crossgen2-<VERSION>-linux-x64.tar.gz the right blob to publish this to?

[ViktorHofer]

cc @MattGal @chcosta for thoughts on where this blob should be stored.

[mmitche]

If it's published as part of the blob artifacts of the repo (like the rest of the msi's and zips that are produced in this repo), it will end up in blob storage as part of the publishing process. The location shouldn't be hardcoded in the build at all.

https://github.com/dotnet/runtime/blob/master/src/installer/publish/prepare-artifacts.proj#L111 shows where the zips, msis, etc. are picked up by PushToAzureDevOpsArtifacts and then those will make their way through publishing later and end up on various feeds. In the case of the .NET 5 channels, these end up in the dotnetcli storage account, under the dotnet container.

The downstream builds would then download those from the target location, constructing the path based on the current runtime version number.

[davidwrighton]

Oops, didn't mean to put anything binary in there.

[davidwrighton]

The goal with this work is to place crossgen2 in location which can be used by our dotnet-docker DockerFile build process. See https://github.com/dotnet/dotnet-docker/blob/nightly/5.0/runtime/alpine3.11/amd64/Dockerfile for an example of how the dotnet-docker repo reads these things. At the moment, I'm not sure if we will need to use crossgen2 in our docker file creation path or not, but that's the model we're planning to use as we prototype using crossgen2 to optimize our Linux docker experience.

[tannergooding]

Oops, didn't mean to put anything binary in there.

Should we have *.zip, and *.gz in the .gitignore to help avoid these issues?

[jkoritzinsky]

@davidwrighton do you want a self-contained or framework-dependent setup for crossgen2 in the compressed file? Currently you have the framework-dependent build. To get a self-contained build, you'll have to do a little bit more MSBuild-foo to wire everything up (since we have to manually assemble the self-contained output as we go).

[mmitche]

The goal with this work is to place crossgen2 in location which can be used by our dotnet-docker DockerFile build process. See https://github.com/dotnet/dotnet-docker/blob/nightly/5.0/runtime/alpine3.11/amd64/Dockerfile for an example of how the dotnet-docker repo reads these things. At the moment, I'm not sure if we will need to use crossgen2 in our docker file creation path or not, but that's the model we're planning to use as we prototype using crossgen2 to optimize our Linux docker experience.

Yep, and it would show up in that location on the CDN:

    && wget -O dotnet.tar.gz https://dotnetcli.azureedge.net/dotnet/Runtime/$dotnet_version/crossgen2-whatever-version.tar.gz \

[hoyosjs]

The goal with this work is to place crossgen2 in location which can be used by our dotnet-docker DockerFile build process. See https://github.com/dotnet/dotnet-docker/blob/nightly/5.0/runtime/alpine3.11/amd64/Dockerfile for an example of how the dotnet-docker repo reads these things. At the moment, I'm not sure if we will need to use crossgen2 in our docker file creation path or not, but that's the model we're planning to use as we prototype using crossgen2 to optimize our Linux docker experience.

Given the version is matching for runtime + crossgen2, as long as crossgen2 is invoked with that host, all will work. Then this all looks ok.

[davidwrighton]

@jkoritzinsky @hoyosjs a framework dependent build is fine. Docker production has access to the matching framework.

[MichaelSimons]

Has anyone validated the resulting .zip/tar.gz files? I took a look at the latest build drop and the files are there but they are only ~8 KB. The one I inspected only contained the license and TPN.

https://dotnetcli.azureedge.net/dotnet/Runtime/5.0.0-preview.4.20209.4/dotnet-crossgen2-5.0.0-preview.4.20209.4-linux-musl-x64.tar.gz

[davidwrighton]

I validated that when I built locally it had stuff in it, but I the build lab appears to work differently. @jkoritzinsky, could you give me some guidance here?