Package name reccomendation for source-build-reference-packages

[omajid]

I have been looking into creating a Linux distro package for source-build-reference-packages. Is there a name suggestion that the source-build folks (as the upstream for Linux distros) would recommend/suggest?

For context, this is the naming guideline for Fedora. It suggests asking upstream and/or coming up with a common consensus on the general package name.

We discussed this on IRC #fedora-dotnet and a few comments popped up:

• source-build isn't a great prefix, it's unclear to other Fedora devs what that means
• The name should indicate that it is a build artifact only. It's not meant for release. But we can't really make things completely internal in Fedora. All packages are available to users.
• It's used for all builds, not just bootstrap ones

And some suggestions for the package name:

• dotnet-bootstrap-reference-packages
• dotnet-build-common
• dotnet-build-packages
• dotnet-build-reference-packages
• dotnet-reference-packages
• dotnet-source-build-reference-packages
• source-build-reference-packages

cc @aslicerh @directhex @dseefeld @RheaAyase

[adaggarwal]

Shall we consider it as an Addon Package? In that case dotnet-source-build-reference-packages might adhere with the convention IMO.

[tmds]

This is about the code in https://github.com/dotnet/source-build-reference-packages?

I don't understand why it needs a separate package, vs getting consumed by source-build like other repos (coreclr, corefx, ...)? @dagood @crummel can you make me a bit wiser?

[dseefeld]

@tmds Yes. This is about the code in that repo. These are historical reference packages for building dotnet from source. We chose not to build these with source-build for two reasons: 1) they're fairly static, so we don't expect much change between builds and 2) we didn't want to increase build time.

[tmds]

There is some complexity to not building these with source-build. You need a different way to build them (since they must be buildable from source). And source-build needs to be able to consume them. These bring some complexity.

If I install this package, where can I expect files to end up? Will this populate the DOTNET_ROOT/packs folder?

[dseefeld]

There is some complexity, but I think it's worth having these separate, since they're not part of the SDK. Regarding where the files end up, I'm not sure about that. This package is required for build only. It's not required when the SDK gets installed. I've been looking through packaging guidelines to see if there's a suggested location for these, but haven't found anything, yet.

[omajid]

For the sake of playing around, I have thrown them in $DOTNET_ROOT/reference-packages/Private.SourceBuild.ReferencePackages.${version}.tar.gz.

Which makes me think: should source-build-reference-packages get tags/releases so there's a "version" that things can point to?

[dagood]

I think this is important:

I don't understand why it needs a separate package, vs getting consumed by source-build like other repos (coreclr, corefx, ...)?

we didn't want to increase build time.

I think it would make sense to:

1. During a bootstrap build, always build source-build-reference-packages from source.
   • This trades some build time against setting up a whole new distro package for this unique case (complexity for distro maintainer).
   • We recently fixed some low-hanging fruit to dramatically improve the build performance of source-build-reference-packages--maybe there's more we can do. (@adaggarwal may have build time numbers?)
2. Set up generic build caches that can apply to any repo. (We need improvements in isolation, etc.)
   • If a distro maintainer wishes, they can put caches from their source-build into distro packages so that if the commit doesn't change in the next release, they don't have to rebuild that specific repo.
   • I believe we need something like this cache to reasonably put source-build CI in the repos, too.
3. Take advantage of the generic build cache for source-build-reference-packages.
   • This brings us back to where we are today, but more flexible and providing broader benefits (if a distro wants to use this infra).

[adaggarwal]

1. During a bootstrap build, always build source-build-reference-packages from source.

   • This trades some build time against setting up a whole new distro package for this unique case (complexity for distro maintainer).
   • We recently fixed some low-hanging fruit to dramatically improve the build performance of source-build-reference-packages--maybe there's more we can do.

In the current bootstrap-ci we do build (yml step) source-build-reference-packages
The build time hovers around 16 minutes. We did optimize for targeting reference packages which contributes about 7-8 minutes to the build time. The other bit is reference-packages which looks good, but we can take a look and discuss if it can be further optimized.

[dseefeld]

There are 3 binary things that each source-build build needs besides the source.

1. a built SDK
2. reference packages and
3. previously source-built packages (these are packages from a previous build that don't get included in the SDK).

A distro maintainer will need to create at least 2 distro packages when building dotnet:

1. the package for the SDK, which will be installed when anyone wants to install dotnet core on their machine and
2. a package for the previously source-built packages / reference packages, which will be installed by distro maintainers when doing a subsequent build. They could be in the same package or in different packages.

@dagood what you're proposing would move reference packages from a distro package to the source, but wouldn't eliminate the need for an additional distro package.

[dseefeld]

Which makes me think: should source-build-reference-packages get tags/releases so there's a "version" that things can point to?

I think this is a good idea. I'll work on making this happen.

[dseefeld]

Regarding the original question of what these should be called, @adaggarwal suggested dotnet-source-build-reference-packages which I think is good except for the usage of the term "source-build". I'm not sure source-build adds anything in this context. Since they're a dotnet artifact used only for build, I would suggest dotnet-build-reference-packages.

[dagood]

@dagood what you're proposing would move reference packages from a distro package to the source, but wouldn't eliminate the need for an additional distro package.

It would eliminate the need to consider naming this particular source-build-reference-packages distro package, no? Do we still need to come up with a distro package name for the previously source-built packages?

It sounds like this is close to being complete anyway, and reducing build time is good. If adding 1 vs. 2 distro packages isn't a big deal, that's fine, I mainly wanted to point out that I think it should be replaced with more generic infra down the line.

$DOTNET_ROOT/reference-packages/Private.SourceBuild.ReferencePackages.${version}.tar.gz.

Shouldn't we install the extracted files instead, to avoid having a globally installed compressed copy plus a redundant extracted copy created during source-build? (I suppose we need to make source-build's MSBuild code accept a tar.gz or pre-extracted dir, looks fairly trivial.)

[tmds]

@dseefeld are there other benefits than the reduced build time (15 minutes)?

[omajid]

This trades some build time against setting up a whole new distro package for this unique case (complexity for distro maintainer).

FWIW, this particular package, as it stands now, is pretty simple as far as distro packages go. I have a prototype at https://pagure.io/dotnet-sig/source-build-reference-packages/blob/master/f/source-build-reference-packages.spec and you can see how simple it is compared to something like the SDK.

A distro maintainer will need to create at least 2 distro packages when building dotnet:

1. the package for the SDK, which will be installed when anyone wants to install dotnet core on their machine and
2. a package for the previously source-built packages / reference packages, which will be installed by distro maintainers when doing a subsequent build. They could be in the same package or in different packages.

This might need a slight explanation. Generally, each distro package takes one source input and produces one or more packages. For .NET Core, we have two sets of source repositories, so it seems natural to have two main distro packages: one for source-build and another for source-build-reference-packages. In this scenario, source-build does the entire build and produces binary distro packages for the runtime, the sdk, the host and also the private source-built-packages. source-build-reference-packages is a separate distro package that produces one artifact: the reference packages tarball. So we end up with:

• source-build source package produces

  • dotnet-sdk-3.1
  • dotnet-runtime-3.1
  • ...
  • dotnet-sdk-3.1-source-built-artifacts

• source-build-reference-packages produces:

  • source-build-reference-packages

So the "source-built packages" and "reference packages" are forced to be in two separate distro packages and produced from two separate distro packages.

It would eliminate the need to consider naming this particular source-build-reference-packages distro package, no? Do we still need to come up with a distro package name for the previously source-built packages?

We kind of have to give distro packages a name if we want to put files in them that we then want to use in a later build. If there's a tarball of reference packages that we want to use in future build, we have to be able to somehow say "install that distro package to get the tarball of reference packages". Same goes for previously source-built packages. We do need to name both.

If there's a guidance on those names, that's great. If not, it's not as big a deal as other package names because the packages are for the distro people building the .NET Core SDK. The other names, like dotnet-sdk-3.1 are for end users, which is a much wider audience.

Shouldn't we install the extracted files instead, to avoid having a globally installed compressed copy plus a redundant extracted copy created during source-build? (I suppose we need to make source-build's MSBuild code accept a tar.gz or pre-extracted dir, looks fairly trivial.)

+1 if we can use a plain directory containing nupkg files instead of a single compressed tarball of source-build-reference-packages. That would be a very nice feature.

It would let the distro package handle the compression instead of putting zip/nupkg files inside gzipped-tarballs which then go inside the final distro package.