Package name reccomendation for source-build-reference-packages

[omajid]

I have been looking into creating a Linux distro package for source-build-reference-packages. Is there a name suggestion that the source-build folks (as the upstream for Linux distros) would recommend/suggest?

For context, this is the naming guideline for Fedora. It suggests asking upstream and/or coming up with a common consensus on the general package name.

We discussed this on IRC #fedora-dotnet and a few comments popped up:

• source-build isn't a great prefix, it's unclear to other Fedora devs what that means
• The name should indicate that it is a build artifact only. It's not meant for release. But we can't really make things completely internal in Fedora. All packages are available to users.
• It's used for all builds, not just bootstrap ones

And some suggestions for the package name:

• dotnet-bootstrap-reference-packages
• dotnet-build-common
• dotnet-build-packages
• dotnet-build-reference-packages
• dotnet-reference-packages
• dotnet-source-build-reference-packages
• source-build-reference-packages

cc @aslicerh @directhex @dseefeld @RheaAyase

[adaggarwal]

Shall we consider it as an Addon Package? In that case dotnet-source-build-reference-packages might adhere with the convention IMO.

[tmds]

This is about the code in https://github.com/dotnet/source-build-reference-packages?

I don't understand why it needs a separate package, vs getting consumed by source-build like other repos (coreclr, corefx, ...)? @dagood @crummel can you make me a bit wiser?

[dseefeld]

@tmds Yes. This is about the code in that repo. These are historical reference packages for building dotnet from source. We chose not to build these with source-build for two reasons: 1) they're fairly static, so we don't expect much change between builds and 2) we didn't want to increase build time.

[tmds]

There is some complexity to not building these with source-build. You need a different way to build them (since they must be buildable from source). And source-build needs to be able to consume them. These bring some complexity.

If I install this package, where can I expect files to end up? Will this populate the DOTNET_ROOT/packs folder?

[dseefeld]

There is some complexity, but I think it's worth having these separate, since they're not part of the SDK. Regarding where the files end up, I'm not sure about that. This package is required for build only. It's not required when the SDK gets installed. I've been looking through packaging guidelines to see if there's a suggested location for these, but haven't found anything, yet.

[omajid]

For the sake of playing around, I have thrown them in $DOTNET_ROOT/reference-packages/Private.SourceBuild.ReferencePackages.${version}.tar.gz.

Which makes me think: should source-build-reference-packages get tags/releases so there's a "version" that things can point to?

[dagood]

I think this is important:

I don't understand why it needs a separate package, vs getting consumed by source-build like other repos (coreclr, corefx, ...)?

we didn't want to increase build time.

I think it would make sense to:

1. During a bootstrap build, always build source-build-reference-packages from source.
   • This trades some build time against setting up a whole new distro package for this unique case (complexity for distro maintainer).
   • We recently fixed some low-hanging fruit to dramatically improve the build performance of source-build-reference-packages--maybe there's more we can do. (@adaggarwal may have build time numbers?)
2. Set up generic build caches that can apply to any repo. (We need improvements in isolation, etc.)
   • If a distro maintainer wishes, they can put caches from their source-build into distro packages so that if the commit doesn't change in the next release, they don't have to rebuild that specific repo.
   • I believe we need something like this cache to reasonably put source-build CI in the repos, too.
3. Take advantage of the generic build cache for source-build-reference-packages.
   • This brings us back to where we are today, but more flexible and providing broader benefits (if a distro wants to use this infra).

[adaggarwal]

1. During a bootstrap build, always build source-build-reference-packages from source.

   • This trades some build time against setting up a whole new distro package for this unique case (complexity for distro maintainer).
   • We recently fixed some low-hanging fruit to dramatically improve the build performance of source-build-reference-packages--maybe there's more we can do.

In the current bootstrap-ci we do build (yml step) source-build-reference-packages
The build time hovers around 16 minutes. We did optimize for targeting reference packages which contributes about 7-8 minutes to the build time. The other bit is reference-packages which looks good, but we can take a look and discuss if it can be further optimized.

[dseefeld]

There are 3 binary things that each source-build build needs besides the source.

1. a built SDK
2. reference packages and
3. previously source-built packages (these are packages from a previous build that don't get included in the SDK).

A distro maintainer will need to create at least 2 distro packages when building dotnet:

1. the package for the SDK, which will be installed when anyone wants to install dotnet core on their machine and
2. a package for the previously source-built packages / reference packages, which will be installed by distro maintainers when doing a subsequent build. They could be in the same package or in different packages.

@dagood what you're proposing would move reference packages from a distro package to the source, but wouldn't eliminate the need for an additional distro package.

[dseefeld]

Which makes me think: should source-build-reference-packages get tags/releases so there's a "version" that things can point to?

I think this is a good idea. I'll work on making this happen.

[dseefeld]

Regarding the original question of what these should be called, @adaggarwal suggested dotnet-source-build-reference-packages which I think is good except for the usage of the term "source-build". I'm not sure source-build adds anything in this context. Since they're a dotnet artifact used only for build, I would suggest dotnet-build-reference-packages.

[dagood]

@dagood what you're proposing would move reference packages from a distro package to the source, but wouldn't eliminate the need for an additional distro package.

It would eliminate the need to consider naming this particular source-build-reference-packages distro package, no? Do we still need to come up with a distro package name for the previously source-built packages?

It sounds like this is close to being complete anyway, and reducing build time is good. If adding 1 vs. 2 distro packages isn't a big deal, that's fine, I mainly wanted to point out that I think it should be replaced with more generic infra down the line.

$DOTNET_ROOT/reference-packages/Private.SourceBuild.ReferencePackages.${version}.tar.gz.

Shouldn't we install the extracted files instead, to avoid having a globally installed compressed copy plus a redundant extracted copy created during source-build? (I suppose we need to make source-build's MSBuild code accept a tar.gz or pre-extracted dir, looks fairly trivial.)

[tmds]

@dseefeld are there other benefits than the reduced build time (15 minutes)?

[omajid]

This trades some build time against setting up a whole new distro package for this unique case (complexity for distro maintainer).

FWIW, this particular package, as it stands now, is pretty simple as far as distro packages go. I have a prototype at https://pagure.io/dotnet-sig/source-build-reference-packages/blob/master/f/source-build-reference-packages.spec and you can see how simple it is compared to something like the SDK.

A distro maintainer will need to create at least 2 distro packages when building dotnet:

1. the package for the SDK, which will be installed when anyone wants to install dotnet core on their machine and
2. a package for the previously source-built packages / reference packages, which will be installed by distro maintainers when doing a subsequent build. They could be in the same package or in different packages.

This might need a slight explanation. Generally, each distro package takes one source input and produces one or more packages. For .NET Core, we have two sets of source repositories, so it seems natural to have two main distro packages: one for source-build and another for source-build-reference-packages. In this scenario, source-build does the entire build and produces binary distro packages for the runtime, the sdk, the host and also the private source-built-packages. source-build-reference-packages is a separate distro package that produces one artifact: the reference packages tarball. So we end up with:

• source-build source package produces

  • dotnet-sdk-3.1
  • dotnet-runtime-3.1
  • ...
  • dotnet-sdk-3.1-source-built-artifacts

• source-build-reference-packages produces:

  • source-build-reference-packages

So the "source-built packages" and "reference packages" are forced to be in two separate distro packages and produced from two separate distro packages.

It would eliminate the need to consider naming this particular source-build-reference-packages distro package, no? Do we still need to come up with a distro package name for the previously source-built packages?

We kind of have to give distro packages a name if we want to put files in them that we then want to use in a later build. If there's a tarball of reference packages that we want to use in future build, we have to be able to somehow say "install that distro package to get the tarball of reference packages". Same goes for previously source-built packages. We do need to name both.

If there's a guidance on those names, that's great. If not, it's not as big a deal as other package names because the packages are for the distro people building the .NET Core SDK. The other names, like dotnet-sdk-3.1 are for end users, which is a much wider audience.

Shouldn't we install the extracted files instead, to avoid having a globally installed compressed copy plus a redundant extracted copy created during source-build? (I suppose we need to make source-build's MSBuild code accept a tar.gz or pre-extracted dir, looks fairly trivial.)

+1 if we can use a plain directory containing nupkg files instead of a single compressed tarball of source-build-reference-packages. That would be a very nice feature.

It would let the distro package handle the compression instead of putting zip/nupkg files inside gzipped-tarballs which then go inside the final distro package.

[dagood]

+1 if we can use a plain directory containing nupkg files instead of a single compressed tarball of source-build-reference-packages. That would be a very nice feature.

Cool, filed #1463. 🙂

It would eliminate the need to consider naming this particular source-build-reference-packages distro package, no? [...]

We kind of have to give distro packages a name if we want to put files in them that we then want to use in a later build.

To clarify what I meant a bit, what I was suggesting is removing the need to use the files in a later build. (Just regenerate them like any other repo in the Runtime/SDK source-build.) What I was getting at was that a package that doesn't exist doesn't need a name. No objection at all to having a name recommendation.

FWIW, I think dotnet-build-reference-packages is fine as @dseefeld pointed to. Having dotnet in the name seems fairly important to associate it with the project, and having it as the prefix makes sense to me.

There's a remote possibility that someone can confuse it as something to do with the dotnet build command. We could go with full dotnet-source-build-reference-packages if that seems concerning. (I'm indifferent to including source- with or without this consideration.)

[adaggarwal]

FWIW, I think dotnet-build-reference-packages is fine as @dseefeld pointed to. Having dotnet in the name seems fairly important to associate it with the project, and having it as the prefix makes sense to me.

Yes, I think this name does make sense. source is not something that may be necessary. Initially, I wanted to include that just to have an associate between the two packages. But the package being built from source is a given, so it may not be necessary to include it in the name.

[dseefeld]

+1 if we can use a plain directory containing nupkg files instead of a single compressed tarball of source-build-reference-packages. That would be a very nice feature.

Cool, filed #1463. 🙂

This already exists. See #1434. This adds the ability to supply both a path to the previously source-built artifacts as well as a path to reference packages.

So the "source-built packages" and "reference packages" are forced to be in two separate distro packages and produced from two separate distro packages.

I didn't realize this. As mentioned before, these are separated out to decrease build time and because of low frequency of changes. If it is too much of a burden for distro maintainers, we can look into moving them into source-build.

[omajid]

If it is too much of a burden for distro maintainers, we can look into moving them into source-build.

With my Fedora distro maintainer hat on: In this case, it really isn't. It's a standalone project that can be built separately fairly easily.

I linked the prototype spec file I am testing: https://pagure.io/dotnet-sig/source-build-reference-packages/blob/master/f/source-build-reference-packages.spec. It's much simpler than the one for SDK.

[dseefeld]

OK. So I think this is complete. We've concluded that the package name should be dotnet-build-reference-packages.

[dagood]

Should we update https://docs.microsoft.com/en-us/dotnet/core/distribution-packaging as part of the work for this issue?

[tmds]

I'm missing context to understand why this change is important.

[adaggarwal]

@tmds I think it goes back to the points @omajid mentioned in the beginning :

• source-build isn't a great prefix, it's unclear to other Fedora devs what that means
• The name should indicate that it is a build artifact only. It's not meant for release. But we can't really make things completely internal in Fedora. All packages are available to users.
• It's used for all builds, not just bootstrap ones

[tmds]

@adaggarwal my question is about splitting this into a separate package, not about the name.

[dagood]

Filed dotnet/docs#16874 to track documentation for dotnet-build-reference-packages and dotnet-sdk-3.1-source-built-artifacts. (We can continue to focus on discussing whether both are required here.)

[dagood]

(We can continue to focus on discussing whether both are required here.)

While working on the 5.0 Arcade-powered source-build plan, we saw that source-build-reference-packages (SBRP) would be a good place to work on changes to standardize how source-build works in Arcade. It's a leaf repo, it produces fairly straightforward artifacts, and our team is the only one with a stake in it.

There's a significant quirk, though: SBRP currently builds as its own distro package, and it has a complicated relationship with the dotnet/source-build repo. In order to be a model "standard" Arcade-powered source-build repo, we shouldn't preserve this quirk because no other repo will have it.

So, for 5.0, I filed #1591, which will turn SBRP into a normal Arcade-using repo and eliminate this distro package, going forward.

Doing this in 5.0 doesn't force it in 3.1, but it's desirable to get 3.1 using the same approach:

• It reduces maintenance to keep a single branch in SBRP.
• Keeping the overall source-build process the same in 3.1 vs. 5.0 simplifies setup for new distros. This might be particularly useful as 3.1 is LTS: perhaps new distros will want to onboard 3.1, not just 5.0. (Perhaps until we get 6.0 LTS.)
• It simplifies servicing releases, and makes it easier to diagnose/fix issues since the build doesn't span two sets of sources. Inputs are also potentially more predictable.

Since the only reason for the split in the first place was to cache the SBRP packages between builds, I think this is fine to apply in both places. Build time doesn't currently seem unsustainable at < 20 minutes (in CI). But I also do expect some build time improvements in 5.0 based on Arcadification cleanup forcing some extra parallelism.

If tarball build time ends up being impossible in 5.0, I think we should be able to produce a more general solution in Arcade for distro package splits, rather than a one-off implementation for SBRP.

I think we should:

1. Branch SBRP and leave 3.1 alone until 5.0 Arcade-powered source-build is running with N-1 and bootstrap scenarios working.
2. Track it as a 3.1 servicing improvement to get 3.1 using the same strategy as 5.0.
3. Ideally, remove the SBRP 3.1 branch.
   • This may end up being too difficult to do across 3.1 and 5.0 Arcade toolset versions, but it seems like we should at least investigate.

[crummel]

[Scrub] Is this all covered by https://github.com/dotnet/source-build/blob/main/Documentation/packaging-installation.md ?