NixOS (alternative source download methods?)

[leo60228]

NixOS is somewhat unique. We currently use the binary releases of the .NET SDK, though we would prefer not to do this. source-build seems close to what we want, though I'm not sure how to handle these:

• NixOS does not follow the FHS. Because of this, binaries compiled on other distributions need to be patched before executing. Since source-build builds from source, I don't think this is a problem.
• Source downloads and package builds are separate steps, with the former being expected to do as little work as possible and the latter not having network access. I can't find any way to download sources separately from the build.
• The same source files should always be downloaded for the same package. This is enforced via a hash. In other words, we need to be able to pin specific commits to download (or some other immutable identifier).
• It's preferable, though not required, that source is downloaded through NixOS infrastructure. The most obvious solution here is to have a list of files/git repos that need to be downloaded to specific locations.

I feel like other distros might have hit some ot all of these. However, I can't find solutions documented anywhere.

[leo60228]

Oh, duh, I didn't realize that the documentation was different from the README... Being able to generate a source tarball alone is a partial solution. Are these source tarballs available anywhere? If so, that would solve other issues too.

[eerhardt]

cc @dagood @dseefeld

[omajid]

I package source-build for Fedora. Hopefully I can provide some answers you are looking for.

NixOS does not follow the FHS. Because of this, binaries compiled on other distributions need to be patched before executing. Since source-build builds from source, I don't think this is a problem.

It shouldn't matter. dotnet is mostly standalone. As long as the entire SDK+Runtime is installed at one location on-disk, everything should be fine. In Fedora, for example, /usr/bin/dotnet is just a symlink to /usr/lib64/dotnet/ (the main .NET Core installation directory).

Are these source tarballs available anywhere? If so, that would solve other issues too.

No, unfortunately. See #298

For Fedora, I build source-build on locally on a Fedora box and then use the just-produced source-tarball for further builds in the official Fedora build system.

The same source files should always be downloaded for the same package. This is enforced via a hash. In other words, we need to be able to pin specific commits to download (or some other immutable identifier).

This is going to be harder in the current infrastructure. There's no obvious way to map a specific commit id to a specific tarball. Rebuilding the tarball might produce different results too.

Can you file a separate issue for this?

It's preferable, though not required, that source is downloaded through NixOS infrastructure. The most obvious solution here is to have a list of files/git repos that need to be downloaded to specific locations.

#298 takes care of this too, right?

[leo60228]

This is going to be harder in the current infrastructure. There's no obvious way to map a specific commit id to a specific tarball. Rebuilding the tarball might produce different results too.

I think having #298 should solve this? My phrasing might have been confusing. All that's necessary is that the results of whatever download step we run are always the same. Assuming that a given source tarball URL is immutable, this should be fine.

[leo60228]

It shouldn't matter. dotnet is mostly standalone. As long as the entire SDK+Runtime is installed at one location on-disk, everything should be fine. In Fedora, for example, /usr/bin/dotnet is just a symlink to /usr/lib64/dotnet/ (the main .NET Core installation directory).

The .NET binaries depend on several libraries in /lib, which doesn't exist on NixOS at all. It's pretty easy to patch the binaries to point to the correct paths, though we do need to have an opportunity to patch the binaries in between them being downloaded and them being ran. Today this is the code that does that: https://github.com/NixOS/nixpkgs/blob/master/pkgs/development/compilers/dotnet/build-dotnet.nix

[dagood]

All that's necessary is that the results of whatever download step we run are always the same. Assuming that a given source tarball URL is immutable, this should be fine.

Yeah, I expect they'll be published 1:1 to a release tag, which we are conscious about not mutating.

we do need to have an opportunity to patch the binaries in between them being downloaded and them being ran.

You might be able to patch something around here: eng/common/tools.sh#L172-L197, where we download the prebuilt/bootstrap/stage0 .NET SDK. (If you think your patch can be upstreamed, note that this script is maintained at https://github.com/dotnet/arcade/blob/master/eng/common/tools.sh and we just have a copy in this repo.)

[dseefeld]

@leo60228 Did this answer your questions?

[dseefeld]

Assuming questions are answered. Closing as resolved.