Skip to content

NixOS (alternative source download methods?) #1887

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Closed
leo60228 opened this issue Nov 16, 2020 · 8 comments
Closed

NixOS (alternative source download methods?) #1887

leo60228 opened this issue Nov 16, 2020 · 8 comments

Comments

@leo60228
Copy link

NixOS is somewhat unique. We currently use the binary releases of the .NET SDK, though we would prefer not to do this. source-build seems close to what we want, though I'm not sure how to handle these:

  • NixOS does not follow the FHS. Because of this, binaries compiled on other distributions need to be patched before executing. Since source-build builds from source, I don't think this is a problem.
  • Source downloads and package builds are separate steps, with the former being expected to do as little work as possible and the latter not having network access. I can't find any way to download sources separately from the build.
  • The same source files should always be downloaded for the same package. This is enforced via a hash. In other words, we need to be able to pin specific commits to download (or some other immutable identifier).
  • It's preferable, though not required, that source is downloaded through NixOS infrastructure. The most obvious solution here is to have a list of files/git repos that need to be downloaded to specific locations.

I feel like other distros might have hit some ot all of these. However, I can't find solutions documented anywhere.

@leo60228
Copy link
Author

Oh, duh, I didn't realize that the documentation was different from the README... Being able to generate a source tarball alone is a partial solution. Are these source tarballs available anywhere? If so, that would solve other issues too.

@eerhardt
Copy link
Member

cc @dagood @dseefeld

@omajid
Copy link
Member

omajid commented Nov 16, 2020

I package source-build for Fedora. Hopefully I can provide some answers you are looking for.

NixOS does not follow the FHS. Because of this, binaries compiled on other distributions need to be patched before executing. Since source-build builds from source, I don't think this is a problem.

It shouldn't matter. dotnet is mostly standalone. As long as the entire SDK+Runtime is installed at one location on-disk, everything should be fine. In Fedora, for example, /usr/bin/dotnet is just a symlink to /usr/lib64/dotnet/ (the main .NET Core installation directory).

Are these source tarballs available anywhere? If so, that would solve other issues too.

No, unfortunately. See #298

For Fedora, I build source-build on locally on a Fedora box and then use the just-produced source-tarball for further builds in the official Fedora build system.

The same source files should always be downloaded for the same package. This is enforced via a hash. In other words, we need to be able to pin specific commits to download (or some other immutable identifier).

This is going to be harder in the current infrastructure. There's no obvious way to map a specific commit id to a specific tarball. Rebuilding the tarball might produce different results too.

Can you file a separate issue for this?

It's preferable, though not required, that source is downloaded through NixOS infrastructure. The most obvious solution here is to have a list of files/git repos that need to be downloaded to specific locations.

#298 takes care of this too, right?

@leo60228
Copy link
Author

This is going to be harder in the current infrastructure. There's no obvious way to map a specific commit id to a specific tarball. Rebuilding the tarball might produce different results too.

I think having #298 should solve this? My phrasing might have been confusing. All that's necessary is that the results of whatever download step we run are always the same. Assuming that a given source tarball URL is immutable, this should be fine.

@leo60228
Copy link
Author

It shouldn't matter. dotnet is mostly standalone. As long as the entire SDK+Runtime is installed at one location on-disk, everything should be fine. In Fedora, for example, /usr/bin/dotnet is just a symlink to /usr/lib64/dotnet/ (the main .NET Core installation directory).

The .NET binaries depend on several libraries in /lib, which doesn't exist on NixOS at all. It's pretty easy to patch the binaries to point to the correct paths, though we do need to have an opportunity to patch the binaries in between them being downloaded and them being ran. Today this is the code that does that: https://github.com/NixOS/nixpkgs/blob/master/pkgs/development/compilers/dotnet/build-dotnet.nix

@dagood
Copy link
Member

dagood commented Nov 16, 2020

All that's necessary is that the results of whatever download step we run are always the same. Assuming that a given source tarball URL is immutable, this should be fine.

Yeah, I expect they'll be published 1:1 to a release tag, which we are conscious about not mutating.

we do need to have an opportunity to patch the binaries in between them being downloaded and them being ran.

You might be able to patch something around here: eng/common/tools.sh#L172-L197, where we download the prebuilt/bootstrap/stage0 .NET SDK. (If you think your patch can be upstreamed, note that this script is maintained at https://github.com/dotnet/arcade/blob/master/eng/common/tools.sh and we just have a copy in this repo.)

@dseefeld
Copy link
Contributor

dseefeld commented Jan 7, 2021

@leo60228 Did this answer your questions?

@dseefeld
Copy link
Contributor

Assuming questions are answered. Closing as resolved.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

5 participants