Not possible to build source-build release/3.0 on new distributions

[omajid]

Darc needs to know about the runtime-id in advance since it it needs to find a libgit2sharp for the current pltaform.

On Fedora 30, this fails:

git clone https://github.com/dotnet/source-build
cd source-build
git checkout release/3.0
git submodule update --init --recursive
./build.sh

The end of the output shows why:

  System.Exception: Something went wrong when cloning repo https://github.com/dotnet/arcade at <default branch> into /
home/omajid/devel/dotnet/source-build/bin/src/arcade ---> System.TypeInitializationException: The type initializer for
 'LibGit2Sharp.Core.NativeMethods' threw an exception. ---> System.DllNotFoundException: Unable to load shared library
 'git2-8e0b172' or one of its dependencies. In order to help diagnose loading problems, consider setting the LD_DEBUG 
environment variable: libgit2-8e0b172: cannot open shared object file: No such file or directory
     at LibGit2Sharp.Core.NativeMethods.git_libgit2_init()
     at LibGit2Sharp.Core.NativeMethods.LoadNativeLibrary() in C:\projects\libgit2sharp\LibGit2Sharp\Core\NativeMethos.cs:line 47
     at LibGit2Sharp.Core.NativeMethods..cctor() in C:\projects\libgit2sharp\LibGit2Sharp\Core\NativeMethods.cs:line 37
     --- End of inner exception stack trace ---
     at LibGit2Sharp.Core.NativeMethods.git_clone(git_repository*& repo, String origin_url, FilePath workdir_path, GitCloneOptions& opts)
     at LibGit2Sharp.Core.Proxy.git_clone(String url, String workdir, GitCloneOptions& opts) in C:\projects\libgit2sharp\LibGit2Sharp\Core\Proxy.cs:line 354
     at LibGit2Sharp.Repository.Clone(String sourceUrl, String workdirPath, CloneOptions options) in C:\projects\libgit2sharp\LibGit2Sharp\Repository.cs:line 715
     at Microsoft.DotNet.DarcLib.RemoteRepoBase.Clone(String repoUri, String commit, String targetDirectory, ILogger _logger, String pat, String gitDirectory) in /home/omajid/devel/dotnet/source-build/tools-local/arcade-services/src/Microsoft.DotNet.Darc/src/DarcLib/RemoteRepoBase.cs:line 176
     --- End of inner exception stack trace ---
     at Microsoft.DotNet.DarcLib.RemoteRepoBase.Clone(String repoUri, String commit, String targetDirectory, ILogger _logger, String pat, String gitDirectory) in /home/omajid/devel/dotnet/source-build/tools-local/arcade-services/src/Microsoft.DotNet.Darc/src/DarcLib/RemoteRepoBase.cs:line 210
     at Microsoft.DotNet.DarcLib.GitHubClient.Clone(String repoUri, String commit, String targetDirectory, String gitDirectory) in /home/omajid/devel/dotnet/source-build/tools-local/arcade-services/src/Microsoft.DotNet.Darc/src/DarcLib/GitHubClient.cs:line 875
     at Microsoft.DotNet.Darc.Operations.CloneOperation.HandleMasterCopyAndCreateGitDir(RemoteFactory remoteFactory, String repoUrl, String masterGitRepoPath, String masterRepoGitDirPath, String gitDirRedirect, ILogger log) in /home/omajid/devel/dotnet/source-build/tools-local/arcade-services/src/Microsoft.DotNet.Darc/src/Darc/Operations/CloneOperation.cs:line 282
     at Microsoft.DotNet.Darc.Operations.CloneOperation.HandleMasterCopyWithGitDirPath(RemoteFactory remoteFactory, String repoUrl, String masterGitRepoPath, String masterRepoGitDirPath, ILogger log) in /home/omajid/devel/dotnet/source-build/tools-local/arcade-services/src/Microsoft.DotNet.Darc/src/Darc/Operations/CloneOperation.cs:line 270
     at Microsoft.DotNet.Darc.Operations.CloneOperation.HandleMasterCopy(RemoteFactory remoteFactory, String repoUrl, String masterGitRepoPath, String masterRepoGitDirPath, ILogger log) in /home/omajid/devel/dotnet/source-build/tools-local/arcade-services/src/Microsoft.DotNet.Darc/src/Darc/Operations/CloneOperation.cs:line 225
     at Microsoft.DotNet.Darc.Operations.CloneOperation.ExecuteAsync() in /home/omajid/devel/dotnet/source-build/tools-local/arcade-services/src/Microsoft.DotNet.Darc/src/Darc/Operations/CloneOperation.cs:line 102

[tmds]

libgit2sharp has an so file for fedora under the fedora-x64 rid:

$ ldd /home/tmds/.nuget/packages/libgit2sharp.nativebinaries/1.0.235/runtimes/fedora-x64/native/libgit2-8e0b172.so 
	linux-vdso.so.1 (0x00007ffeaa91c000)
	librt.so.1 => /lib64/librt.so.1 (0x00007f4013b41000)
	libpthread.so.0 => /lib64/libpthread.so.0 (0x00007f4013b20000)
	libcurl.so.4 => /lib64/libcurl.so.4 (0x00007f4013a8e000)
	libssl.so.1.1 => /lib64/libssl.so.1.1 (0x00007f40139f8000)
	libcrypto.so.1.1 => /lib64/libcrypto.so.1.1 (0x00007f4013718000)
	libz.so.1 => /lib64/libz.so.1 (0x00007f40136fe000)
	libc.so.6 => /lib64/libc.so.6 (0x00007f4013536000)
	/lib64/ld-linux-x86-64.so.2 (0x00007f4013e55000)
	libnghttp2.so.14 => /lib64/libnghttp2.so.14 (0x00007f401350d000)
	libidn2.so.0 => /lib64/libidn2.so.0 (0x00007f40134ed000)
	libssh.so.4 => /lib64/libssh.so.4 (0x00007f4013469000)
	libpsl.so.5 => /lib64/libpsl.so.5 (0x00007f4013456000)
	libgssapi_krb5.so.2 => /lib64/libgssapi_krb5.so.2 (0x00007f4013404000)
	libkrb5.so.3 => /lib64/libkrb5.so.3 (0x00007f4013311000)
	libk5crypto.so.3 => /lib64/libk5crypto.so.3 (0x00007f40132f3000)
	libcom_err.so.2 => /lib64/libcom_err.so.2 (0x00007f40132ec000)
	libldap-2.4.so.2 => /lib64/libldap-2.4.so.2 (0x00007f401329b000)
	liblber-2.4.so.2 => /lib64/liblber-2.4.so.2 (0x00007f401328a000)
	libbrotlidec.so.1 => /lib64/libbrotlidec.so.1 (0x00007f401327b000)
	libdl.so.2 => /lib64/libdl.so.2 (0x00007f4013273000)
	libunistring.so.2 => /lib64/libunistring.so.2 (0x00007f40130ef000)
	libkrb5support.so.0 => /lib64/libkrb5support.so.0 (0x00007f40130dd000)
	libkeyutils.so.1 => /lib64/libkeyutils.so.1 (0x00007f40130d6000)
	libresolv.so.2 => /lib64/libresolv.so.2 (0x00007f40130bc000)
	libsasl2.so.3 => /lib64/libsasl2.so.3 (0x00007f401309a000)
	libbrotlicommon.so.1 => /lib64/libbrotlicommon.so.1 (0x00007f4013077000)
	libselinux.so.1 => /lib64/libselinux.so.1 (0x00007f401304a000)
	libcrypt.so.2 => /lib64/libcrypt.so.2 (0x00007f401300f000)
	libpcre2-8.so.0 => /lib64/libpcre2-8.so.0 (0x00007f4012f89000)

that one would work, but instead it is using the linux-x64 rid:

$ ldd /home/tmds/.nuget/packages/libgit2sharp.nativebinaries/1.0.235/runtimes/linux-x64/native/libgit2-8e0b172.so
/home/tmds/.nuget/packages/libgit2sharp.nativebinaries/1.0.235/runtimes/linux-x64/native/libgit2-8e0b172.so: /lib64/libcurl.so.4: no version information available (required by /home/tmds/.nuget/packages/libgit2sharp.nativebinaries/1.0.235/runtimes/linux-x64/native/libgit2-8e0b172.so)
	linux-vdso.so.1 (0x00007ffd0a7a0000)
	librt.so.1 => /lib64/librt.so.1 (0x00007f8c3e285000)
	libpthread.so.0 => /lib64/libpthread.so.0 (0x00007f8c3e264000)
	libcurl.so.4 => /lib64/libcurl.so.4 (0x00007f8c3e1d2000)
	libssl.so.1.0.0 => not found
	libcrypto.so.1.0.0 => not found
	libz.so.1 => /lib64/libz.so.1 (0x00007f8c3e1b8000)
	libc.so.6 => /lib64/libc.so.6 (0x00007f8c3dff0000)
	/lib64/ld-linux-x86-64.so.2 (0x00007f8c3e5bf000)
	libnghttp2.so.14 => /lib64/libnghttp2.so.14 (0x00007f8c3dfc7000)
	libidn2.so.0 => /lib64/libidn2.so.0 (0x00007f8c3dfa7000)
	libssh.so.4 => /lib64/libssh.so.4 (0x00007f8c3df23000)
	libpsl.so.5 => /lib64/libpsl.so.5 (0x00007f8c3df10000)
	libssl.so.1.1 => /lib64/libssl.so.1.1 (0x00007f8c3de7a000)
	libcrypto.so.1.1 => /lib64/libcrypto.so.1.1 (0x00007f8c3db98000)
	libgssapi_krb5.so.2 => /lib64/libgssapi_krb5.so.2 (0x00007f8c3db46000)
	libkrb5.so.3 => /lib64/libkrb5.so.3 (0x00007f8c3da55000)
	libk5crypto.so.3 => /lib64/libk5crypto.so.3 (0x00007f8c3da37000)
	libcom_err.so.2 => /lib64/libcom_err.so.2 (0x00007f8c3da30000)
	libldap-2.4.so.2 => /lib64/libldap-2.4.so.2 (0x00007f8c3d9df000)
	liblber-2.4.so.2 => /lib64/liblber-2.4.so.2 (0x00007f8c3d9cc000)
	libbrotlidec.so.1 => /lib64/libbrotlidec.so.1 (0x00007f8c3d9bd000)
	libunistring.so.2 => /lib64/libunistring.so.2 (0x00007f8c3d839000)
	libdl.so.2 => /lib64/libdl.so.2 (0x00007f8c3d833000)
	libkrb5support.so.0 => /lib64/libkrb5support.so.0 (0x00007f8c3d821000)
	libkeyutils.so.1 => /lib64/libkeyutils.so.1 (0x00007f8c3d818000)
	libresolv.so.2 => /lib64/libresolv.so.2 (0x00007f8c3d7fe000)
	libsasl2.so.3 => /lib64/libsasl2.so.3 (0x00007f8c3d7de000)
	libbrotlicommon.so.1 => /lib64/libbrotlicommon.so.1 (0x00007f8c3d7bb000)
	libselinux.so.1 => /lib64/libselinux.so.1 (0x00007f8c3d78e000)
	libcrypt.so.2 => /lib64/libcrypt.so.2 (0x00007f8c3d751000)
	libpcre2-8.so.0 => /lib64/libpcre2-8.so.0 (0x00007f8c3d6cb000)

which has missing libraries for openssl.

If you set DOTNET_RUNTIME_ID to fedora-x64, things may work.

[fmauNeko]

Hello,

I am having the same issue under Arch Linux, copying the fedora-x64 .so file over the linux-x64 one works.

This is because libgit2sharp.nativebinaries builds its linux-x64 binary using Ubuntu 14.04 (https://github.com/libgit2/libgit2sharp.nativebinaries/blob/master/Dockerfile.linux-x64).
As Ubuntu 14.04 reached EOL on April 30, 2019, an issue should be opened upstream.

[tmds]

If you set DOTNET_RUNTIME_ID to fedora-x64, things may work.

@omajid did you give this a try?

This issue also relates to providing a complete source tarball per release (cfr #298). The issue happens here because dotnet is used to fetch sources, dependencies, etc. Such things would already be packed in the source tarball.

CC @dagood @dseefeld

[dagood]

SourceLink removed its dependency on this native binary with dotnet/sourcelink#288, maybe similar could be done for Darc? My understanding is that ./build.sh from a clone is no longer expected to work on unprepared distros, but would be nice to expand support by removing dependencies like this IMO.

/cc @crummel

[omajid]

If you set DOTNET_RUNTIME_ID to fedora-x64, things may work.

@omajid did you give this a try?

No, I didn't. fedora.30-x64 is now in the runtime id graph so things Just Worked.

[omajid]

No, I didn't. fedora.30-x64 is now in the runtime id graph so things Just Worked.

Oops. Turns out I was wrong. I built source build to create the release tarball on Fedora 28. I then built from the tarball (which doesn't need git?) on Fedora 30. The issue still persists.

This Dockerfile reproduces the problem on any machine:

FROM fedora:30

RUN dnf install --setopt tsflags=nodocs --refresh -y \
        clang \
        cmake \
        compat-openssl10 \
        curl-devel \
        findutils \
        git \
        hostname \
        krb5-devel \
        libicu-devel \
        libunwind-devel \
        lldb-devel \
        llvm \
        lttng-ust-devel \
        make \
        openssl-devel \
        python3 \
        tar \
        wget \
        which \
        zlib-devel && \
    dnf upgrade -y && \
    dnf clean all -y

CMD git clone https://github.com/dotnet/source-build && \
        cd source-build && \
        git checkout release/3.0 && \
        git submodule update --init --recursive && \
        ./build.sh

[tmds]

I then built from the tarball (which doesn't need git?)

Yes, this matches my earlier comment: #1083 (comment).

Try setting DOTNET_RUNTIME_ID.

[omajid]

Try setting DOTNET_RUNTIME_ID.

Yes, that works. But it doesn't address the real issue that results from having very specific runtime ids, no real runtime id fallback mechanism, and bundling native libraries.

[tmds]

no real runtime id fallback mechanism

This is indeed an issue.

Even with a fallback mechanism, source-build wouldn't work on some platforms.
I'm considering source-build's output the source for building .NET Core rather than source-build itself (because source-build relies on dotnet to build).

[dseefeld]

If we acquire DARC rather than build it, this should be solved. @crummel to confirm.

[omajid]

Recent versions of sourcelink don't depend on libgit2sharp: dotnet/sourcelink#288. We just need to use a recent enough version and then we can use it everywhere without issues.

Edit: I just realized that arcade-services doesn't use SourceLink, it uses libgit2sharp directly for git operations.

[dseefeld]

If we acquire DARC rather than build it, this should be solved. @crummel to confirm.

@crummel Will #1041 take care of this issue?

Even with a fallback mechanism, source-build wouldn't work on some platforms.
I'm considering source-build's output the source for building .NET Core rather than source-build itself (because source-build relies on dotnet to build).

@ericstj Can you comment on the above?