NuGetFallbackFolder adds unnecessary bloat to build containers

[natemcmaster]

In .NET Core 2.0, the CLI started shipping a feature that adds an offline package feed called the NuGetFallbackFolder. This adds unnecessary bloat to SDK images in the following ways:

• ~660 MB of xmldocs are stored in ~/.dotnet/NuGetFallbackFolder. xmldocs are unnecessary in build images.
• first run of dotnet-restore creates about 5,700 duplicate files which adds about 326 MB. This duplication is caused by copying files from ~/.dotnet/NuGetFallbackFolder to ~/.nuget/packages
• Any users that build inside the image and don't clear out the ~/.nuget/packages folder end up with a bloated image. Here is the resulting duplication between the nuget cache and fallback folders:

  fdupes -r -m ~/.dotnet/NuGetFallbackFolder/ ~/.nuget/packages/
  11439 duplicate files (in 2809 sets), occupying 876.3 megabytes
  

Suggestions

• Exclude xmldocs from NuGet fallback folder
• Disable the offline package cache feature altogether.
• Warmup the NuGet cache by running dotnet-restore as a part of the image creation.

Or some combination of the ideas above.

Details
How I measured the numbers
Using: microsoft/dotnet-nightly:2.0.0-preview1-sdk

• Measure the effect of a dotnet-restore
  1. Install fdupes tool. apt-get update -qq && apt-get install -y fdupes
  2. mkdir /app && cd /app
  3. ls ~/.nuget/. The ~/.nuget/packages/ folder does not yet exist.
  4. dotnet new web (automatically launches dotnet restore)
  5. fdupes -r -m ~/.dotnet/NuGetFallbackFolder/ ~/.nuget/packages/
  6. Count files added to nuget cache: find ~/.nuget/packages -type f | wc -l
• Count xmldoc files
  find ~/.dotnet -name "*.xml" -type f | wc -l
• Count xmldoc filesize:
  find ~/.dotnet/ -name "*.xml" -type f -print0 | du -sb --files0-from=- | awk '{ total += $1} END { print total/1024/1024 }' (number is in MB)

cc @emgarten @rrelyea @dotnet/dotnet-cli @glennc

[emgarten]

This will improve once the fallback folder is used as an actual fallback folder.

Long term to solve this I'd like to see either a dotnet or NuGet command that takes a set of project.assets.json files and prunes out all unneeded files from the package folders. This would include assemblies from unused frameworks, all dlls and files not listed in the assets file, and resource dlls that aren't used.

[natemcmaster]

This will improve once the fallback folder is used as an actual fallback folder.

@emgarten is this just a preview1 limitation that will be fixed before 2.0.0 RTM?

Also FYI the ~/.dotnet/NuGetFallbackFolder on Linux takes up about 1.0 GB.

[emgarten]

@emgarten is this just a preview1 limitation that will be fixed before 2.0.0 RTM?

That is the current goal

[tmds]

Does this help?

ENV NUGET_XMLDOC_MODE=skip
ENV DOTNET_SKIP_FIRST_TIME_EXPERIENCE=1

[natemcmaster]

The images already set NUGET_XMLDOC_MODE=skip, but it's not honored by the first-time experience. If we decided to disable the offline package cache, setting DOTNET_SKIP_FIRST_TIME_EXPERIENCE is the best way to do that.

[tmds]

Is the /opt/dotnet/store also used during dotnet restore to avoid duplicating those packages to ~/.nuget/packages?

[natemcmaster]

If you meant /usr/share/dotnet/store, that is the runtime store feature that's new to 2.0. These are a subset of files from the ~/.nuget/packages folder that have been crossgened for faster boot time, similar to the "optimization cache" feature in .NET Core 1.x. The subset doesn't include all assets required to build an app.

[tmds]

The subset doesn't include all assets required to build an app.

Ah. So this doesn't get used by dotnet restore and the sdk image won't contain a store since it would only bloat the image.

[MichaelSimons]

@emgarten, What is the latest status of the underlying issue? Can you link to the product issue?

[emgarten]

@MichaelSimons it is tracked here: https://github.com/dotnet/cli/issues/6507

[natemcmaster]

It looks like the fallback folder behavior is working now. However, I think there is still an opportunity here to trim some bloat. If I've done my bash/math right, here are the (uncompressed) size of files in /usr/share/dotnet

[MichaelSimons]

I logged a cli issue to add support for this scenario. There are numerous use cases which would benefit from this trimming capability. Our Dockerfiles should not have to define this trimming logic.

[JeanMdK]

The NuGetFallbackFolder is actually taking to much room in my disk : since it's only caching packages to avoid searching them online if they're not there, can we just remove all the files from the folder ? Is it safe ? Thank you for your time,
John

[natemcmaster]

Hey @MichaelSimons, I'd like to take a stab at this for 2.2 (after #662 is in, of course). We started producing a version of the fallback folder which trims the .nupkg and .xml files which is the biggest part of the (uncompressed) bloat.

@JeanMdK yes, you can safely delete this folder. NuGet will download what you need. You are likely to find, however, that this will just shift content from the NuGetFallbackFolder to %USERPROFILE%/.nuget/packages.

[MichaelSimons]

@natemcmaster - Can you provide more details on this lighter fallback folder? Is this a published artifact I can take a look at? @richlander and I have discussed this and are in favor of this for 2.2.

[natemcmaster]

Yes, there are published artifacts for this starting in 2.1.3. I've written up more details about the various package archives we produce in aspnet/Universe#1324. For Docker, you would want to use the nuGetPackagesArchive-ci-server-$(Version).zip. Example: http://dotnetcli.blob.core.windows.net/dotnet/aspnetcore/Runtime/2.1.3/nuGetPackagesArchive-ci-server-2.1.3.zip. This contains the same contents as the LZMA bundled in the SDK, but without the .nupkg and docxml files.

Under the hood, NuGet restore takes a list of fallback folders in the MSBuild property RestoreAdditionalProjectFallbackFolders. By default, the .NET Core SDK adds $(DotNetInstallRoot)/sdk/NuGetFallbackFolder/ to this list. (See Microsoft.NET.NuGetOfflineCache.targets).

There is more than one way to do this, but I would recommend the following for 2.2 previews. I'm like 80% sure this should work, but in full disclosure, I haven't tried it on all platforms, just in Azure services.

• after extracting dotnet-sdk.{tgz/zip}, delete sdk/$(SdkVersion)/nuGetPackagesArchive.lzma. The first-run experience will skip the fallback folder creation when this file is gone (and it saves ~36 MB on disk)
• expand nuGetPackagesArchive-ci-server-$(RuntimeVersion).zip to sdk/NuGetFallbackFolder/

[MichaelSimons]

The SDK is removing the lzma/NuGetFallbackFolder in 3.0 and making use of targeting packs instead. The removal has already happened.

Closing this issue, there are no plans to port any of this work back to previous versions.