Mitigate the NuGet restore behavior

[jaredpar]

There is a NuGet restore issue which is failing a significant number of our CI / PR builds (as of this time about 6%).

• EvaluateCacheFile is causing restore crashes on Linux NuGet/Home#9199

It will likely be several weeks before an actual NuGet fix can make it to our repository (just following standard code flow). That's too long given the magnitude of the failure here.

Short term though there does appear to be a viable work around for us. The linux based jobs need to add a step before arcade build which:

1. Creates /tmp/NuGetScratch (using the appropriate temp variable)
2. Run chmod -R ugo+rwx on the created directory

This should remove the restore error until the actual fix comes along.

Places we need to insert this script call:

• https://github.com/dotnet/runtime/blob/master/eng/pipelines/libraries/build-job.yml#L87
• https://github.com/dotnet/runtime/blob/master/eng/pipelines/coreclr/templates/build-job.yml#L21
• https://github.com/dotnet/runtime/blob/master/eng/pipelines/mono/templates/build-job.yml#L57

---

Symptoms of the bug:

src/libraries/Directory.Build.props#L38
src/libraries/Directory.Build.props(38,3): error : System.AggregateException: One or more errors occurred. (Could not find a part of the path '/var/folders/24/8k48jl6d249_n_qfxwsl6xvm0000gn/T/NuGetScratch/ccd9d22adb6549f086362689e42245d6/9c159c0681d8495c9d48c5e0d296844d.proj.nuget.dgspec.json'.)
 ---> System.IO.DirectoryNotFoundException: Could not find a part of the path '/var/folders/24/8k48jl6d249_n_qfxwsl6xvm0000gn/T/NuGetScratch/ccd9d22adb6549f086362689e42245d6/9c159c0681d8495c9d48c5e0d296844d.proj.nuget.dgspec.json'.
   at Interop.ThrowExceptionForIoErrno(ErrorInfo errorInfo, String path, Boolean isDirectory, Func`2 errorRewriter)
   at Microsoft.Win32.SafeHandles.SafeFileHandle.Open(String path, OpenFlags flags, Int32 mode)
   at System.IO.FileStream..ctor(String path, FileMode mode, FileAccess access, FileShare share, Int32 bufferSize, FileOptions options)
   at System.IO.FileStream..ctor(String path, FileMode mode)
   at NuGet.ProjectModel.DependencyGraphSpec.Save(String path)
   at NuGet.Commands.NoOpRestoreUtilities.PersistDGSpecFile(DependencyGraphSpec spec, String dgPath, ILogger log)
   at NuGet.Commands.RestoreCommand.EvaluateCacheFile()
   at NuGet.Commands.RestoreCommand.ExecuteAsync(CancellationToken token)
   at NuGet.Commands.RestoreRunner.ExecuteAsync(RestoreSummaryRequest summaryRequest, CancellationToken token)
   at NuGet.Commands.RestoreRunner.CompleteTaskAsync(List`1 restoreTasks)
   at NuGet.Commands.RestoreRunner.RunWithoutCommit(IEnumerable`1 restoreRequests, RestoreArgs restoreContext)
   --- End of inner exception stack trace ---
   at System.Threading.Tasks.Task`1.GetResultCore(Boolean waitCompletionNotification)
   at Microsoft.Build.NuGetSdkResolver.NuGetSdkResolver.NuGetAbstraction.GetSdkResult(SdkReference sdk, Object nuGetVersion, SdkResolverContext context, SdkResultFactory factory)

[jaredpar]

@safern would you be able to take a look at this? It's killing a large number of our builds ATM.

[CoffeeFlux]

Hit in #32502 https://dev.azure.com/dnceng/public/_build/results?buildId=540591&view=logs&jobId=40f773ae-1f50-5e5e-c198-773a224261de&j=40f773ae-1f50-5e5e-c198-773a224261de&t=67eb29bf-94a4-5bc4-21bc-5f2ac1aca0aa

[jaredpar]

@CoffeeFlux yeah it's hitting roughly 6% of all builds in dotnet/runtime across all our build definitions. Having trouble finding someone with sufficient Unix skills + AzDO understanding to get this worked around. Typical group is otherwise engaged at the moment.

[CoffeeFlux]

Gotcha, unfortunate. There's no way to get a backport or something from the Nuget team? It's particularly annoying since it seems I can't restart the lane until everything is done thanks to the way AzDO works, and it's an early failure.

[jaredpar]

@CoffeeFlux

The NuGet fix is available now. I'm guessing it will take another few days for this to flow to arcade and then to runtime. Given how deeply it's integrated into the SDK it's not an easy piece to isolate and update separately.

Best path is for us to implement the work around. It's targeted and fairly straight forward.

[jaredpar]

Mitigation is ineffective. Have to wait on the official bits to flow. Keeping this open to track the underlying problem.