[mono] Restore failure due to `The process cannot access the file x because it is being used by another process`

[ayakael]

Description

When building runtime with mono-flavored runtime, restore often fails in CI environment with The process cannot access the file x because it is being used by another process. This affects runtime, as well as other builds (confirmed roslyn), thus specificity to runtime is that this does not occur on coreclr-flavored runtime.

Reproduction Steps

In an Alpine Edge linux-musl-x64 environment, you can use this modified aport to reproduce bug. Following steps to reproduce:

git clone https://gitlab.alpinelinux.org/ayakael/aports -b dotnet7/mono-restore
cd  aports/testing/dotnet7-stage0
abuild deps unpack prepare build

It should eventually fail.

The aport builds a minimum set of components (runtime-mono, roslyn, sdk, aspnetcore, installer) to be able to build an SDK tar, then using that produced tarball with mono-flavored runtime it builds the whole stack again. This aport is usually used to crossbuild to other platforms, but in this case I am using it to easily reproduce the bug.

You'd likely be able to reproduce this on linux-x64 by building runtime with /p:PrimaryRuntimeFlavor=Mono` and trying to build runtime with produced artifacts.

Expected behavior

Restore should occur without issue

Actual behavior

Restore fails with the following error:

/builds/ayakael/aports/testing/dotnet7-stage0/src/bootstrap/sdk/7.0.100-rtm.22519.39/NuGet.RestoreEx.targets(19,5): error : The process cannot access the file '/builds/ayakael/aports/testing/dotnet7-stage0/src/dotnet-d41bfecf5090e9163aa2da251246abed9e756e53/src/runtime/artifacts/obj/System.IO.Ports' because it is being used by another process. [/builds/ayakael/aports/testing/dotnet7-stage0/src/dotnet-d41bfecf5090e9163aa2da251246abed9e756e53/src/runtime/Build.proj]
/builds/ayakael/aports/testing/dotnet7-stage0/src/bootstrap/sdk/7.0.100-rtm.22519.39/NuGet.RestoreEx.targets(19,5): error :    at System.IO.FileSystem.CreateDirectory(String fullPath, UnixFileMode unixCreateMode) [/builds/ayakael/aports/testing/dotnet7-stage0/src/dotnet-d41bfecf5090e9163aa2da251246abed9e756e53/src/runtime/Build.proj]
/builds/ayakael/aports/testing/dotnet7-stage0/src/bootstrap/sdk/7.0.100-rtm.22519.39/NuGet.RestoreEx.targets(19,5): error :    at System.IO.FileSystem.CreateDirectory(String fullPath) [/builds/ayakael/aports/testing/dotnet7-stage0/src/dotnet-d41bfecf5090e9163aa2da251246abed9e756e53/src/runtime/Build.proj]
/builds/ayakael/aports/testing/dotnet7-stage0/src/bootstrap/sdk/7.0.100-rtm.22519.39/NuGet.RestoreEx.targets(19,5): error :    at System.IO.Directory.CreateDirectory(String path) [/builds/ayakael/aports/testing/dotnet7-stage0/src/dotnet-d41bfecf5090e9163aa2da251246abed9e756e53/src/runtime/Build.proj]
/builds/ayakael/aports/testing/dotnet7-stage0/src/bootstrap/sdk/7.0.100-rtm.22519.39/NuGet.RestoreEx.targets(19,5): error :    at NuGet.Commands.BuildAssetsUtils.WriteFiles(IEnumerable`1 files, ILogger log) [/builds/ayakael/aports/testing/dotnet7-stage0/src/dotnet-d41bfecf5090e9163aa2da251246abed9e756e53/src/runtime/Build.proj]
/builds/ayakael/aports/testing/dotnet7-stage0/src/bootstrap/sdk/7.0.100-rtm.22519.39/NuGet.RestoreEx.targets(19,5): error :    at NuGet.Commands.RestoreResult.CommitAssetsFileAsync(LockFileFormat lockFileFormat, IRestoreResult result, ILogger log, Boolean toolCommit, CancellationToken token) [/builds/ayakael/aports/testing/dotnet7-stage0/src/dotnet-d41bfecf5090e9163aa2da251246abed9e756e53/src/runtime/Build.proj]
/builds/ayakael/aports/testing/dotnet7-stage0/src/bootstrap/sdk/7.0.100-rtm.22519.39/NuGet.RestoreEx.targets(19,5): error :    at NuGet.Commands.RestoreResult.CommitAsync(ILogger log, CancellationToken token) [/builds/ayakael/aports/testing/dotnet7-stage0/src/dotnet-d41bfecf5090e9163aa2da251246abed9e756e53/src/runtime/Build.proj]
/builds/ayakael/aports/testing/dotnet7-stage0/src/bootstrap/sdk/7.0.100-rtm.22519.39/NuGet.RestoreEx.targets(19,5): error :    at NuGet.Commands.RestoreRunner.CommitAsync(RestoreResultPair restoreResult, CancellationToken token) [/builds/ayakael/aports/testing/dotnet7-stage0/src/dotnet-d41bfecf5090e9163aa2da251246abed9e756e53/src/runtime/Build.proj]
/builds/ayakael/aports/testing/dotnet7-stage0/src/bootstrap/sdk/7.0.100-rtm.22519.39/NuGet.RestoreEx.targets(19,5): error :    at NuGet.Commands.RestoreRunner.ExecuteAndCommitAsync(RestoreSummaryRequest summaryRequest, IRestoreProgressReporter progressReporter, CancellationToken token) [/builds/ayakael/aports/testing/dotnet7-stage0/src/dotnet-d41bfecf5090e9163aa2da251246abed9e756e53/src/runtime/Build.proj]
/builds/ayakael/aports/testing/dotnet7-stage0/src/bootstrap/sdk/7.0.100-rtm.22519.39/NuGet.RestoreEx.targets(19,5): error :    at NuGet.Commands.RestoreRunner.CompleteTaskAsync(List`1 restoreTasks) [/builds/ayakael/aports/testing/dotnet7-stage0/src/dotnet-d41bfecf5090e9163aa2da251246abed9e756e53/src/runtime/Build.proj]
/builds/ayakael/aports/testing/dotnet7-stage0/src/bootstrap/sdk/7.0.100-rtm.22519.39/NuGet.RestoreEx.targets(19,5): error :    at NuGet.Commands.RestoreRunner.RunAsync(IEnumerable`1 restoreRequests, RestoreArgs restoreArgs, CancellationToken token) [/builds/ayakael/aports/testing/dotnet7-stage0/src/dotnet-d41bfecf5090e9163aa2da251246abed9e756e53/src/runtime/Build.proj]
/builds/ayakael/aports/testing/dotnet7-stage0/src/bootstrap/sdk/7.0.100-rtm.22519.39/NuGet.RestoreEx.targets(19,5): error :    at NuGet.Commands.RestoreRunner.RunAsync(RestoreArgs restoreContext, CancellationToken token) [/builds/ayakael/aports/testing/dotnet7-stage0/src/dotnet-d41bfecf5090e9163aa2da251246abed9e756e53/src/runtime/Build.proj]
/builds/ayakael/aports/testing/dotnet7-stage0/src/bootstrap/sdk/7.0.100-rtm.22519.39/NuGet.RestoreEx.targets(19,5): error :    at NuGet.Build.Tasks.BuildTasksUtility.RestoreAsync(DependencyGraphSpec dependencyGraphSpec, Boolean interactive, Boolean recursive, Boolean noCache, Boolean ignoreFailedSources, Boolean disableParallel, Boolean force, Boolean forceEvaluate, Boolean hideWarningsAndErrors, Boolean restorePC, Boolean cleanupAssetsForUnsupportedProjects, ILogger log, CancellationToken cancellationToken) [/builds/ayakael/aports/testing/dotnet7-stage0/src/dotnet-d41bfecf5090e9163aa2da251246abed9e756e53/src/runtime/Build.proj]
/builds/ayakael/aports/testing/dotnet7-stage0/src/bootstrap/sdk/7.0.100-rtm.22519.39/NuGet.RestoreEx.targets(19,5): error :    at NuGet.Build.Tasks.Console.MSBuildStaticGraphRestore.RestoreAsync(String entryProjectFilePath, IDictionary`2 globalProperties, IReadOnlyDictionary`2 options) [/builds/ayakael/aports/testing/dotnet7-stage0/src/dotnet-d41bfecf5090e9163aa2da251246abed9e756e53/src/runtime/Build.proj]

Regression?

This error is confirmed not present with .NET runtime 6.0.10 in s390x, thus it seems to be a regression.

Known Workarounds

None so far.

Configuration

Version: SDK 7.0.100-rtm.22519.39
OS: Alpine LInux Edge docker-based CI Pipelines (although occurs to a lesser degree in s390x VM)
Architecure: Confirmed reproducible in s390x, ppc64le and x64 (with mono-flavored runtime)
Specificity: specific to mono on all platforms

Other information

logs
ppc64le with runtime: link
s390x with runtime: link
s390x with roslyn: link

I couldn't figure out the best area label to add to this issue. If you have write-permissions please help me learn by adding exactly one area label.

[ayakael]

This is confirmed reproducible on x64. I've updated the post for steps to reproduce on x64.

[ayakael]

False alarm - updating to latest commit of release/7.0.1xx on installer did the trick.

[ayakael]

Never mind, indeed it happens less, but it still occurs enough to make our build fail consistently:

/builds/ayakael/aports/testing/dotnet7-build/src/dotnet-e6dd91c290b808f971a1ac69c2fb29395bbf1051/Tools/source-built/Microsoft.DotNet.Arcade.Sdk/tools/SourceBuild/SourceBuildArcadeBuild.targets(116,5): error MSB3026: Could not copy "/builds/ayakael/aports/testing/dotnet7-build/src/dotnet-e6dd91c290b808f971a1ac69c2fb29395bbf1051/src/fsharp/vsintegration/src/FSharp.ProjectSystem.PropertyPages/PropertyPages/xlf/ApplicationPropPage.de.xlf" to "/builds/ayakael/aports/testing/dotnet7-build/src/dotnet-e6dd91c290b808f971a1ac69c2fb29395bbf1051/src/fsharp/artifacts/source-build/self/src/vsintegration/src/FSharp.ProjectSystem.PropertyPages/PropertyPages/xlf/ApplicationPropPage.de.xlf". Beginning retry 1 in 1000ms. The process cannot access the file '/builds/ayakael/aports/testing/dotnet7-build/src/dotnet-e6dd91c290b808f971a1ac69c2fb29395bbf1051/src/fsharp/artifacts/source-build/self/src/vsintegration/src/FSharp.ProjectSystem.PropertyPages/PropertyPages/xlf' because it is being used by another process.  [/builds/ayakael/aports/testing/dotnet7-build/src/dotnet-e6dd91c290b808f971a1ac69c2fb29395bbf1051/Tools/source-built/Microsoft.DotNet.Arcade.Sdk/tools/Build.proj]

Tagging subscribers to this area: @dotnet/runtime-infrastructure
See info in area-owners.md if you want to be subscribed.

Issue Details

---

Description

When building runtime with mono-flavored runtime, restore often fails in CI environment with The process cannot access the file x because it is being used by another process. This affects runtime, as well as other builds (confirmed roslyn), thus specificity to runtime is that this does not occur on coreclr-flavored runtime.

Reproduction Steps

In an Alpine Edge linux-musl-x64 environment, you can use this modified aport to reproduce bug. Following steps to reproduce:

git clone https://gitlab.alpinelinux.org/ayakael/aports -b dotnet7/mono-restore
cd  aports/testing/dotnet7-stage0
abuild deps unpack prepare build

It should eventually fail.

The aport builds a minimum set of components (runtime-mono, roslyn, sdk, aspnetcore, installer) to be able to build an SDK tar, then using that produced tarball with mono-flavored runtime it builds the whole stack again. This aport is usually used to crossbuild to other platforms, but in this case I am using it to easily reproduce the bug.

You'd likely be able to reproduce this on linux-x64 by building runtime with /p:PrimaryRuntimeFlavor=Mono` and trying to build runtime with produced artifacts.

Expected behavior

Restore should occur without issue

Actual behavior

Restore fails with the following error:

/builds/ayakael/aports/testing/dotnet7-stage0/src/bootstrap/sdk/7.0.100-rtm.22519.39/NuGet.RestoreEx.targets(19,5): error : The process cannot access the file '/builds/ayakael/aports/testing/dotnet7-stage0/src/dotnet-d41bfecf5090e9163aa2da251246abed9e756e53/src/runtime/artifacts/obj/System.IO.Ports' because it is being used by another process. [/builds/ayakael/aports/testing/dotnet7-stage0/src/dotnet-d41bfecf5090e9163aa2da251246abed9e756e53/src/runtime/Build.proj]
/builds/ayakael/aports/testing/dotnet7-stage0/src/bootstrap/sdk/7.0.100-rtm.22519.39/NuGet.RestoreEx.targets(19,5): error :    at System.IO.FileSystem.CreateDirectory(String fullPath, UnixFileMode unixCreateMode) [/builds/ayakael/aports/testing/dotnet7-stage0/src/dotnet-d41bfecf5090e9163aa2da251246abed9e756e53/src/runtime/Build.proj]
/builds/ayakael/aports/testing/dotnet7-stage0/src/bootstrap/sdk/7.0.100-rtm.22519.39/NuGet.RestoreEx.targets(19,5): error :    at System.IO.FileSystem.CreateDirectory(String fullPath) [/builds/ayakael/aports/testing/dotnet7-stage0/src/dotnet-d41bfecf5090e9163aa2da251246abed9e756e53/src/runtime/Build.proj]
/builds/ayakael/aports/testing/dotnet7-stage0/src/bootstrap/sdk/7.0.100-rtm.22519.39/NuGet.RestoreEx.targets(19,5): error :    at System.IO.Directory.CreateDirectory(String path) [/builds/ayakael/aports/testing/dotnet7-stage0/src/dotnet-d41bfecf5090e9163aa2da251246abed9e756e53/src/runtime/Build.proj]
/builds/ayakael/aports/testing/dotnet7-stage0/src/bootstrap/sdk/7.0.100-rtm.22519.39/NuGet.RestoreEx.targets(19,5): error :    at NuGet.Commands.BuildAssetsUtils.WriteFiles(IEnumerable`1 files, ILogger log) [/builds/ayakael/aports/testing/dotnet7-stage0/src/dotnet-d41bfecf5090e9163aa2da251246abed9e756e53/src/runtime/Build.proj]
/builds/ayakael/aports/testing/dotnet7-stage0/src/bootstrap/sdk/7.0.100-rtm.22519.39/NuGet.RestoreEx.targets(19,5): error :    at NuGet.Commands.RestoreResult.CommitAssetsFileAsync(LockFileFormat lockFileFormat, IRestoreResult result, ILogger log, Boolean toolCommit, CancellationToken token) [/builds/ayakael/aports/testing/dotnet7-stage0/src/dotnet-d41bfecf5090e9163aa2da251246abed9e756e53/src/runtime/Build.proj]
/builds/ayakael/aports/testing/dotnet7-stage0/src/bootstrap/sdk/7.0.100-rtm.22519.39/NuGet.RestoreEx.targets(19,5): error :    at NuGet.Commands.RestoreResult.CommitAsync(ILogger log, CancellationToken token) [/builds/ayakael/aports/testing/dotnet7-stage0/src/dotnet-d41bfecf5090e9163aa2da251246abed9e756e53/src/runtime/Build.proj]
/builds/ayakael/aports/testing/dotnet7-stage0/src/bootstrap/sdk/7.0.100-rtm.22519.39/NuGet.RestoreEx.targets(19,5): error :    at NuGet.Commands.RestoreRunner.CommitAsync(RestoreResultPair restoreResult, CancellationToken token) [/builds/ayakael/aports/testing/dotnet7-stage0/src/dotnet-d41bfecf5090e9163aa2da251246abed9e756e53/src/runtime/Build.proj]
/builds/ayakael/aports/testing/dotnet7-stage0/src/bootstrap/sdk/7.0.100-rtm.22519.39/NuGet.RestoreEx.targets(19,5): error :    at NuGet.Commands.RestoreRunner.ExecuteAndCommitAsync(RestoreSummaryRequest summaryRequest, IRestoreProgressReporter progressReporter, CancellationToken token) [/builds/ayakael/aports/testing/dotnet7-stage0/src/dotnet-d41bfecf5090e9163aa2da251246abed9e756e53/src/runtime/Build.proj]
/builds/ayakael/aports/testing/dotnet7-stage0/src/bootstrap/sdk/7.0.100-rtm.22519.39/NuGet.RestoreEx.targets(19,5): error :    at NuGet.Commands.RestoreRunner.CompleteTaskAsync(List`1 restoreTasks) [/builds/ayakael/aports/testing/dotnet7-stage0/src/dotnet-d41bfecf5090e9163aa2da251246abed9e756e53/src/runtime/Build.proj]
/builds/ayakael/aports/testing/dotnet7-stage0/src/bootstrap/sdk/7.0.100-rtm.22519.39/NuGet.RestoreEx.targets(19,5): error :    at NuGet.Commands.RestoreRunner.RunAsync(IEnumerable`1 restoreRequests, RestoreArgs restoreArgs, CancellationToken token) [/builds/ayakael/aports/testing/dotnet7-stage0/src/dotnet-d41bfecf5090e9163aa2da251246abed9e756e53/src/runtime/Build.proj]
/builds/ayakael/aports/testing/dotnet7-stage0/src/bootstrap/sdk/7.0.100-rtm.22519.39/NuGet.RestoreEx.targets(19,5): error :    at NuGet.Commands.RestoreRunner.RunAsync(RestoreArgs restoreContext, CancellationToken token) [/builds/ayakael/aports/testing/dotnet7-stage0/src/dotnet-d41bfecf5090e9163aa2da251246abed9e756e53/src/runtime/Build.proj]
/builds/ayakael/aports/testing/dotnet7-stage0/src/bootstrap/sdk/7.0.100-rtm.22519.39/NuGet.RestoreEx.targets(19,5): error :    at NuGet.Build.Tasks.BuildTasksUtility.RestoreAsync(DependencyGraphSpec dependencyGraphSpec, Boolean interactive, Boolean recursive, Boolean noCache, Boolean ignoreFailedSources, Boolean disableParallel, Boolean force, Boolean forceEvaluate, Boolean hideWarningsAndErrors, Boolean restorePC, Boolean cleanupAssetsForUnsupportedProjects, ILogger log, CancellationToken cancellationToken) [/builds/ayakael/aports/testing/dotnet7-stage0/src/dotnet-d41bfecf5090e9163aa2da251246abed9e756e53/src/runtime/Build.proj]
/builds/ayakael/aports/testing/dotnet7-stage0/src/bootstrap/sdk/7.0.100-rtm.22519.39/NuGet.RestoreEx.targets(19,5): error :    at NuGet.Build.Tasks.Console.MSBuildStaticGraphRestore.RestoreAsync(String entryProjectFilePath, IDictionary`2 globalProperties, IReadOnlyDictionary`2 options) [/builds/ayakael/aports/testing/dotnet7-stage0/src/dotnet-d41bfecf5090e9163aa2da251246abed9e756e53/src/runtime/Build.proj]

Regression?

This error is confirmed not present with .NET runtime 6.0.10 in s390x, thus it seems to be a regression.

Known Workarounds

None so far.

Configuration

Version: SDK 7.0.100-rtm.22519.39
OS: Alpine LInux Edge docker-based CI Pipelines (although occurs to a lesser degree in s390x VM)
Architecure: Confirmed reproducible in s390x, ppc64le and x64 (with mono-flavored runtime)
Specificity: specific to mono on all platforms

Other information

logs
ppc64le with runtime: link
s390x with runtime: link
s390x with roslyn: link

Author:	ayakael
Assignees:	-
Labels:	area-Infrastructure, untriaged
Milestone:	-

[uweigand]

Not sure what causes this, I've never see this particular error in my builds ... Looking at the runtime sources, it seems this message is triggered only if some Linux native library call returns with errno set to EWOULDBLOCK - but I'm not sure why this would happen when just copying a file.

[ayakael]

Not sure what causes this, I've never see this particular error in my builds ... Looking at the runtime sources, it seems this message is triggered only if some Linux native library call returns with errno set to EWOULDBLOCK - but I'm not sure why this would happen when just copying a file.

Indeed, can't make heads or tails. One piece of data is that it going from tarball from dotnet/installer commit d41bfec to e6dd91c lowered occurence of this bug substantially. Although still, build always fails at fsharp with the above error on s390x and always fails during runtime restore with ppc64le.

Diff between those two commits is hard to figure out.

@omajid Have you started trying build of dotnet7 on s390x, do you encounter this error?

[ayakael]

The diff to git-info shows the following:

--- dotnet-d41bfecf5090e9163aa2da251246abed9e756e53/git-info/AllRepoVersions.props
+++ dotnet-e6dd91c290b808f971a1ac69c2fb29395bbf1051/git-info/AllRepoVersions.props
@@ -15,8 +15,8 @@
     <templatingOutputPackageVersion>7.0.100-rtm.22510.12</templatingOutputPackageVersion>
     <xdtGitCommitHash>9a1c3e1b7f0c8763d4c96e593961a61a72679a7b</xdtGitCommitHash>
     <xdtOutputPackageVersion>7.0.0-preview.22423.2</xdtOutputPackageVersion>
-    <arcadeGitCommitHash>720af493900b2f2bdc48e9ee12577983a5c9be36</arcadeGitCommitHash>
-    <arcadeOutputPackageVersion>7.0.0-beta.22464.4</arcadeOutputPackageVersion>
+    <arcadeGitCommitHash>02e28316bf35d1028683ee313f0794776bff18d1</arcadeGitCommitHash>
+    <arcadeOutputPackageVersion>7.0.0-beta.22513.4</arcadeOutputPackageVersion>
     <aspnetcoreGitCommitHash>c6865355c01e1fb170b65427846d939559ab3789</aspnetcoreGitCommitHash>
     <aspnetcoreOutputPackageVersion>7.0.0-rtm.22512.1</aspnetcoreOutputPackageVersion>
     <deploymenttoolsGitCommitHash>c3ad00ae84489071080a606f6a8e43c9a91a5cc2</deploymenttoolsGitCommitHash>
@@ -49,7 +49,7 @@
     <vstestOutputPackageVersion>17.4.0-release-20220926-01</vstestOutputPackageVersion>
     <xlifftasksGitCommitHash>740189d758fb3bbdc118c5b6171ef1a7351a8c44</xlifftasksGitCommitHash>
     <xlifftasksOutputPackageVersion>1.0.0-beta.22427.1</xlifftasksOutputPackageVersion>
-    <installerGitCommitHash>d41bfecf5090e9163aa2da251246abed9e756e53</installerGitCommitHash>
+    <installerGitCommitHash>e6dd91c290b808f971a1ac69c2fb29395bbf1051</installerGitCommitHash>
     <installerOutputPackageVersion>7.0.100</installerOutputPackageVersion>
   </PropertyGroup>
 </Project>

The following components have thus changed:

• arcade
• installer