'The process cannot access the file' during CI build sometimes

[BrennanConroy]

This has been happening a few times recently. It's odd because BlazorTemplates.Tests doesn't depend on Microsoft.DotNet.Web.Spa.ProjectTemplates so it shouldn't be copying it anywhere.

There is some custom msbuild logic being invoked right before the error: https://github.com/dotnet/aspnetcore/blob/main/src/ProjectTemplates/TestInfrastructure/PrepareForTest.targets
So maybe that has something to do with it. We might just need to add /bl to this part of the build temporarily.

Example build:
https://dev.azure.com/dnceng/public/_build/results?buildId=1106179&view=logs&j=48c285a2-daf4-5ecb-cb99-42232769ca29&t=73646f8d-7520-5361-e1ad-d9d207b2cd21

./src/ProjectTemplates/build.sh --ci --nobl --pack --no-restore --no-build-deps
========================== Starting Command Output ===========================
/bin/bash --noprofile --norc /datadisks/disk1/workspace/_work/_temp/7404866d-8c11-4324-8bfe-b62b36813c80.sh
  RepoTasks -> /datadisks/disk1/workspace/_work/1/s/artifacts/bin/RepoTasks/Release/net6.0/RepoTasks.dll
  Successfully created package '/datadisks/disk1/workspace/_work/1/s/artifacts/packages/Release/Shipping/Microsoft.DotNet.Web.Client.ItemTemplates.6.0.0-ci.nupkg'.
  Successfully created package '/datadisks/disk1/workspace/_work/1/s/artifacts/packages/Release/Shipping/Microsoft.DotNet.Web.ItemTemplates.6.0.0-ci.nupkg'.
  Preparing environment for tests
  Removed directory /datadisks/disk1/workspace/_work/1/s/src/ProjectTemplates/BlazorTemplates.Tests/bin/Release/net6.0/TestTemplates/
  Removed directory /datadisks/disk1/workspace/_work/1/s/obj/template-restore/
  Created directory /datadisks/disk1/workspace/_work/1/s/src/ProjectTemplates/BlazorTemplates.Tests/bin/Release/net6.0/TestTemplates/
  Created directory /datadisks/disk1/workspace/_work/1/s/obj/template-restore/
  Successfully created package '/datadisks/disk1/workspace/_work/1/s/artifacts/packages/Release/Shipping/Microsoft.DotNet.Web.ProjectTemplates.6.0.6.0.0-ci.nupkg'.
/datadisks/disk1/workspace/_work/1/s/.dotnet/sdk/6.0.100-preview.4.21222.6/Microsoft.Common.CurrentVersion.targets(4965,5): error MSB3026: Could not copy "/datadisks/disk1/workspace/_work/1/s/artifacts/packages/Release/Shipping/Microsoft.DotNet.Web.Spa.ProjectTemplates.6.0.6.0.0-ci.nupkg" to "bin/Release/net6.0/Microsoft.DotNet.Web.Spa.ProjectTemplates.6.0.6.0.0-ci.nupkg". Beginning retry 1 in 1000ms. The process cannot access the file '/datadisks/disk1/workspace/_work/1/s/artifacts/packages/Release/Shipping/Microsoft.DotNet.Web.Spa.ProjectTemplates.6.0.6.0.0-ci.nupkg' because it is being used by another process.  [/datadisks/disk1/workspace/_work/1/s/src/ProjectTemplates/BlazorTemplates.Tests/BlazorTemplates.Tests.csproj]
##[error].dotnet/sdk/6.0.100-preview.4.21222.6/Microsoft.Common.CurrentVersion.targets(4965,5): error MSB3026: (NETCORE_ENGINEERING_TELEMETRY=Build) Could not copy "/datadisks/disk1/workspace/_work/1/s/artifacts/packages/Release/Shipping/Microsoft.DotNet.Web.Spa.ProjectTemplates.6.0.6.0.0-ci.nupkg" to "bin/Release/net6.0/Microsoft.DotNet.Web.Spa.ProjectTemplates.6.0.6.0.0-ci.nupkg". Beginning retry 1 in 1000ms. The process cannot access the file '/datadisks/disk1/workspace/_work/1/s/artifacts/packages/Release/Shipping/Microsoft.DotNet.Web.Spa.ProjectTemplates.6.0.6.0.0-ci.nupkg' because it is being used by another process. 
  Successfully created package '/datadisks/disk1/workspace/_work/1/s/artifacts/packages/Release/Shipping/Microsoft.DotNet.Web.Spa.ProjectTemplates.6.0.6.0.0-ci.nupkg'.
  Preparing environment for tests
  Removed directory /datadisks/disk1/workspace/_work/1/s/src/ProjectTemplates/test/bin/Release/net6.0/TestTemplates/
  Removed directory /datadisks/disk1/workspace/_work/1/s/obj/template-restore/
  Created directory /datadisks/disk1/workspace/_work/1/s/src/ProjectTemplates/test/bin/Release/net6.0/TestTemplates/
  Created directory /datadisks/disk1/workspace/_work/1/s/obj/template-restore/
  BlazorTemplates.Tests -> /datadisks/disk1/workspace/_work/1/s/src/ProjectTemplates/BlazorTemplates.Tests/bin/Release/net6.0/BlazorTemplates.Tests.dll
  ProjectTemplates.Tests -> /datadisks/disk1/workspace/_work/1/s/src/ProjectTemplates/test/bin/Release/net6.0/ProjectTemplates.Tests.dll

[dougbu]

@JunTaoLuo this may be the start of the general issue around file contention in our builds that we discussed yesterday.

[JunTaoLuo]

We're hitting this somewhat frequently. Today there were two occurences: https://dev.azure.com/dnceng/public/_build/results?buildId=1172134&view=results, https://dev.azure.com/dnceng/public/_build/results?buildId=1172278&view=results

[TanayParikh]

Bin Logs:

CI: Pipelines - https://dev.azure.com/dnceng/public/_build/results?buildId=1176795&view=artifacts&pathAsName=false&type=publishedArtifacts

Commit Used: 0d75fcb

Original Failure: https://dev.azure.com/dnceng/public/_build/results?buildId=1172134&view=results

[dougbu]

Thought someone considered putting up a PR to implement @rainersigwald's idea (thanks @rainersigwald). Was that #33565 @pranavkm❔

Given that's done, do we have additional monitoring to do to ensure we addressed the main problem or to find additional build contention problems❔ Basically wondering if we can close this or need an over-arching "our build contends w/ itself (ick ❕) issue❔

If we don't need to track other build ordering / contention issues (@Pilchie @dotnet/aspdoi your thoughts ❔), suggest closing this near the end of your ops rotation @BrennanConroy.

[pranavkm]

Was that #33565 @pranavkm❔

Yeah, that was based on @rainersigwald's analysis. He had suggested making the outputs different, after playing around with it enough I realized we didn't need two sets of them.

[BrennanConroy]

Today:

##[error].dotnet/sdk/6.0.100-preview.6.21313.2/Microsoft.Common.CurrentVersion.targets(2202,5):
error MSB3248: (NETCORE_ENGINEERING_TELEMETRY=Build) Parameter "AssemblyFiles" has invalid value 
"/Users/runner/work/1/s/artifacts/bin/Microsoft.AspNetCore.Hosting/Release/net6.0/Microsoft.AspNetCore.Hosting.dll".
The process cannot access the file '/Users/runner/work/1/s/artifacts/bin/Microsoft.AspNetCore.Hosting/Release/net6.0/Microsoft.AspNetCore.Hosting.dll'
because it is being used by another process.

https://dev.azure.com/dnceng/public/_build/results?buildId=1198450&view=logs&jobId=f43b187f-6c15-531c-a59b-e4fba4e9ed20&j=f43b187f-6c15-531c-a59b-e4fba4e9ed20&t=98a09275-a4dd-593e-c21a-1f8bea3be5b1

[Eilon]

Failing on multiple runs in my PR: https://dev.azure.com/dnceng/public/_build/results?buildId=1199902&view=logs&jobId=f43b187f-6c15-531c-a59b-e4fba4e9ed20&j=f43b187f-6c15-531c-a59b-e4fba4e9ed20&t=98a09275-a4dd-593e-c21a-1f8bea3be5b1