Regression: `dotnet test` hangs for 15 minutes after test run completed on mac and linux

[AArnott]

Our builds on VSTS include dotnet test, which restores, builds, and tests our product. With dotnet CLI 2.1.100 all went well. But when we upgraded to 2.1.300, the dotnet tool doesn't exit for 15 minutes after the test run is completed with no output to explain the delay.

Expected behavior

The dotnet tool exits immediately, as can be seen on this 2.1.100 run:

Actual behavior

The dotnet tool exits after waiting 15 minutes, as can be seen on this 2.1.300 run:

Steps to reproduce

Please contact me via my microsoft.com email address for links to the builds if you'd like to investigate further.

[livarcocc]

I am not sure what you are doing in VSTS, but I suspect this might be long running servers that are still live when the test finishes.

In 2.1.300, we now have a long running VBCSCompiler and a MSBuild node re-use server and a Razor compilation server.

They all stay around for some time to avoid JITing and improve perf if you attempt a build again.

You can turn them off by running dotnet build-server shutdown after the test execution. Or, you can set msbuild properties/env variables to prevent them from running to begin with.

[AArnott]

I suspect this might be long running servers that are still live when the test finishes.

Perhaps. But why would that prevent the tool from exiting?

You can turn them off by running dotnet build-server shutdown after the test execution

How would I run that on VSTS after test execution, since test execution itself doesn't return control to the next step in my build till the 15 minutes are over?

Or, you can set msbuild properties/env variables to prevent them from running to begin with.

I'm willing to try that. Where do I find the variables to set? And if it works, is that a bug in dotnet CLI then that it's waiting for those servers when the work is finished anyway?

[livarcocc]

@peterhuene can you help @AArnott in getting the properties to turn off the long running servers?

@AArnott I don't understand enough about how VSTS does test execution to know why it would consider as if test was not done in that case.

[peterhuene]

Use -p:UseRazorBuildServer=false to disable the Razor (rzc) server.

Use -p:UseSharedCompilation=false to disable the Roslyn (vbcscompiler) server.

For MSBuild, pass /nodeReuse:false on the command line to disable node re-use.

I don't know of anything that might be waiting on the servers. There was an issue where MSBuild might hang with node re-use enabled on non-Windows platforms (dotnet/msbuild#3161), but that was fixed for RTM.

Would it be possible to capture the stacks of the hang?

[jskeet]

FWIW, I'm seeing hangs on Linux for dotnet build unless I disable node reuse too. This happens on a machine where I'm running BenchmarkDotNet benchmarks, which does its own builds as well - so it's possible that something in that is jamming a build node. (My next step will be to find a way of disabling build reuse globally...)

[AArnott]

I can confirm that setting MSBUILDDISABLENODEREUSE=1 on mac and linux removes the hang at the end of the build+test invocation as well.

[AArnott]

FWIW, I vaguely recall a bug in a Windows process my team owned where a parent process spawned my child process, and so long as my child process was running the parent process never entirely exited. We fixed it (I don't remember how but could potentially find out). I wonder if something similar is happening here, where dotnet test (which builds first) spawned the reusable node processes and children (instead of as top-level processes perhaps?) and now the parent process cannot exit till the reusable nodes timeout and exit. In that case, perhaps the fix is the change how the child processes are spawned so that they do not retain a child relationship with the parent.

[peterhuene]

@rainersigwald Some reports of hitting hangs with node reuse above.

[rainersigwald]

This sounds exactly like what dotnet/msbuild@e604f5f addressed. But evidently that was incomplete or doesn't work everywhere.

@jskeet @AArnott what's the best way for me to repro your cases? Can I just clone nodatime (before nodatime/nodatime#1157) and run build/runbenchmarks.sh?

[jskeet]

@rainersigwald: I believe that should repro it, yes. You'll need to specify the targets to run - netcoreapp2.0 netcoreapp2.1 is what I do normally. It's slightly harder to tell as I've normally run it as part of a cron job. I'd be happy to try it beforehand if that would help. Note that it's expected that it takes multiple hours, but there should be a lot of output a lot of the time.

When it's broken, you should be able to run dotnet build anywhere (e.g. in src/NodaTime) and it'll hang.

[rainersigwald]

@jskeet Oh, that's interesting. It repros with dotnet build and not just dotnet test?

I think I understand the dotnet test problem: I think we're accidentally running the before-test build within the context of the

https://github.com/dotnet/cli/blob/7ce47778b4a5238db133cd0d4ffc3aa2695e406c/src/dotnet/commands/dotnet-test/Program.cs#L99

environment, leading the build nodes to be created in a hangy way.

But that wouldn't apply to build so I'll have to try debugging through the nodatime repro.

[jskeet]

@rainersigwald: I don't run dotnet test at all (in this workflow, or on that machine) - it repros just using dotnet run with BenchmarkDotNet, but that does some more builds in the background. I don't know all the details of what it does, but I doubt that it calls dotnet test.

But once it's in the "broken" state, then yes, dotnet build hangs - as does dotnet restore. (Specifying -nodeReuse:false on the command line makes both work.)

I'm happy to try to get it into the broken state again and then poke around - are there diagnostics that would be useful to try at that point?

[reduckted]

I just want to chime in and say that I'm experiencing this issue as well. I haven't been able to reproduce it reliably yet, but I'll keep trying and once I do, I'll post a link to a repo here.

The scenario I have is a unit test that is testing some MSBuild targets files. This involves calling dotnet build during the unit test. This is basically what it does:

1. Write a project file that uses the targets to disk.
2. Compile the project using dotnet build.
3. Look at the output from the build and assert some stuff.

When running this via dotnet test, it sometimes hangs for 15 minutes after the tests complete (the test duration reported is only a few seconds), but sometimes it doesn't hang at all.