0% coverage is being shown after running tests

[robvanpamel]

Hi,

After running my tests with Nunit I get an unexpected result from coverlet.
My folder structure is

root 
  \src 
     \projectfolder\project.csproj
  \tests
     \testprojectfolder\testproject.csproj
  \solution.sln

I executed the tests from within the testprojectfolder
The result is printed below.

dotnet test /p:CollectCoverage=true
Build started, please wait...
Build completed.

Test run for /home/rob/git/svc_system_center/tests/Sag.SystemCenter.Api.Tests/bin/Debug/netcoreapp2.0/Sag.SystemCenter.Api.Tests.dll(.NETCoreApp,Version=v2.0)
Microsoft (R) Test Execution Command Line Tool Version 15.5.0
Copyright (c) Microsoft Corporation.  All rights reserved.

Starting test execution, please wait...
NUnit Adapter 3.9.0.0: Test execution started
Running all tests in /home/rob/git/svc_system_center/tests/Sag.SystemCenter.Api.Tests/bin/Debug/netcoreapp2.0/Sag.SystemCenter.Api.Tests.dll
NUnit3TestExecutor converted 456 of 456 NUnit test cases

NUnit Adapter 3.9.0.0: Test execution complete

Total tests: 456. Passed: 452. Failed: 0. Skipped: 4.
Test Run Successful.
Test execution time: 3.8804 Seconds

Calculating coverage result...
  Generating report '/home/rob/git/svc_system_center/tests/Sag.SystemCenter.Api.Tests/coverage.json'

+--------------------------------------------+--------+--------+--------+
| Module                                     | Line   | Branch | Method |
+--------------------------------------------+--------+--------+--------+
| Sag.SystemCenter.LoginManagement.BL        | 0%     | 0%     | 0%     |
+--------------------------------------------+--------+--------+--------+
| Sag.SystemCenter.UserManagement.BL         | 0%     | 0%     | 0%     |
+--------------------------------------------+--------+--------+--------+
| Sag.SystemCenter.Identity                  | 0%     | 0%     | 0%     |
+--------------------------------------------+--------+--------+--------+
| Sag.SystemCenter.UserManagement.DAL        | 0%     | 0%     | 0%     |
+--------------------------------------------+--------+--------+--------+
| Sag.SystemCenter.ApiManagement             | 0%     | 0%     | 0%     |
+--------------------------------------------+--------+--------+--------+
| Sag.SystemCenter.Captcha.BL                | 0%     | 0%     | 0%     |
+--------------------------------------------+--------+--------+--------+
| Sag.SystemCenter.Domain                    | 0%     | 0%     | 0%     |
+--------------------------------------------+--------+--------+--------+
| Sag.SystemCenter.Api                       | 0%     | 0%     | 0%     |
+--------------------------------------------+--------+--------+--------+
| Sag.SystemCenter.Messaging                 | 0%     | 0%     | 0%     |
+--------------------------------------------+--------+--------+--------+
| Sag.SystemCenter.ApplicationManagement.DAL | 0%     | 0%     | 0%     |
+--------------------------------------------+--------+--------+--------+

I'm running on a Linux Mint machine with this dotnet configuration

.NET Command Line Tools (2.1.4)

Product Information:
 Version:            2.1.4
 Commit SHA-1 hash:  5e8add2190

Runtime Environment:
 OS Name:     linuxmint
 OS Version:  18.3
 OS Platform: Linux
 RID:         linux-x64
 Base Path:   /usr/share/dotnet/sdk/2.1.4/

Microsoft .NET Core Shared Framework Host

  Version  : 2.0.5
  Build    : 17373eb129b3b05aa18ece963f8795d65ef8ea54

[tonerdo]

Hi @robvanpamel can you try with the previous version of Coverlet (v1.2.0)? This seems to be a 2.0.0 issue. Let me know

[aburgett87]

Hi @tonerdo,
I can see the issue as well, though only on Linux. Using v1.2.0 coverage results output as expected, though on 2.0.0 it's as @robvanpamel described.

Using macOS 10.13.4, both 2.0.0 and 1.2.0 work as expected.

Linux dotnet cofiguration

.NET Command Line Tools (2.1.4)

Product Information:
 Version:            2.1.4
 Commit SHA-1 hash:  5e8add2190

Runtime Environment:
 OS Name:     debian
 OS Version:  9
 OS Platform: Linux
 RID:         linux-x64
 Base Path:   /usr/share/dotnet/sdk/2.1.4/

Microsoft .NET Core Shared Framework Host

  Version  : 2.0.5
  Build    : 17373eb129b3b05aa18ece963f8795d65ef8ea54

macOS dotnet configuration

.NET Command Line Tools (2.1.200)

Product Information:
 Version:            2.1.200
 Commit SHA-1 hash:  2edba8d7f1

Runtime Environment:
 OS Name:     Mac OS X
 OS Version:  10.13
 OS Platform: Darwin
 RID:         osx.10.13-x64
 Base Path:   /usr/local/share/dotnet/sdk/2.1.200/

Microsoft .NET Core Shared Framework Host

  Version  : 2.0.7
  Build    : 2d61d0b043915bc948ebf98836fefe9ba942be11

[aburgett87]

@tonerdo,
Interestingly, if I run my project with coverlet 2.0.0 on Linux using dotnet core 2.0.7 or 2.1-rc everything works as expected.

Linux dotnet 2.0.7 configuration

.NET Command Line Tools (2.1.200)

Product Information:
 Version:            2.1.200
 Commit SHA-1 hash:  2edba8d7f1

Runtime Environment:
 OS Name:     debian
 OS Version:  9
 OS Platform: Linux
 RID:         debian.9-x64
 Base Path:   /usr/share/dotnet/sdk/2.1.200/

Microsoft .NET Core Shared Framework Host

  Version  : 2.0.7
  Build    : 2d61d0b043915bc948ebf98836fefe9ba942be11

Linux dotnet configuration 2.1-rc1

.NET Core SDK (reflecting any global.json):
 Version:   2.1.300-rc1-008673
 Commit:    f5e3ddbe73

Runtime Environment:
 OS Name:     debian
 OS Version:  9
 OS Platform: Linux
 RID:         debian.9-x64
 Base Path:   /usr/share/dotnet/sdk/2.1.300-rc1-008673/

Host (useful for support):
  Version: 2.1.0-rc1
  Commit:  eb9bc92051

.NET Core SDKs installed:
  2.1.300-rc1-008673 [/usr/share/dotnet/sdk]

.NET Core runtimes installed:
  Microsoft.AspNetCore.All 2.1.0-rc1-final [/usr/share/dotnet/shared/Microsoft.AspNetCore.All]
  Microsoft.AspNetCore.App 2.1.0-rc1-final [/usr/share/dotnet/shared/Microsoft.AspNetCore.App]
  Microsoft.NETCore.App 2.1.0-rc1 [/usr/share/dotnet/shared/Microsoft.NETCore.App]

To install additional .NET Core runtimes or SDKs:
  https://aka.ms/dotnet-download

FYI @robvanpamel

[robvanpamel]

Hi,
For your information it is working with the 1.2 release

[basilfx]

@robvanpamel Can you try to exclude NUnit3 Test Adapter assemblies: dotnet test /p:CollectCoverage=true /p:Exclude=[NUnit3.TestAdapter]*. See #101 for more information.

[mikkokupsu]

I'm facing a similar problem.
I'm running below commands in .NET Docker container (2.0-sdk) and coverlet.msbuild version 2.0.0.
By removing /p:Threshold=90, I'm getting a coverage result. When the threshold is present, I get a 0 percentage report.

dotnet test /p:CollectCoverage=true /p:CoverletOutputFormat=lcov /p:ThresholdType=line /p:Threshold=90 /p:CoverletOutput=./lcov Test

Test run for /builds/Test/bin/Debug/netcoreapp2.0/Test.dll(.NETCoreApp,Version=v2.0)
Microsoft (R) Test Execution Command Line Tool Version 15.7.0
Copyright (c) Microsoft Corporation. All rights reserved.

Starting test execution, please wait...

Total tests: 98. Passed: 98. Failed: 0. Skipped: 0.
Test Run Successful.
Test execution time: 4.8294 Seconds

Calculating coverage result...
Generating report './lcov.info'

+-------------------+--------+--------+--------+
| Module | Line | Branch | Method |
+-------------------+--------+--------+--------+
| XXXX | 0% | 0% | 0% |
+-------------------+--------+--------+--------+

/root/.nuget/packages/coverlet.msbuild/2.0.0/build/netstandard2.0/coverlet.msbuild.targets(23,5): error : 'XXXX' has a line coverage '0%' below specified threshold '90%' [/builds/Test/Test.csproj]

dotnet test /p:CollectCoverage=true /p:CoverletOutputFormat=lcov /p:ThresholdType=line /p:CoverletOutput=./lcov Test

Test run for /builds/Test/bin/Debug/netcoreapp2.0/Test.dll(.NETCoreApp,Version=v2.0)
Microsoft (R) Test Execution Command Line Tool Version 15.7.0
Copyright (c) Microsoft Corporation. All rights reserved.

Starting test execution, please wait...

Total tests: 98. Passed: 98. Failed: 0. Skipped: 0.
Test Run Successful.
Test execution time: 4.7816 Seconds

Calculating coverage result...
Generating report './lcov.info'

+-------------------+--------+--------+--------+
| Module | Line | Branch | Method |
+-------------------+--------+--------+--------+
| XXXX | 100% | 85% | 100% |
+-------------------+--------+--------+--------+

[tlogik]

I have the issue of 0% coverage even on coverlet version 1.2.0
I am using
Xunit2,
Tested on Coverlet 1.2.0 and latest 2.0.0
on a netstand 2.0 projects.

Note: It works on other netstandard/netcore projects i have, running same framework versions etc.
Currently i do not see any pattern to the reason why some projects does not generate coverage where others do.

Update 2:
Message from dotnet test --no-build --filter Category=Unit /p:CollectCoverage=true
using XUNIT2 for tests.

No test is available in /Users/thomasblitz/Private/code/gitlab/Framework/framework-logging-applicationlogging/src/ApplicationLogging.DAL.Tests/bin/Debug/netcoreapp2.0/ApplicationLogging.DAL.Tests.dll. Make sure that test discoverer & executors are registered and platform & framework version settings are appropriate and try again. Additionally, path to test adapters can be specified using /TestAdapterPath command. Example /TestAdapterPath:<pathToCustomAdapters>.

Update 3:
Got it working

• I updated Microsoft.NET.Test.Sdk to 15.7.2 and maintained coverlet at 1.2.
  Still same result.
• I ran dotnet test --no-build --filter Category=Unit and because i had been writing some testcode i now could see some test failing. The failures were suppressed when i had /p:CollectCoverage=true
• Fixed the code, rebuild and run tests with success.
• ran dotnet test --no-build --filter Category=Unit /p:CollectCoverage=true and now i had coverage

Update 4:

Bumped coverlet to 2.0.0 and i still get coverage results :-D

There is a discrepancy in the results from coverlet 1.2 to 2.0.
Observe:
Coverlet 1.2

Module	Coverage
ApplicationLogging.Dal	0.3%

Coverlet 2.0

Module	Line	Branch	Method
ApplicationLogging.Dal	2.3%	2.7%	3.9%

[queen-of-code]

I'm seeing this as well in one of my test suites. I moved to using the nuget package, just to eliminate potential variables:

    <PackageReference Include="coverlet.msbuild" Version="2.2.1">
      <IncludeAssets>runtime; build; native; contentfiles; analyzers</IncludeAssets>
      <PrivateAssets>all</PrivateAssets>
    </PackageReference>
    <PackageReference Include="Microsoft.NET.Test.Sdk" Version="15.7.0" />
    <PackageReference Include="xunit.runner.visualstudio" Version="2.3.1" />
    <PackageReference Include="TeamCity.VSTest.TestAdapter" Version="1.0.10" />
    <PackageReference Include="xunit" Version="2.3.1" />
    <PackageReference Include="Moq" Version="4.8.3" />

Test run for C:\GitHub\api\test\unit\API.Partner.Service.Test\bin\Debug\netcoreapp2.1\API.Partner.Service.Test.dll(.NETCoreApp,Version=v2.1)
Microsoft (R) Test Execution Command Line Tool Version 15.8.0
Copyright (c) Microsoft Corporation.  All rights reserved.

Starting test execution, please wait...

Total tests: 53. Passed: 53. Failed: 0. Skipped: 0.
Test Run Successful.
Test execution time: 3.1755 Seconds

Calculating coverage result...
  Generating report 'C:\GitHub\api\test\unit\API.Partner.Service.Test\coverage.json'

+------------------------------------+--------+--------+--------+
| Module                             | Line   | Branch | Method |
+------------------------------------+--------+--------+--------+
| API                                 | 0%     | 0%     | 0%     |
+------------------------------------+--------+--------+--------+
| API.Partner.Service          | 0%     | 0%     | 0%     |
+------------------------------------+--------+--------+--------+

I know that coverage should actually be around 60-70% - certainly not 0. Interestingly, it generates an enormous coverage.json file full of lines, but all the Hits are :0.

"System.Collections.Generic.Dictionary`2<System.String,System.Func`2<System.Object,System.Object>> API.Partner.Service.AppCreatorBase`1::CreateCompiledGetters(System.Collections.Generic.IEnumerable`1<System.Reflection.PropertyInfo>)": {
          "Lines": {
            "114": 0,
            "115": 0,
            "117": 0,
            "118": 0,
            "119": 0,
            "121": 0,
            "123": 0,
            "125": 0,
            "126": 0,
            "127": 0,
            "129": 0,
            "130": 0
          },
          "Branches": [
            {
              "Line": 117,
              "Offset": 135,
              "EndOffset": 17,
              "Path": 1,
              "Ordinal": 1,
              "Hits": 0
            },
            {
              "Line": 117,
              "Offset": 135,
              "EndOffset": 137,
              "Path": 0,
              "Ordinal": 0,
              "Hits": 0
            }
          ]
        },

Is there a diagnostic logger for coverlet that I can enable?

[tonerdo]

Hi @queen-of-code @robvanpamel @tlogik take a look at #72. Try adding the following line to your csproj:

<CopyLocalLockFileAssemblies>true</CopyLocalLockFileAssemblies>

Is there a diagnostic logger for coverlet that I can enable?

Not at this time

[queen-of-code]

Unfortunately no luck. I added that, and also tried switching my test csproj from

    <PropertyGroup Condition="'$(Configuration)'=='Debug'">
        <DebugType>pdbonly</DebugType>
    </PropertyGroup>

    <PropertyGroup Condition="'$(Configuration)'=='Debug'">
        <DebugType>full</DebugType>
    </PropertyGroup>

Aside from making my test run exponentially slower, it didn't provide any coverage information. Just a dump full of 0% coverage-claims.

[tonerdo]

#181 @queen-of-code completely changes the way Coverlet instruments assemblies and makes the test run orders of magnitude faster. You can see perf improvements here: #181 (comment).

I expect this change should fix the 0% coverage issue you're all getting, should make a release in 48 hours so you can test it

[queen-of-code]

Looking forward to it @tonerdo - when you cut the new release, I will test it out ASAP.

[tonerdo]

@queen-of-code The new releases are out

Coverlet Global Tool: https://www.nuget.org/packages/coverlet.console/1.2.0
Coverlet MSBuild: https://www.nuget.org/packages/coverlet.msbuild/2.3.0

Let me know if your issue gets fixed when you upgrade.
Thanks!

[tonerdo]

@queen-of-code I didn't notice this before but you should try changing DebugType to portable