Update to Microsoft.Data.SqlClient 2.0.0 breaks a self-contained .NET Core 3.1 ServiceFabric service

[dmytro-gokun]

We have a ServiceFabric service, which is running the latest .NET Core 3.1. We publish & deploy it as self-contained. It references Microsoft.Data.SqlClient. Using version 1.1.3 everything works perfectly. After updating to 2.0.0, we can not event deploy it to the fabric anymore. Unfortunataly, ServiceFabric gives very little information why. This is what we see:

Service package for manifest 'XXXX' and service package activation ID 'XXXX' is in Error.
		'System.Hosting' reported Error for property 'ServiceTypeRegistration:XXX'.
		The ServiceType was disabled on the node.

It tries that several times and then gives up.
By digging into SF events i was able to find this as well:

There was an error during CodePackage activation.The process/container terminated with exit code:2147516556.

By comparing service package i can see that it used to include sni.dll when using 1.1.3 and then its gone after we updated to 2.0.0. My suspicion is that is the reason why the service fails to start.

And i think is my theory is correct because the service works just fine locally, since sni.dll is availaible on my PC. But it does not exist on the ServiceFabric's VMs.

[cheenamalhotra]

@dmytro-gokun

Please look at 2.0.0 release notes for breaking changes in SNI naming (introduced in #570). For .NET Core, with 2.0.0 we've introduced our own replacement for runtime.native.System.Data.SqlClient.sni NuGet as well that renames sni.dll to Microsoft.Data.SqlClient.SNI.dll.

If you can provide us a repro, we can investigate more in what your observations are.

[dmytro-gokun]

@cheenamalhotra Yes, I've read that. And the deployment package contains Microsoft.Data.SqlClient.SNI.dll, so no problem here. The problem is that it somehow still needs sni.dll to load.

About a repro. As I said, it does not reproduce with a Service Fabric Local Cluster on a typical development machine. I can make a repro that you will have to deploy to a real cluster and then you will see it works just fine with 1.1.3 and does not even start with 2.0.0. Will this work for you?

[cheenamalhotra]

The problem is that it somehow still needs sni.dll to load.

The SqlClient driver does not need the old sni.dll anymore. Can you confirm which component needs the old 'sni.dll' by capturing more logs from your cluster? The above errors don't provide any context.

I can make a repro that you will have to deploy to a real cluster and then you will see it works just fine with 1.1.3 and does not even start with 2.0.0. Will this work for you?

I don't have that environment to deploy but if you can gather logs we can investigate the errors.

[dmytro-gokun]

The SqlClient driver does not need the old sni.dll anymore.

Do you perhaps know, in which situations it was loaded before? That will help me build a repro.

Can you confirm which component needs the old 'sni.dll' by capturing more logs from your cluster? The above errors don't provide any context.

Well, that's the problem with ServiceFabric. It does not give you any more info than that. I've spent several hours googling and trying different tricks, but that's all I've got :( Any chance, you know anyone who can help with getting more info from SF?

[cheenamalhotra]

Do you perhaps know, in which situations it was loaded before? That will help me build a repro.

It's a NuGet package reference, and was loaded from runtime.native.System.Data.SqlClient.sni same way as currently Microsoft.Data.SqlClient.SNI.runtime NuGet is loaded. It's a package reference that contains native runtime DLLs. The old package referenced another package implicitly based on arch, but now we added arch specific runtime folders in the same NuGet. But again. the driver doesn't need old sni.dll anymore, so I don't think it will stop you from working if it's absent from your environment.

Have you tried to remove sni.dll from your local dev environment and confirm if you see same error?

If it's related, it should fail locally too, but if not - this error is probably not related to driver and it's dependencies. I also see similar issues on service fabric (same error code): microsoft/service-fabric#868

You can find guide to debug issues here: Debug service errors using the log files, or else contact Service Fabric teams.

[dmytro-gokun]

So, after further investigation, I was able to find the root cause. It has nothing to do with sni.dll.
The root cause is this exception:

Error:
  An assembly specified in the application dependencies manifest (XXXX.deps.json) was not found:
    package: 'Microsoft.Data.SqlClient.SNI.runtime', version: '2.0.0'
    path: 'runtimes/win-x64/native/Microsoft.Data.SqlClient.SNI.pdb'

This exception happens during the deployment and before our code even has a chance to run.

Now:

1. Why a .pdb is even required for the app to load? Are not pdbs only needed for debugging/stack trace formatting etc? To me this looks really weird, maybe I do not understand something about pdbs here;

2. The .pdb is not here because that's how Azure DevOps deploys services by default: without pdbs. This is described in detail here: https://docs.microsoft.com/en-us/azure/service-fabric/service-fabric-tutorial-deploy-app-with-cicd-vsts#configure-continuous-delivery-with-azure-pipelines. Particularly, on this screenshot: https://docs.microsoft.com/en-us/azure/service-fabric/media/service-fabric-tutorial-deploy-app-with-cicd-vsts/saveandqueue.png you can (partially) see the step which copies *.pdb to a separate folder (which is later zipped to an artifact) and then the next step which deletes original *.pdb from the final deployment package. This explains why pdbs are not copied to Service Fabric and why we get the exception we get.

3. After we removed steps which remove pdbs from the deployment package, it deploys and works just fine.

To me it looks like the correct fix to this issue should be not to require the pdb presence after all. If, for some reason, it is absolutely required, this should be thoroughfuly documented somewhere. Noone should kill so much time on this again.

[ErikEJ]

@dmytro-gokun This looks very Service Fabric specific - wonder why the Service Fabric team does not want to deploy .pdb files? Maybe the Azure DevOps template should be updated?

[dmytro-gokun]

@ErikEJ well, I do not know why. I can only guess that they want to preserve disk space. If you are using microservices architecture and run hundreds of services, all those pdbs (which are identical for each service) may eat all your limited disk space very fast.

Nevertheless, deploying a .pdb alongside your app or not deploying it should be the developer's choice and not a requirement for the app to run at all.

[ErikEJ]

Agree, wonder if the pdb was included in the previous package? @cheenamalhotra ??

[dmytro-gokun]

@ErikEJ It obviously was not, as the same app/DevOps pipeline worked just fine with 1.1.3

[cheenamalhotra]

Hi @dmytro-gokun

The pdbs were included starting with v2.0.0 since the DLLs are closed source, as tools like Binskim fail to work with driver otherwise as reported in #598

I would expect these errors to be handled by applications or application frameworks and there should be a solution to this.

But products cannot NOT ship PDBs due to such issues, it looks like a generic issue to me that should be reported to Service Fabrics and documented by their teams how to handle PDB related issues their tools may encounter.

[ErikEJ]

This issue is called by this step in the Azure DevOps build pipeline:

steps:
- task: DeleteFiles@1
  displayName: 'Delete files from $(build.artifactstagingdirectory)\applicationpackage'
  inputs:
    SourceFolder: '$(build.artifactstagingdirectory)\applicationpackage'
    Contents: '**\*.pdb'

[dmytro-gokun]

@cheenamalhotra In my humble opinion, Azure DevOps does the right thing: they treat debug info as debug info. By requiring a pdb to be present for an app to start, you are essentially changing the semantic of what pdb is. IMO.

[cheenamalhotra]

By requiring a pdb to be present for an app to start, you are essentially changing the semantic of what pdb is. IMO.

@dmytro-gokun I'm not sure I follow.. We don't require PDBs to be present, there's no dependency! They're copied with DLL when you build/deploy. You should be able to work without them.

What seems to be going on is these PDB references are copied to your deps.json file but when you remove these PDBs the deps.json fails to find them when dotnet reads the file. So you need to probably remove them before deps.json is created or fix your deps.json essentially if you absolutely need to remove them.