The MARS TDS header contained errors using ASP.NET Core and EF Core connecting to Azure SQL Server

[AfsanehR-zz]

Copied from here:

I have an ASP.NET Core app [Microsoft.AspNetCore.App 2.1.4] with EF Core 2.1.4 [DbContext pooling enabled] and data stored on an Azure SQL database.

Occasionally [once in 1-2 days] I get unrecoverable error stating System.Data.SqlClient.SqlException (0x80131904): The incoming tabular data stream (TDS) protocol stream is incorrect. The MARS TDS header contained errors.

Once I restart the app, everything resumes working until it happens again.

Probably import to note is that I don't have MultipleActiveResultSets enabled in my connection string at all, which makes the issue even more strange.

Has anyone else noticed something similar? Are there any ways I can trace the problem?

Stack trace:

System.Data.SqlClient.SqlException (0x80131904): The incoming tabular data stream (TDS) protocol stream is incorrect. The MARS TDS header contained errors.
   at System.Data.SqlClient.SqlConnection.OnError(SqlException exception, Boolean breakConnection, Action`1 wrapCloseInAction)
   at System.Data.SqlClient.SqlInternalConnection.OnError(SqlException exception, Boolean breakConnection, Action`1 wrapCloseInAction)
   at System.Data.SqlClient.TdsParser.ThrowExceptionAndWarning(TdsParserStateObject stateObj, Boolean callerHasConnectionLock, Boolean asyncClose)
   at System.Data.SqlClient.TdsParser.TryRun(RunBehavior runBehavior, SqlCommand cmdHandler, SqlDataReader dataStream, BulkCopySimpleResultSet bulkCopyHandler, TdsParserStateObject stateObj, Boolean& dataReady)
   at System.Data.SqlClient.SqlDataReader.TryConsumeMetaData()
   at System.Data.SqlClient.SqlDataReader.get_MetaData()
   at System.Data.SqlClient.SqlCommand.FinishExecuteReader(SqlDataReader ds, RunBehavior runBehavior, String resetOptionsString)
   at System.Data.SqlClient.SqlCommand.RunExecuteReaderTds(CommandBehavior cmdBehavior, RunBehavior runBehavior, Boolean returnStream, Boolean async, Int32 timeout, Task& task, Boolean asyncWrite, SqlDataReader ds)
   at System.Data.SqlClient.SqlCommand.RunExecuteReader(CommandBehavior cmdBehavior, RunBehavior runBehavior, Boolean returnStream, TaskCompletionSource`1 completion, Int32 timeout, Task& task, Boolean asyncWrite, String method)
   at System.Data.SqlClient.SqlCommand.ExecuteReader(CommandBehavior behavior)
   at System.Data.SqlClient.SqlCommand.ExecuteDbDataReader(CommandBehavior behavior)
   at Microsoft.EntityFrameworkCore.Storage.Internal.RelationalCommand.Execute(IRelationalConnection connection, DbCommandMethod executeMethod, IReadOnlyDictionary`2 parameterValues)
   at Microsoft.EntityFrameworkCore.Storage.Internal.RelationalCommand.ExecuteReader(IRelationalConnection connection, IReadOnlyDictionary`2 parameterValues)
   at Microsoft.EntityFrameworkCore.Query.Internal.QueryingEnumerable`1.Enumerator.BufferlessMoveNext(DbContext _, Boolean buffer)
   at Microsoft.EntityFrameworkCore.Storage.ExecutionStrategy.ExecuteImplementation[TState,TResult](Func`3 operation, Func`3 verifySucceeded, TState state)
   at Microsoft.EntityFrameworkCore.Storage.ExecutionStrategy.Execute[TState,TResult](TState state, Func`3 operation, Func`3 verifySucceeded)
   at Microsoft.EntityFrameworkCore.Query.Internal.QueryingEnumerable`1.Enumerator.MoveNext()
   at Microsoft.EntityFrameworkCore.Query.Internal.LinqOperatorProvider.ExceptionInterceptor`1.EnumeratorExceptionInterceptor.MoveNext()
   at System.Collections.Generic.List`1.AddEnumerable(IEnumerable`1 enumerable)
   at System.Collections.Generic.List`1..ctor(IEnumerable`1 collection)
   at System.Linq.Enumerable.ToList[TSource](IEnumerable`1 source)
   at Inviton.Web.Portal.Controllers.EventListController.GetEvents(BL bl, Int32 venueId, Int32 promoterId) 

[brunolau]

Regarding the stackoverflow issue addiotional questions asked by Afsaneh:

Unfortunately can't post the csproj file contents...

• I'm using EF Core 2.1.4 with Linq2Db provider (https://github.com/linq2db/linq2db.EntityFrameworkCore) for some of the queries that are poorly translated by the EF Core.., I'm however also referencing the System.Data.SqlClient 4.5.1 package in on of the solution's projects

• I'm hosting on Azure Websites, so I assume it's some sort of IIS

• I'll have to log this more explicitly. Is logging the "State" field sufficient for you ?

[brunolau]

Here's the Dump of the SqlError object:

Class:
16

Errors:
System.Data.SqlClient.SqlError: The incoming tabular data stream (TDS) protocol stream is incorrect. The MARS TDS header contained errors.

Error code:
-2146232060

Line number:
1

Number:
4011

SQL Procedure:
None

Source:
Core .Net SqlClient Data Provider

State:
3

[stap123]

I had a random spate of these errors appear in my ASP.NET application today. I use ASP.NET Core 2.1, EF Core 2.1 and Azure SQL Database. They occurred in a small time window and once I stopped and started the web app the errors seemed to stop.

I think I queried successfully from SSMS while the errors were happening but couldn't say for sure that it was at the same time. If it happens again I'll query in SSMS to confirm that I can.

Is there anything I can provide that will help with a solution to this? (I have application insights and database logging turned on for my application)

NOTES:
MARS is explicitly set to off in my connection string (I use the one Azure Provides via the portal UI)

[AfsanehR-zz]

@stap123 @brunolau Would you be able to provide us a minimal repro sample application? Thanks.

[stap123]

@afsanehr I wouldn't even know where to start, I've so far only seen it once today and restarting my application made it stop happening. I will keep my eyes open and if it starts happening in a pattern of some kind I'll try and put one together but at the moment it seems random so I've got no starting point for a repro.

I realise that's useless to you as well though haha

I have got about 130x exceptions logged in my App Insights at the moment but I doubt the stack trace would be enough for you?

It also looks like the error occurred on ALL queries no matter where they originated from so it doesn't appear to be a query specific issue.

[brunolau]

I confirm the same - errors occurring at random queries, random time. I've tried to stress-test the application by running 30 threads that was making DB queries in parallel, but that didn't reproduce the issue either.

I've tried to use the SSMS query while the error was occurring and it worked, I've also tried to deploy to another azure release slot at the time the issue was occurring and it worked.

What I've temporarily ended up with was writing a middleware that once detects this particular error [based on the err message], automatically restarts the web app. As the issue occurs 1-2 times a day, it's acceptable, but much better would be having the issue solved.

What connects both mine and Adam's issue is the ASP.NET Core 2.1 + EF Core 2.1 + Azure SQL DB combo, but I cannot even say what part of the chain to blame.

[AfsanehR-zz]

ok thanks @brunolau @stap123 . We will investigate this with the configurations you mentioned, although I think it would be highly unlikely we hit the same issue as we need to know what queries are running. Will get back to you on this. Could you also please give us the output of dotnet --info?

[brunolau]

Here you go, obtained from the Azure console

.NET Core SDK (reflecting any global.json):
Version: 2.2.100
Commit: 51868761f2

Runtime Environment:
OS Name: Windows
OS Version: 10.0.14393
OS Platform: Windows
RID: win10-x86
Base Path: D:\Program Files (x86)\dotnet\sdk\2.2.100\

Host (useful for support):
Version: 2.2.0
Commit: 1249f08fed

.NET Core SDKs installed:
1.1.10 [D:\Program Files (x86)\dotnet\sdk]
2.1.403 [D:\Program Files (x86)\dotnet\sdk]
2.1.500 [D:\Program Files (x86)\dotnet\sdk]
2.2.100 [D:\Program Files (x86)\dotnet\sdk]

.NET Core runtimes installed:
Microsoft.AspNetCore.All 2.1.3 [D:\Program Files (x86)\dotnet\shared\Microsoft.AspNetCore.All]
Microsoft.AspNetCore.All 2.1.5 [D:\Program Files (x86)\dotnet\shared\Microsoft.AspNetCore.All]
Microsoft.AspNetCore.All 2.1.6 [D:\Program Files (x86)\dotnet\shared\Microsoft.AspNetCore.All]
Microsoft.AspNetCore.All 2.2.0 [D:\Program Files (x86)\dotnet\shared\Microsoft.AspNetCore.All]
Microsoft.AspNetCore.App 2.1.3 [D:\Program Files (x86)\dotnet\shared\Microsoft.AspNetCore.App]
Microsoft.AspNetCore.App 2.1.5 [D:\Program Files (x86)\dotnet\shared\Microsoft.AspNetCore.App]
Microsoft.AspNetCore.App 2.1.6 [D:\Program Files (x86)\dotnet\shared\Microsoft.AspNetCore.App]
Microsoft.AspNetCore.App 2.2.0 [D:\Program Files (x86)\dotnet\shared\Microsoft.AspNetCore.App]
Microsoft.NETCore.App 1.0.12 [D:\Program Files (x86)\dotnet\shared\Microsoft.NETCore.App]
Microsoft.NETCore.App 1.1.9 [D:\Program Files (x86)\dotnet\shared\Microsoft.NETCore.App]
Microsoft.NETCore.App 2.0.9 [D:\Program Files (x86)\dotnet\shared\Microsoft.NETCore.App]
Microsoft.NETCore.App 2.1.3 [D:\Program Files (x86)\dotnet\shared\Microsoft.NETCore.App]
Microsoft.NETCore.App 2.1.5 [D:\Program Files (x86)\dotnet\shared\Microsoft.NETCore.App]
Microsoft.NETCore.App 2.1.6 [D:\Program Files (x86)\dotnet\shared\Microsoft.NETCore.App]
Microsoft.NETCore.App 2.2.0 [D:\Program Files (x86)\dotnet\shared\Microsoft.NETCore.App]

To install additional .NET Core runtimes or SDKs:
https://aka.ms/dotnet-download

[stap123]

Here's mine 👍

.NET Core SDK (reflecting any global.json):
Version: 2.2.100
Commit: 51868761f2

Runtime Environment:
OS Name: Windows
OS Version: 10.0.14393
OS Platform: Windows
RID: win10-x86
Base Path: D:\Program Files (x86)\dotnet\sdk\2.2.100\

Host (useful for support):
Version: 2.2.0
Commit: 1249f08fed

.NET Core SDKs installed:
1.1.10 [D:\Program Files (x86)\dotnet\sdk]
2.1.403 [D:\Program Files (x86)\dotnet\sdk]
2.1.500 [D:\Program Files (x86)\dotnet\sdk]
2.2.100 [D:\Program Files (x86)\dotnet\sdk]

.NET Core runtimes installed:
Microsoft.AspNetCore.All 2.1.3 [D:\Program Files (x86)\dotnet\shared\Microsoft.AspNetCore.All]
Microsoft.AspNetCore.All 2.1.5 [D:\Program Files (x86)\dotnet\shared\Microsoft.AspNetCore.All]
Microsoft.AspNetCore.All 2.1.6 [D:\Program Files (x86)\dotnet\shared\Microsoft.AspNetCore.All]
Microsoft.AspNetCore.All 2.2.0 [D:\Program Files (x86)\dotnet\shared\Microsoft.AspNetCore.All]
Microsoft.AspNetCore.App 2.1.3 [D:\Program Files (x86)\dotnet\shared\Microsoft.AspNetCore.App]
Microsoft.AspNetCore.App 2.1.5 [D:\Program Files (x86)\dotnet\shared\Microsoft.AspNetCore.App]
Microsoft.AspNetCore.App 2.1.6 [D:\Program Files (x86)\dotnet\shared\Microsoft.AspNetCore.App]
Microsoft.AspNetCore.App 2.2.0 [D:\Program Files (x86)\dotnet\shared\Microsoft.AspNetCore.App]
Microsoft.NETCore.App 1.0.12 [D:\Program Files (x86)\dotnet\shared\Microsoft.NETCore.App]
Microsoft.NETCore.App 1.1.9 [D:\Program Files (x86)\dotnet\shared\Microsoft.NETCore.App]
Microsoft.NETCore.App 2.0.9 [D:\Program Files (x86)\dotnet\shared\Microsoft.NETCore.App]
Microsoft.NETCore.App 2.1.3 [D:\Program Files (x86)\dotnet\shared\Microsoft.NETCore.App]
Microsoft.NETCore.App 2.1.5 [D:\Program Files (x86)\dotnet\shared\Microsoft.NETCore.App]
Microsoft.NETCore.App 2.1.6 [D:\Program Files (x86)\dotnet\shared\Microsoft.NETCore.App]
Microsoft.NETCore.App 2.2.0 [D:\Program Files (x86)\dotnet\shared\Microsoft.NETCore.App]

To install additional .NET Core runtimes or SDKs:
https://aka.ms/dotnet-download

[hackfaq]

Hello, we have experiensed same problem:

System.Data.SqlClient.SqlException (0x80131904): The incoming tabular data stream (TDS) protocol stream is incorrect. The MARS TDS header contained errors.
at System.Data.SqlClient.SqlCommand.<>c.b__122_0(Task1 result) at System.Threading.Tasks.ContinuationResultTaskFromResultTask2.InnerInvoke()
at System.Threading.ExecutionContext.RunInternal(ExecutionContext executionContext, ContextCallback callback, Object state)
--- End of stack trace from previous location where exception was thrown ---
at System.Threading.Tasks.Task.ExecuteWithThreadLocal(Task& currentTaskSlot)
--- End of stack trace from previous location where exception was thrown ---
at Dapper.SqlMapper.QueryAsync[T](IDbConnection cnn, Type effectiveType, CommandDefinition command) in C:\projects\dapper\Dapper\SqlMapper.Async.cs:line 419

We use

• Dapper 1.50.5
• .NETCOREAPP 2.1.2
• Runtime in docker: FROM microsoft/dotnet:2.1-aspnetcore-runtime AS final

How to reproduce:
When we experienced problem: our database log file was full, and we got exception: the transaction log for database '' is full due to "LOG_BACKUP", after that exception all queries in .net core app was with errors.

When we increased log file, .net 4.7 apps starts to work, but .net core app continue to throw exception. Only restart of application helped

[stap123]

I got this again this morning between 07:38:09 and 07:39:22.

Not sure if that's useful to help you track anything down or not. (I did nothing to any of my services to stop it happening but I then have successful requests being processed at 07:40:06)

[brunolau]

I managed to temporarily auto-mitigate the issue by creating custom Entity framework execution strategy that once encounters the MARS error, closes the connection, clears the pool and reopens. Since I've introduced this workaround, I've been running errorless for 3 days.

To sum it up. Once the error is encountered, closing & reopening the error connection + clearing the connection pool seems to re-establish working order

[PolitovArtyom]

We are getting the same problem.
Errors starts occuring in same moment on different servers. Application pool restart helps,

Application Server
Microsoft Windows Server 2012 R2 Standard 6.3.9600 Build 9600
dotnet host Version: 2.1.5 Commit: 290303f510
.NET Core runtimes installed:
Microsoft.AspNetCore.All 2.1.5 [C:\Program Files\dotnet\shared\Microsoft.AspNetCore
Microsoft.AspNetCore.App 2.1.5 [C:\Program Files\dotnet\shared\Microsoft.AspNetCore
Microsoft.NETCore.App 2.1.5 [C:\Program Files\dotnet\shared\Microsoft.NETCore.App]

Database
Microsoft SQL Server 2017 (RTM-CU11) (KB4462262) - 14.0.3038.14 (X64) Sep 14 2018 13:53:44 Copyright (C) 2017 Microsoft Corporation Enterprise Edition (64-bit) on Windows Server 2012 R2 Standard 6.3 (Build 9600: )

Application
Dapper 1.50.5
Microsoft.AspNetCore.App 2.1.5

[sepehr1014]

We're having the same issue. Any solutions?

[PolitovArtyom]

@afsanehr
Do you need some additional data for investigation?
I can try to get application and tcp dump while errors occuring