AccessViolationException PredictionEngine when 100-200 concurrent predictions running async

[vslynko]

System information

• Windows 10 64bit latest. 4 core CPU with hyperthreading.
• Main app net48, that loads dependency in net472 that loads Microsoft.ML.Tensorflow 1.4:

Issue

• Multiple concurrent tasks scheduled with Task.WhenAll. All tasks perform the same lambda, that involves calling PredictionEngineBase.Predict call. Number of tasks in parallel stacks window is about 100-200.
• AccessViolationException thrown with message "Attempted to read or write protected memory. This is often an indication that other memory is corrupt."
• As in case when number of tasks is not that big, ~30-50 I would expect no exception.

Source code / logs

Two different stack traces point to about same location in the code. The difference is that I rearrange a little async workflow by switch couple of async tasts around.

   at System.SpanHelpers.CopyTo[T](T& dst, Int32 dstLength, T& src, Int32 srcLength)
   at System.Span`1.TryCopyTo(Span`1 destination)
   at Microsoft.ML.TensorFlow.TensorTypeExtensions.CopyTo[T](Tensor tensor, Span`1 values)
   at Microsoft.ML.Transforms.TensorFlowTransformer.Mapper.<>c__DisplayClass9_0`1.<MakeGetter>b__4(VBuffer`1& dst)
   at Microsoft.ML.Data.TypedCursorable`1.TypedRowBase.<>c__DisplayClass8_0`1.<CreateDirectVBufferSetter>b__0(TRow row)
   at Microsoft.ML.Data.TypedCursorable`1.TypedRowBase.FillValues(TRow row)
   at Microsoft.ML.PredictionEngineBase`2.Predict(TSrc example)
   at InstaFlow.TensorFlow.Gp2.TfGpBaseV2`1.Predict(TfGpSeriesV2 series)
   at InstaFlow.TensorFlow.Gp2.TfGpBaseV2`1.CorrectInternal(T model, TfInput input)
   at InstaFlow.TensorFlow.TfEstimatorV2.TimerAction[T](Func`1 func)
   at InstaFlow.TensorFlow.TfEstimatorV2.EstimateAndEvaluate(TfInput input)

   at Tensorflow.c_api.TF_TensorByteSize(IntPtr tensor)
   at Tensorflow.Tensor.get_bytesize()
   at Tensorflow.Tensor.get_size()
   at Microsoft.ML.TensorFlow.TensorTypeExtensions.CopyTo[T](Tensor tensor, Span`1 values)
   at Microsoft.ML.Transforms.TensorFlowTransformer.Mapper.<>c__DisplayClass9_0`1.<MakeGetter>b__4(VBuffer`1& dst)
   at Microsoft.ML.Data.TypedCursorable`1.TypedRowBase.<>c__DisplayClass8_0`1.<CreateDirectVBufferSetter>b__0(TRow row)
   at Microsoft.ML.Data.TypedCursorable`1.TypedRowBase.FillValues(TRow row)
   at Microsoft.ML.PredictionEngineBase`2.Predict(TSrc example)
   at InstaFlow.TensorFlow.Estimator.TfEstimatorBase.Predict(TfSeries series) in C:\src\instaflow\dotnet\InstaFlow.TensorFlow\Estimator\TfEstimatorBase.cs:line 101

[eerhardt]

This appears to be the same issue as dotnet/runtime#13701.

@jkotas @GrabYourPitchforks @BruceForstall @ericstj - FYI another instance of this issue on .NET Framework.

[BruceForstall]

@briansull @CarolEidt Apparently another .NET Framework case of https://github.com/dotnet/coreclr/issues/27924 (dotnet/runtime#13818)

[najeeb-kazmi]

@BruceForstall I see that the underlying issue has been fixed. Is a 4.8.1 patch expected or will the fix be made available in .NET 5?

[BruceForstall]

@BruceForstall I see that the underlying issue has been fixed. Is a 4.8.1 patch expected or will the fix be made available in .NET 5?

A decision hasn't been made. fwiw, it's very difficult to get changes in to 4.8 servicing releases.

[CarolEidt]

The fix has been back-ported to 3.1, and will also be in .NET 5.

[mstfbl]

Hi @vslynko, can you provide sample code to replicate this issue? I'm trying to see if this is actually fixed when run on NetCoreApp 3.1. Thanks!

Edit: The sample code provided in the aforementioned dotnet/runtime#13701 does make use of multiple concurrent tasks scheduled with Task.WhenAll. I ran this on Release 3.0 and 3.1 builds, and in both instances, the program ran successfully without crashes.

[mstfbl]

I have merged the PR to update our builds to run on NetCoreApp 3.1, and I'm closing this issue as @CarolEidt said the fix for this has been back-ported to 3.1. @vslynko please run your code on NetCoreApp 3.1, and let us know if your issue has not been fixed.