ONNX Transform Crashing or Freezing

[vaeksare]

ONNX Transform occasionally crashes or freezes when running certain onnx models (currently found with the Split operator).

• The error is non deterministic and varies based on different machines
• On some machines, the ML.NET process will freeze, while on others it will simply silently crash without producing any output
• The larger the input, the more likely the issue seems to occur - almost never happens with <100 inputs, sometimes happens with 100-300 inputs, and almost always happens with >300 inputs
• When ML.NET is built in debug mode, the error will never occur. It only happens if it is built in release mode
• The issue is not tied to any specific input. Smaller inputs will never produce an error, while larger ones almost always will (in release mode)

[Zruty0]

Could you give an update please? Is it actively worked on?

[vaeksare]

@Zruty0 Jignesh is currently investigating this issue.

[jignparm]

To close this out, the crash was occurring at the Sonoma Tensor.CopyTo(List) function. Replacing it with a call to CopyTo(float []) has resolved the issue. The backend Sonoma code for these two functions is quite different, especially memory management. Arrays need to have memory pre-allocated, whereas Lists are appended onto one element at a time. Additionally, the CopyTo(float[]) function explicitly pins the destination memory using "fixed" statement before copying the contents, whereas the CopyTo(List) function is not using "fixed", and therefore not explicitly pinning the destination during the append operations.

We'll need to fix the CopyTo(list) function in Sonoma, but for now using the more stable (and more efficient) CopyTo(float[]) function has fixed the issue in several trial runs offline.

[shauheen]

This issue is not fixed only avoided by #1310