Safe Exception Handling

[bmteller]

Elixir and Erlang/OTP versions

Erlang/OTP 26 [erts-14.2.5.3] [source] [64-bit] [smp:10:10] [ds:10:10:10] [async-threads:1] [jit]

Elixir 1.17.3 (compiled with Erlang/OTP 26)

Operating system

Linux

Current behavior

There is no safe way to call Exception.message to get a description of an Exception without risking allocating a huge amount of memory. This is due to parts of the Elixir core API and maybe parts of erlang capturing arbitrary terms as part of the exception and then the message calling inspect on these terms. This is also a problem with STACKTRACE as well because some of the erlang methods will also capture arbitrary terms and calling inspect on such a STACKTRACE can also cause a huge amount of memory to be allocated. Though, users of STACKTRACE should probably just be more careful. This is an issue when using a crash handling product like Sentry and maybe even be an issue with logging handlers that might try to print a stack trace or call Exception.message to print the message of an Exception. I think the main cause of this is inspect does not actually limit the output for tree structures. I think ideally inspect would just have a parameter controlling the total amount of bytes to generate so callers could choose a suitably safe value.

See also an issue raised with Sentry: getsentry/sentry-elixir#881

Here is an example of the issue occurring where the Exception captures a large tree structure and then allocates a large amount of memory to print it out. However, in production I think we were seeing it allocate ~ 4 gig of memory when handling an exception. Though, I think in our case the underlying Exception.message size was a bit lower because this 4GB also included the inspect(__STACKTRACE__) which would have had a similar size and these values duplicated multiple times due to the way Sentry/HTTPoison works.

big = fn(y, d, n) -> if d == 0 do; []; else; Enum.map(1..n, fn(_) -> y.(y, d - 1, n) end) end end
evil_map = %{"hello" => big.(big, 7, 5)}; 1
evil_exception = try do; Map.fetch!(evil_map, "world"); rescue e -> e; end; 1
byte_size(Exception.message(evil_exception))

> 668008

Expected behavior

inspect should have an option to limit output and ideally Exception.message implementations in the core library should have a reasonable limit on how much memory they allocate.

[josevalim]

We should fix the inspect and particular map exception to allocate fewer entries. I will look into it. Thank you.