Consider clearing JsonSerializerOptions.Caching cache using a timer, not just based on incoming calls

[SteveSandersonMS]

First, it's totally possible I'm misinterpreting what STJ does here. We got this report saying that a Blazor Server app was holding hundreds of MB in memory for longer than expected, and using the VS memory inspection tools, it looks like at least one (possibly the only) thing pinning this data is rooted in JsonSerializerOptions's s_cache. The cache uses WeakReference for values, but the problem is with the keys.

It looks like the only thing that clears dangling entries is subsequent calls into the JSON serializer. This is likely sufficient in server scenarios where the server will never stop processing requests for any long duration, and even if it does, people probably don't care about freeing any memory until the server experiences memory pressure. However, for desktop scenarios, an app may stop doing things for a long time, and during that period, people really want it to release memory that's no longer required.

Suggestion

Instead of triggering TryEvictDanglingEntries only when there's a call to GetCachingContext, consider also having some guarantee like "it will always clear dangling entries within 10s even if there are no such incoming calls" or similar.

This isn't blocking anything in Blazor directly, but would be a good solution for people using STJ directly in desktop apps, or Blazor Server as an implementation detail of a desktop app (like this person).

Tagging subscribers to this area: @dotnet/area-system-text-json, @gregsdennis
See info in area-owners.md if you want to be subscribed.

Issue Details

---

First, it's totally possible I'm misinterpreting what STJ does here. We got this report saying that a Blazor Server app was holding hundreds of MB in memory for longer than expected, and using the VS memory inspection tools, it looks like at least one (possibly the only) thing pinning this data is rooted in JsonSerializerOptions's s_cache. The cache uses WeakReference for values, but the problem is with the keys.

It looks like the only thing that clears dangling entries is subsequent calls into the JSON serializer. This is likely sufficient in server scenarios where the server will never stop processing requests for any long duration, and even if it does, people probably don't care about freeing any memory until the server experiences memory pressure. However, for desktop scenarios, an app may stop doing things for a long time, and during that period, people really want it to release memory that's no longer required.

Suggestion

Instead of triggering TryEvictDanglingEntries only when there's a call to GetCachingContext, consider also having some guarantee like "it will always clear dangling entries within 10s even if there are no such incoming calls" or similar.

This isn't blocking anything in Blazor directly, but would be a good solution for people using STJ directly in desktop apps, or Blazor Server as an implementation detail of a desktop app (like this person).

Author:	SteveSandersonMS
Assignees:	-
Labels:	area-System.Text.Json
Milestone:	-

[SteveSandersonMS]

Also, if this caching behavior was introduced in 7.0, then it could be argued that it's a regression. (Subjective, I know.)

[jkotas]

Is the actual problem the shortcut described in https://github.com/dotnet/runtime/blob/main/src/libraries/System.Text.Json/src/System/Text/Json/Serialization/JsonSerializerOptions.Caching.cs#L165-L167? Should this code rather use ConditionalWeakTable to avoid unnecessarily keeping the keys around?

[eiriktsarpalis]

Thanks @SteveSandersonMS, your analysis is accurate. At the time when the cache was implemented, we debated as to whether there should be a timer that ran evictions in the background but ultimately found that approach to be unsavory. In hindsight, I had not anticipated that user configuration in JsonSerializerOptions would realistically capture a large amount of memory but I guess I should have known better :-)

That being said, it is almost certain that the application in question is creating single-use JsonSerializerOptions instances, a known anti-pattern that would have caused performance problems even in .NET 6. The author of the original issue could almost certainly address the problem by creating singleton options instances. We're planning on adding an analyzer in the future which should help users avoid that particular anti-pattern.

Also, if this caching behavior was introduced in 7.0, then it could be argued that it's a regression. (Subjective, I know.)

I think we could definitely improve the cache based on this feedback but I don't necessarily think this requires a .NET 7 fix. I think we should wait on more user feedback in case it arrives.

Is the actual problem the shortcut described in https://github.com/dotnet/runtime/blob/main/src/libraries/System.Text.Json/src/System/Text/Json/Serialization/JsonSerializerOptions.Caching.cs#L165-L167? Should this code rather use ConditionalWeakTable to avoid unnecessarily keeping the keys around?

Original prototypes did use ConditionalWeakTable, effectively performing a linear search over all JsonSerializerOptions instances in the process:

runtime/src/libraries/System.Text.Json/src/System/Text/Json/Serialization/JsonSerializerOptions.cs

Lines 133 to 141 in f96723a

	internal static class TrackedOptionsInstances
	{
	/// <summary>Tracks all live JsonSerializerOptions instances.</summary>
	/// <remarks>Instances are added to the table in their constructor.</remarks>
	public static ConditionalWeakTable<JsonSerializerOptions, object?> All { get; } =
	// TODO https://github.com/dotnet/runtime/issues/51159:
	// Look into linking this away / disabling it when hot reload isn't in use.
	new ConditionalWeakTable<JsonSerializerOptions, object?>();
	}

But IIRC enumeration performance for the particular type was bad enough that it almost invalidated any performance benefits gained from caching.

[willdean]

That being said, it is almost certain that the application in question is creating single-use JsonSerializerOptions instances, a known anti-pattern that would have caused #40072. The author of the original issue could almost certainly address the problem by creating singleton options instances.

I think the original author was using an unmodified (except for the ability to manually trigger GC) Blazor template to demonstrate the problem.

[chrdlx]

That being said, it is almost certain that the application in question is creating single-use JsonSerializerOptions instances, a known anti-pattern that would have caused performance problems even in .NET 6. The author of the original issue could almost certainly address the problem by creating singleton options instances. We're planning on adding an analyzer in the future which should help users avoid that particular anti-pattern.

Hey @eiriktsarpalis Just in case you couldn't read the complete original issue (I'm the author) and just for clarification, the application in question which you are referring to is the Blazor framework itself using the default Blazor Server template.
So, if there're problems (like the anti-patterns you mentioned), they are in .NET 7 RC1 itself and its default templates, not user code.

Regards!

[chrdlx]

Regarding what @SteveSandersonMS mentioned about a timer, you can configure the retention period of circuits for Blazor, for example:

builder.Services.AddServerSideBlazor(options => {
                options.DisconnectedCircuitMaxRetained = 5;
                options.DisconnectedCircuitRetentionPeriod = TimeSpan.FromSeconds(10);
            });

So maybe this should be tied somehow with the serializer cache clearing call, I guess it wouldn't make much sense to configure different amounts of time for each since one depends on the other to be useful.
I mean, the Blazor App could disconnect forcibly all the users and release their circuits, but if the instances are being held by the serializer cache, memory won't be released, which is the intention.

Regards!

[eiriktsarpalis]

Hi @chrdlx, that's interesting. If there are indeed keys in the cache that require eviction that would suggest to me that Blazor is using temporary options instances, which should be avoided under most circumstances. I'm not too familiar with Blazor internals however, is there anything in its configuration that could prompt single-use instances to be created? Perhaps some options instance only used for initialized purposes that then gets garbage collected? cc @SteveSandersonMS