[API Proposal]: Support Custom Data on `JsonSerializerOptions`

[stevejgordon]

Background and motivation

While working on a client library for Elasticsearch, making heavy use of System.Text.Json, a common issue I have run into is the need to access some additional state inside custom converters. The Elasticsearch client has a settings class ElasticsearchClientSettings which includes state used to infer values for certain types at runtime.

Currently, this forces an instance of each converter which requires access to the settings to be created in advance and added to the JsonSerializerOptions (example). These converters can then accept an ElasticsearchClientSettings instance via their constructor. Some of these instances may never be needed if the types they (de)serialize are not used by consumers of our library. Additionally, we have some converters which we code generate which makes creating such instances and adding them to the JsonSerializerOptions more complicated.

I have come up with a rather nasty workaround where I register a converter purely to hold state, which can then be retrieved from the options to gain access to the settings.

internal sealed class ExtraSerializationData : JsonConverter<ExtraSerializationData>
{
    public ExtraSerializationData(IElasticsearchClientSettings settings) => Settings = settings;

    public IElasticsearchClientSettings Settings { get; }

    public override ExtraSerializationData? Read(ref Utf8JsonReader reader, Type typeToConvert, JsonSerializerOptions options) => throw new NotImplementedException();
    public override void Write(Utf8JsonWriter writer, ExtraSerializationData value, JsonSerializerOptions options) => throw new NotImplementedException();
}

This can then be accessed in the Write method of a custom converter:

public override void Write(Utf8JsonWriter writer, SortOptions value, JsonSerializerOptions options)
{
    writer.WriteStartObject();

    if (value.AdditionalPropertyName is IUrlParameter urlParameter)
    {
        var extraData = options.GetConverter(typeof(ExtraSerializationData)) as ExtraSerializationData;
        var propertyName = urlParameter.GetString(extraData.Settings);
        writer.WritePropertyName(propertyName);
    }
    else
    {
        throw new JsonException();
    }
    
    JsonSerializer.Serialize(writer, value.Variant, value.Variant.GetType(), options);
    writer.WriteEndObject();
}

API Proposal

I would like to propose adding support to attach extra custom data (a property bag) to JsonSerializerOptions, to be accessible inside the Read and Write methods of custom converters derived from JsonConveter<T> by accessing the JsonSerializerOptions.

public sealed class JsonSerializerOptions
{
    public IReadOnlyDictionary<string, object> CustomData { get; set; } = new Dictionary<string, object>(0); // some cached default empty value
}

This adds a property to act as a property bag for user-provided data. I'm proposing this be exposed as IReadOnlyDictionary to avoid new items being added/modified after the options become immutable. It could be IDictionary to support scenarios where some converters need to add data for subsequent use, although that sounds risky. I imagine this should be initialised with a cached empty dictionary in cases where the user does not explicitly set this.

API Usage

Define options with custom data:

var options = new JsonSerializerOptions
{
    CustomData = new Dictionary<string, object> {{ "settings", new Settings() }} 
}

Access within a custom converter:

public override void Write(Utf8JsonWriter writer, SortOptions value, JsonSerializerOptions options)
{
    writer.WriteStartObject();

    if (value.AdditionalPropertyName is IUrlParameter urlParameter && options.CustomData.TryGetValue("settings", out var settings))
    {
        var propertyName = urlParameter.GetString(settings);
        writer.WritePropertyName(propertyName);
    }
    else
    {
        throw new JsonException();
    }
    
    JsonSerializer.Serialize(writer, value.Variant, value.Variant.GetType(), options);
    writer.WriteEndObject();
}

Alternative Designs

If the JsonSerializerOptions were unsealed, that could also support my scenario. We could define a subclass of JsonSerializerOptions with extra properties. Inside our converters that require that data, they could try casting to the derived type and access any necessary extra properties.

- public sealed class JsonSerializerOptions
+ public class JsonSerializerOptions
{
}

Risks

This could potentially be misused to hold mutable types which could cause thread-safety issues. Documentation could be added to guide the intended usage.

Tagging subscribers to this area: @dotnet/area-system-text-json, @gregsdennis
See info in area-owners.md if you want to be subscribed.

Issue Details

---

Background and motivation

While working on a client library for Elasticsearch, making heavy use of System.Text.Json, a common issue I have run into is the need to access some additional state inside custom converters. The Elasticsearch client has a settings class ElasticsearchClientSettings which includes state used to infer values for certain types at runtime.

Currently, this forces an instance of each converter which requires access to the settings to be created in advance and added to the JsonSerializerOptions (example). These converters can then accept an ElasticsearchClientSettings instance via their constructor. Some of these instances may never be needed if the types they (de)serialize are not used by consumers of our library. Additionally, we have some converters which we code generate which makes creating such instances and adding them to the JsonSerializerOptions more complicated.

I have come up with a rather nasty workaround where I register a converter purely to hold state, which can then be retrieved from the options to gain access to the settings.

internal sealed class ExtraSerializationData : JsonConverter<ExtraSerializationData>
{
	public ExtraSerializationData(IElasticsearchClientSettings settings) => Settings = settings;

	public IElasticsearchClientSettings Settings { get; }

	public override ExtraSerializationData? Read(ref Utf8JsonReader reader, Type typeToConvert, JsonSerializerOptions options) => throw new NotImplementedException();
	public override void Write(Utf8JsonWriter writer, ExtraSerializationData value, JsonSerializerOptions options) => throw new NotImplementedException();
}

This can then be accessed in the Write method of a custom converter:

public override void Write(Utf8JsonWriter writer, SortOptions value, JsonSerializerOptions options)
{
    writer.WriteStartObject();

    if (value.AdditionalPropertyName is IUrlParameter urlParameter)
    {
        var extraData = options.GetConverter(typeof(ExtraSerializationData)) as ExtraSerializationData;
        var propertyName = urlParameter.GetString(extraData.Settings);
        writer.WritePropertyName(propertyName);
    }
    else
    {
        throw new JsonException();
    }
    
    JsonSerializer.Serialize(writer, value.Variant, value.Variant.GetType(), options);
    writer.WriteEndObject();
}

API Proposal

I would like to propose adding support to attach extra custom data (a property bag) to JsonSerializerOptions, to be accessible inside the Read and Write methods of custom converters derived from JsonConveter<T> by accessing the JsonSerializerOptions.

public sealed class JsonSerializerOptions
{
    public IReadOnlyDictionary<string, object> CustomData { get; set; } = new Dictionary<string, object>(0); // some cached default empty value
}

This adds a property to act as a property bag for user-provided data. I'm proposing this be exposed as IReadOnlyDictionary to avoid new items being added/modified after the options become immutable. It could be IDictionary to support scenarios where some converters need to add data for subsequent use, although that sounds risky. I imagine this should be initialised with a cached empty dictionary in cases where the user does not explicitly set this.

API Usage

Define options with custom data:

var options = new JsonSerializerOptions
{
    CustomData = new Dictionary<string, object> {{ "settings", new Settings() }} 
}

Access within a custom converter:

public override void Write(Utf8JsonWriter writer, SortOptions value, JsonSerializerOptions options)
{
    writer.WriteStartObject();

    if (value.AdditionalPropertyName is IUrlParameter urlParameter && options.CustomData.TryGetValue("settings", out var settings))
    {
        var propertyName = urlParameter.GetString(settings);
        writer.WritePropertyName(propertyName);
    }
    else
    {
        throw new JsonException();
    }
    
    JsonSerializer.Serialize(writer, value.Variant, value.Variant.GetType(), options);
    writer.WriteEndObject();
}

Alternative Designs

Alternative Design

If the JsonSerializerOptions were unsealed, that could also support my scenario. We could define a subclass of JsonSerializerOptions with extra properties. Inside our converters that require that data, they could try casting to the derived type and access any necessary extra properties.

Risks

This could potentially be misused to hold mutable types which could cause thread-safety issues. Documentation could be added to guide the intended usage.

Author:	stevejgordon
Assignees:	-
Labels:	api-suggestion, area-System.Text.Json
Milestone:	-

[eiriktsarpalis]

It's a reasonable requirement, but I don't think JsonSerializerOptions is the right type to attach state scoped to a particular serialization operation. As I think you're already pointing out, JsonSerializerOptions is meant as a singleton configuration/caching context, as such attaching state there would make it susceptible to race conditions (or encourage the anti-pattern of creating a new options instance per serialization operation).

Related to #63795, we've been thinking about exposing async converter primitives which would also involve exposing the internal WriteState/ReadState types. It should be possible to let users attach custom state to these types a la StreamingContext.

[stevejgordon]

Thanks for the feedback @eiriktsarpalis. To be clear, in our use case, our ElasticsearchClientSettings are also immutable and required across many converters for many serialization operations for the lifetime of a client instance (which is reused for the life of consumer apps). We create and use a single instance of JsonSerializerOptions.

There's already potential for misuse if developers create new instances of the options per serialization operation. I'm not sure this promotes that much more for the common scenario of attaching additional context needed for many/all serialization in converters.

I'll have to dig into the proposal for async converters to understand if custom state there would meet our need. Would we replace the existing converters with async ones? Would those apply to all scenarios, even synchronous serializer methods?

[eiriktsarpalis]

Would we replace the existing converters with async ones?

It would most likely involve exposing a couple of virtual methods in JsonConverter<T> that are currently marked internal. We haven't produced any prototypes yet, so that plan might be subject to change.

Would those apply to all scenarios, even synchronous serializer methods?

Yes, the converter methods that pass serialization state are used in both synchronous and asynchronous serialization. It just happens that async converters are implemented by passing their state machines via the serialization state objects.

[davidfowl]

It should be possible to let users attach custom state to these types a la StreamingContext.

Do you have an example of what this would look like?

[eiriktsarpalis]

Do you have an example of what this would look like?

Feature needs prototyping before we can propose something concrete, but roughly speaking it might look as follows:

var writer = new Utf8JsonWriter();
var state = new WriteState { UserState = new MyState() };
string json = JsonSerializer.Serialize(writer, new Foo(), ref state);

public class MyConverter : JsonResumableConverter<Foo>
{
    public bool TryWrite(Utf8JsonWriter writer, Foo value,  JsonSerializerOptions options, ref WriteState state)
    {
         var myState = (MyState)state.UserState;
         /* serialization logic using state */
    }
}

where JsonResumableConverter<T> would be roughly equivalent to the existing internal class of the same name.

[davidfowl]

What happens when you don't control the callsite responsible for calling Serialize/Deserailize (like when using a framework)? Attaching this to options seems like the logical place since it:

• Already flows everywhere
• Is user defined options state for these calls.

[eiriktsarpalis]

That might be an option, although given the nature of JsonSerializerOptions I would guess a factory (i.e. Func<object>) might be more appropriate. It would still be less flexible than an argument being passed to serialization methods directly though.

[eiriktsarpalis]

@stevejgordon in the meantime, have you considered using a ConditionalWeakTable to dynamically attach data to options instances?