Newtonsoft.Json icon indicating copy to clipboard operation
Newtonsoft.Json copied to clipboard

It's currently not possible to deserialize dictionaries with complex classes as keys.

Open petermorlion opened this issue 9 years ago • 10 comments

Not really an issue, but something I needed and might be handy for other people too.

When serializing a Dictionary, it works great for dictionaries where the keys are primitive types (i.e. string, int,...). However, when the key is a more complex class, serializing will work, but deserializing won't.

The solution can be a custom converter, but I wrote a converter that can handle dictionaries with any type of key (see the Gist for the implementation).

Possible downsides are:

  • the JSON is more verbose, so larger in size
  • performance?

petermorlion avatar Mar 19 '15 11:03 petermorlion

@petermorlion Other downsides:

  • No support for non-generic dictionaries
  • No support for generic dictionaries not implementing non-generic dictionary interface (and CanConvert method doesn't check that)
  • No support for read-only dictionaries
  • It doesn't respect object type
  • It doesn't use existing value
  • Reflection is performed on every conversion
  • etc.

I wish Json.NET included "nonsensical dictionary serialization" as an option — the only "missing feature" compared to built-in serializers will finally be implemented. :smile:

P.S. foreach (var key in dictionary.Keys) { ... key ... dictionary[key] ... } isn't the most performant way of enumerating dictionaries. Use foreach (DictionaryEntry entry in dictionary) { ... entry.Key ... entry.Value ... } instead.

Athari avatar Mar 19 '15 23:03 Athari

Thanks for the foreach tip, didn't know that.

I realize my code isn't fit for some usages, but I've been using it for some time now. Performance-wise, I haven't really noticed anything significant, and it works great for my scenario's. But YMMV of course.

petermorlion avatar Mar 25 '15 10:03 petermorlion

A TypeConverter will deserialize complex classes in keys.

JamesNK avatar Jun 14 '15 08:06 JamesNK

@JamesNK FYI your solution of using a type converter still does not work for most cases because for some reason [OnDeserialized] doesn't get called for them

https://stackoverflow.com/questions/40157640/newtonsoft-jsoncontract-ondeserializedcallbacks-not-called-for-primitive-types

mklingen avatar Oct 08 '17 19:10 mklingen

@JamesNK but what about when you don't own the key type, such as IPAddress? I can't really make a type converter for that class, can I?

jorgensigvardsson avatar Nov 22 '17 07:11 jorgensigvardsson

I can try to make a PR with a setting to enable ser/desser of key/value list from/to dictionary, if this is something that will be accepted. But what to call the setting? I imagine a simple bool setting JsonSerialazationSettings.ComplexKeyDictioraryAsKeyValueList. Or alternative, a handler JsonSerialazationSettings.ComplexKeyDictioraryHandler, where i guess the only 2 things make sense to configure is name of "key" and "value" itself (thou I can not imagine anything than "key" and "value" to make sense to me) Edit: I am using this now https://stackoverflow.com/a/67153528/2671330 and it satisfy my need I think.

osexpert avatar Apr 17 '21 17:04 osexpert

I am surprised this issue has not been addressed to date. Anyway, there is a limited solution to this problem -- one of the checks deep inside the library looks for op_Explicit/op_Implicit, so if your custom dictionary key type declares a static explicit operator T(string value), this one will be used during deserialization (and deserialization is the main pain point for me).

TypeConverter is a poor solution, since when applied with TypeConverterAttribute, it loses any information about the target type, nor is it declared with Inherit = true. So while JsonConverter is inherited and can be made generic (as ReadJson/WriteJson receives information about the actual type, to/from which serialization happens), TypeConverter is not, and requires the attribute to be applied to every type that can be potentially used as a dictionary key.

Unless I am missing something, this is extremely frustrating and all solutions I found so far are just different kinds of hacks.

P.S.: With .NET 7, one solution could be relying on ISpanParsable/ISpanFormattable and IParsable/IFormatable if all existing checks fail.

Ilia-Kosenkov avatar Nov 23 '22 14:11 Ilia-Kosenkov

Also ran into this. just wanted to make a simple small record for the key, but when I have to create TypeConverter or something else, it's not a simple solution anymore.

praschl avatar Dec 29 '23 10:12 praschl

Thanks for reviving this thread @praschl. While I gave up on having some 'official' support on this issue and completely forgot I asked a question here, I have actually found a proper solution to the problem, which works nicely with inheritance (but has some limitations). I know it does not fall into the 'simple solution' category, but it might help.

internal sealed class JsonDictionaryKeyTypeConverter : TypeConverter
{
    private readonly Type _convertedType;
    private readonly bool _isQuotedWhenSerialized;
    
    // Converted type is injected into constructor automatically
    public JsonDictionaryKeyTypeConverter(Type convertedType)
    {
          _convertedType = convertedType;
         // This is only needed if you want to prettify your dictionary keys and avoid unnecessary double quotes, 
         // or if you need to communicate with other frameworks. Should be computed based on the logic of your app.
          _isQuotedWhenSerialized = IsQuotedWhenSerialized(convertedType); // This is a placeholder for the actual method
    }
    
    public override bool CanConvertFrom(ITypeDescriptorContext? context, Type sourceType) => sourceType == typeof(string);

    public override bool CanConvertTo(ITypeDescriptorContext? context, Type? destinationType) => destinationType == typeof(string);

    public override object? ConvertFrom(ITypeDescriptorContext? context, CultureInfo? culture, object value)
    {
        if (!CanConvertFrom(context, value.GetType()) || value is not string s)
        {
            throw new InvalidOperationException($"Source type '{value.GetType().Name}' not supported.");
        }

        if (_isQuotedWhenSerialized)
        {
            s = $"\"{s}\"";
        }

        return JsonConvert.DeserializeObject(s, _convertedType);
    }

    public override object ConvertTo(ITypeDescriptorContext? context, CultureInfo? culture, object? value, Type destinationType)
    {
        if (!CanConvertTo(context, destinationType))
        {
            throw new InvalidOperationException($"Destination type '{destinationType.Name}' not supported.");
        }

        var json = JsonConvert.SerializeObject(value);
        if (_isQuotedWhenSerialized)
        {
            json = json.Trim('"');
        }

        return json;
    }
}

Then you can just slap it onto your type (the logic is inherited as with json converter), something like this:

[TypeConverter(typeof(JsonDictionaryKeyTypeConverter))]
[JsonConverter(typeof(MyNewtonsoftJsonConverter))]
public sealed record MyRecord;

This approach currently powers multiple microservices. We use a similar type converter for a bunch of strong id types, i.e. thin wrappers around non-empty strings or Guids.

Ilia-Kosenkov avatar Dec 29 '23 21:12 Ilia-Kosenkov

Tried that, but I only added the TypeConverter to my record that serves as the dictionary key. Since that gave me a Stackoverflow Exception from the depths of Newtonsoft, I suppose, I would still have to write a JsonConverter for each key and add that, too?

In this case, unfortunately, it's not the quick and easy solution I was looking for, but it's still a good one 😊 Thank you.

praschl avatar Dec 30 '23 13:12 praschl