Open polymorphism, classDiscriminator, keep "type" property in the class.

[hovi]

What is your use-case and why do you need this feature?

I have existing design. I have class that I am serializing, which contains "BlockType" property. Currently there's only one class, but I wanted to switch to open polymorphism and add subclasses, each having different properties based on this "BlockType", original class being default fallback with no additional properties.

I cannot get it working, because I get message:

Polymorphic serializer for class spaceEngineers.model.Block has property 'BlockType' that conflicts with JSON class discriminator. You can either change class discriminator in JsonConfiguration, rename property with @SerialName annotation or fall back to array polymorphism

If I understand correctly, framework doesn't like, that this "BlockType" property is used both as normal property in class and also as classDiscriminator. But why is that? I need to keep it while having working polymorphism and I see no reason, why it should not work.

Describe the solution you'd like

Either allow using classDiscriminator and a class property at the same time or add configuration, that allows that.

[Dominaezzz]

You can get the discriminator from the type, no?

What if there's a mismatch between the discriminator and the type?

[hovi]

It is a single JSON field, in my case called "BlockType", which is discriminator at the same time so there cannot be mismatch.

[aSemy]

This is useful for me as I'm using the discriminator field to generate Typescript classes (using https://github.com/ntrrgc/ts-generator).

I've found a couple of potential work-arounds that I'm experimenting with:

import kotlin.reflect.full.findAnnotation
import kotlin.reflect.full.starProjectedType
import kotlinx.serialization.SerialName
import kotlinx.serialization.Serializable
import kotlinx.serialization.json.JsonClassDiscriminator


@Serializable
@JsonClassDiscriminator("object_name")
sealed class EventData() {

  //  get the discriminator as a field (the field name, objectName, is unimportant)
  // this must be a delegated field so there's no backing field, so kxs ignores it
  val objectName: String by lazy {
    // this probably won't work for generic classes?
    kotlinx.serialization.serializer(this::class.starProjectedType).descriptor.serialName
  }

  // alternative: find the @SerialName annotation.
  // Again, the field must be delegated so there's no backing field. 
  // This requires that every subclass has @SerialName
  val objectNameByAnnotation: String by lazy {
    requireNotNull(this::class.findAnnotation<SerialName>()?.value) {
      "Couldn't find @SerialName for ${this::class}!"
    }
  }
}

[pdvrieze]

The way I read it is that you are effectively using blockType as a type discriminator, but in a more flexible way. You can't really serialize this directly but have to use a custom serializer. It may be possible to use a delegate type with all possible properties and convert from that, or just write the full implementation by hand.

[Frontrider]

This issue can be fixed, as of now Kotlin Serialization supports the original issue.

My case was with AWS Step functions, which use Type as the discriminator.

Replace the classDiscriminator with BlockType for OP's case.

@OptIn(ExperimentalSerializationApi::class)
internal val json = Json {
    namingStrategy = JsonNamingStrategy { descriptor, elementIndex, serialName ->
        serialName.asTitle()
    }
    classDiscriminator = "Type"
}

Still not ideal, as the root element of a step function does not support the "Type" field so it will create an error. But at least it fixes it for most cases if nothing else.

[pramod-knidal]

is it possible to achieve it using custom serializers? will this feature be added in future versions?

[sandwwraith]

No, I think it's impossible to write custom serializer to work around this. I think yes, this should be implemented eventually

[pramod-knidal]

Hmmm. Are there any plans to support nested hierarchies? Currently, the nested hierarchy's class discriminator cannot be defined. It throws an error. If so, how soon? near or distant future.

@Serializable
@JsonClassDiscriminator("a_type")
sealed class BaseA {
     
    @Serializable
    data class A_0(): BaseA()
    
    @Serializable
    data class A_1(): BaseA()
     
    @Serializable
    @JsonClassDiscriminator("b_type")
     sealed class BaseB {

            @Serializable
            data class B_0(): BaseB()
            
            @Serializable
            data class B_1(): BaseB()
     }

}

[Frontrider]

Hmmm. Are there any plans to support nested hierarchies? Currently, the nested hierarchy's class discriminator cannot be defined. It throws an error. If so, how soon? near or distant future.

@Serializable
@JsonClassDiscriminator("a_type")
sealed class BaseA {
     
    @Serializable
    data class A_0(): BaseA()
    
    @Serializable
    data class A_1(): BaseA()
     
    @Serializable
    @JsonClassDiscriminator("b_type")
     sealed class BaseB {

            @Serializable
            data class B_0(): BaseB()
            
            @Serializable
            data class B_1(): BaseB()
     }

}

Add a SerialName annotation to your type (NOT any of the fields!), that does exactly what you are doing here.

[pramod-knidal]

Add a SerialName annotation to your type (NOT any of the fields!), that does exactly what you are doing here.

For the code below, we are getting the kotlinx.serialization.json.internal.JsonDecodingException: Polymorphic serializer was not found for class discriminator 'base-b'

@Serializable
sealed class BaseA {
     
    @Serializable
    @SerialName("a-0")
    data class A_0(): BaseA()
    
    @Serializable
    @SerialName("a-1")
    data class A_1(): BaseA()
     
    @Serializable
    @SerialName("base-b")
    @JsonClassDiscriminator("b_type")
     sealed class BaseB {

            @Serializable
            @SerialName("b-0")
            data class B_0(): BaseB()
            
            @Serializable
            @SerialName("b-1")
            data class B_1(): BaseB()
     }
}

Full exception

kotlinx.serialization.json.internal.JsonDecodingException: Polymorphic serializer was not found for class discriminator 'base-b'
JSON input: .....":"base-b","updated_at":"2023-06-01T11:01:13.457301Z"}
    at kotlinx.serialization.json.internal.JsonExceptionsKt.JsonDecodingException(JsonExceptions.kt:24)
    at kotlinx.serialization.json.internal.JsonExceptionsKt.JsonDecodingException(JsonExceptions.kt:32)
    at kotlinx.serialization.json.internal.PolymorphicKt.throwSerializerNotFound(Polymorphic.kt:79)
    at kotlinx.serialization.json.internal.PolymorphicKt.decodeSerializableValuePolymorphic(Polymorphic.kt:68)
    at kotlinx.serialization.json.internal.StreamingJsonDecoder.decodeSerializableValue(StreamingJsonDecoder.kt:81)
    at kotlinx.serialization.json.Json.decodeFromString(Json.kt:107)
    at com.tgsys.jaya.data.SimpleJayaRepository$mainCache$1.create(JModuleManager.kt:613)
    at com.tgsys.jaya.data.SimpleJayaRepository$mainCache$1.create(JModuleManager.kt:354)
    at android.util.LruCache.get(LruCache.java:135)
    at com.appmattus.layercache.LruCacheWrapper$get$2.invokeSuspend(LruCacheWrapper.kt:40)
    at kotlin.coroutines.jvm.internal.BaseContinuationImpl.resumeWith(ContinuationImpl.kt:33)
    at kotlinx.coroutines.DispatchedTask.run(DispatchedTask.kt:106)
    at kotlinx.coroutines.internal.LimitedDispatcher$Worker.run(LimitedDispatcher.kt:115)
    at kotlinx.coroutines.scheduling.TaskImpl.run(Tasks.kt:100)
    at kotlinx.coroutines.scheduling.CoroutineScheduler.runSafely(CoroutineScheduler.kt:584)
    at kotlinx.coroutines.scheduling.CoroutineScheduler$Worker.executeTask(CoroutineScheduler.kt:793)
    at kotlinx.coroutines.scheduling.CoroutineScheduler$Worker.runWorker(CoroutineScheduler.kt:697)
    at kotlinx.coroutines.scheduling.CoroutineScheduler$Worker.run(CoroutineScheduler.kt:684)
    Suppressed: kotlinx.serialization.json.internal.JsonDecodingException: Polymorphic serializer was not found for class discriminator 'base-b'

[Frontrider]

Add a SerialName annotation to your type (NOT any of the fields!), that does exactly what you are doing here.

For the code below, we are getting the kotlinx.serialization.json.internal.JsonDecodingException: Polymorphic serializer was not found for class discriminator 'base-b'

@Serializable
sealed class BaseA {
     
    @Serializable
    @SerialName("a-0")
    data class A_0(): BaseA()
    
    @Serializable
    @SerialName("a-1")
    data class A_1(): BaseA()
     
    @Serializable
    @SerialName("base-b")
    @JsonClassDiscriminator("b_type")
     sealed class BaseB {

            @Serializable
            @SerialName("b-0")
            data class B_0(): BaseB()
            
            @Serializable
            @SerialName("b-1")
            data class B_1(): BaseB()
     }
}

Full exception

kotlinx.serialization.json.internal.JsonDecodingException: Polymorphic serializer was not found for class discriminator 'base-b'
JSON input: .....":"base-b","updated_at":"2023-06-01T11:01:13.457301Z"}
    at kotlinx.serialization.json.internal.JsonExceptionsKt.JsonDecodingException(JsonExceptions.kt:24)
    at kotlinx.serialization.json.internal.JsonExceptionsKt.JsonDecodingException(JsonExceptions.kt:32)
    at kotlinx.serialization.json.internal.PolymorphicKt.throwSerializerNotFound(Polymorphic.kt:79)
    at kotlinx.serialization.json.internal.PolymorphicKt.decodeSerializableValuePolymorphic(Polymorphic.kt:68)
    at kotlinx.serialization.json.internal.StreamingJsonDecoder.decodeSerializableValue(StreamingJsonDecoder.kt:81)
    at kotlinx.serialization.json.Json.decodeFromString(Json.kt:107)
    at com.tgsys.jaya.data.SimpleJayaRepository$mainCache$1.create(JModuleManager.kt:613)
    at com.tgsys.jaya.data.SimpleJayaRepository$mainCache$1.create(JModuleManager.kt:354)
    at android.util.LruCache.get(LruCache.java:135)
    at com.appmattus.layercache.LruCacheWrapper$get$2.invokeSuspend(LruCacheWrapper.kt:40)
    at kotlin.coroutines.jvm.internal.BaseContinuationImpl.resumeWith(ContinuationImpl.kt:33)
    at kotlinx.coroutines.DispatchedTask.run(DispatchedTask.kt:106)
    at kotlinx.coroutines.internal.LimitedDispatcher$Worker.run(LimitedDispatcher.kt:115)
    at kotlinx.coroutines.scheduling.TaskImpl.run(Tasks.kt:100)
    at kotlinx.coroutines.scheduling.CoroutineScheduler.runSafely(CoroutineScheduler.kt:584)
    at kotlinx.coroutines.scheduling.CoroutineScheduler$Worker.executeTask(CoroutineScheduler.kt:793)
    at kotlinx.coroutines.scheduling.CoroutineScheduler$Worker.runWorker(CoroutineScheduler.kt:697)
    at kotlinx.coroutines.scheduling.CoroutineScheduler$Worker.run(CoroutineScheduler.kt:684)
    Suppressed: kotlinx.serialization.json.internal.JsonDecodingException: Polymorphic serializer was not found for class discriminator 'base-b'

And the json? I literally just had my tests pass with code like that as I'm writing this. Also I'm on kotlin version 1.8.10, with kotlin serialization json version 1.5.0.

[pramod-knidal]

And the json? I literally just had my tests pass with code like that as I'm writing this. Also I'm on kotlin version 1.8.10, with kotlin serialization json version 1.5.0.

1.8.21 and 1.5.1

Can you share a link to your tests? I'll check your json once.

which json do you need? json for one of the subclasses of BaseB is sufficient?

[pramod-knidal]

Here is the json for B_0

{
  "b_type": "b-0",
  "created_at": "2023-06-01T12:27:59.345234Z",
  "id": "e19c5a4f-4ebf-4b3f-a4d7-e3e636797bbf",
  "module_id": "com.tgsys.tabjoy",
  "text": "is not held responsible for any changes that are made to the unit nor accepts any liabilities for these changes.",
  "type": "base-b",
  "updated_at": "2023-06-01T12:27:59.345235Z"
}

[pramod-knidal]

@Frontrider: please also let me know if you are registering any Serializers explicitly during Json instance construction. I am thinking that you might be adding some serializers for classes B_0,B_1.

[Frontrider]

{
  "b_type": "b-0",
  "created_at": "2023-06-01T12:27:59.345234Z",
  "id": "e19c5a4f-4ebf-4b3f-a4d7-e3e636797bbf",
  "module_id": "com.tgsys.tabjoy",
  "text": "is not held responsible for any changes that are made to the unit nor accepts any liabilities for these changes.",
  "type": "b-0",
  "updated_at": "2023-06-01T12:27:59.345235Z"
}

I started having these types of issues with it. https://github.com/Kotlin/kotlinx.serialization/issues/1797 https://stackoverflow.com/questions/72487782/jsonclassdiscriminator-doesnt-change-json-class-discriminator

This is my most secure solution, I do not think I can answer with anything else at this point:

internal val json = Json {
   //we need this so the parser can ignore the type discriminator on an as-needed basis.
    ignoreUnknownKeys = true
   //set the discriminator globally for consistency, setting it for a subtype only increases complexity with no gain whatsoever
    classDiscriminator = "b_type"
}

The @JsonTypeDiscriminator is marked as an unstable api, so this specific case with multiple type discriminators is most likely a separate bug report. (the original case just works on my side, by setting it globally)

Check if you actually need that field to be different (why it can't just be the default type, that may fix the issues with experimental stuff), and note that in your json "b_type": "b-0" is the real discriminator, "type": "b-0", is a 100% invalid field (according to whatever documentation I see) that at best gives you an unknown field error, or may be the reason why it can't figure out which child to go for.

@Frontrider: please also let me know if you are registering any Serializers explicitly during Json instance construction. I am thinking that you might be adding some serializers for classes B_0,B_1.

In the tree where I use discriminators, I do not have any special serializers, only a customized json parser seen above. One may get in later, but other data classes use so many that I do not think it can cause any problems.

[pramod-knidal]

Ummm. So, in our model, we need both type field and b-type field. We use a NoSQL database where the type field gives us the information about the top level document-type (that is used for more than just class discrimination) and there is another field b_type which gives us sub-document type. In theory, it is possible to have a flat data model where type field is used to resolve (somehow I prefer the term resolve over discriminator) the type of the document & sub-document type but it is just not the right way to model it because of the nature of the processing each documents would have to go through.

And... another thing I just noticed is that, if i just use b_type as the classDiscriminator by using @JsonClassDiscriminator('b_type'), I still get an error because type field exists in the model (for purposes other than class discrimination). So, it is not just about having b_type as classDiscriminator, it is about making sure there is no field by the name type.

[Frontrider]

And... another thing I just noticed is that, if i just use b_type as the classDiscriminator by using @JsonClassDiscriminator('b_type'), I still get an error because type field exists in the model (for purposes other than class discrimination). So, it is not just about having b_type as classDiscriminator, it is about making sure there is no field by the name type.

Exactly, that is why I wrote that you need to set ignoreUnknownKeys to true, so it ignores the unused field. That is as far as I can go with it.

[pramod-knidal]

Just to leave note for anyone's reference, we implemented a custom deserialization using Content-based polymorphic deserialization.

[vendelieu]

isn't there any update?

[vendelieu]

This is useful for me as I'm using the discriminator field to generate Typescript classes (using https://github.com/ntrrgc/ts-generator).

I've found a couple of potential work-arounds that I'm experimenting with:

import kotlin.reflect.full.findAnnotation
import kotlin.reflect.full.starProjectedType
import kotlinx.serialization.SerialName
import kotlinx.serialization.Serializable
import kotlinx.serialization.json.JsonClassDiscriminator


@Serializable
@JsonClassDiscriminator("object_name")
sealed class EventData() {

  //  get the discriminator as a field (the field name, objectName, is unimportant)
  // this must be a delegated field so there's no backing field, so kxs ignores it
  val objectName: String by lazy {
    // this probably won't work for generic classes?
    kotlinx.serialization.serializer(this::class.starProjectedType).descriptor.serialName
  }

  // alternative: find the @SerialName annotation.
  // Again, the field must be delegated so there's no backing field. 
  // This requires that every subclass has @SerialName
  val objectNameByAnnotation: String by lazy {
    requireNotNull(this::class.findAnnotation<SerialName>()?.value) {
      "Couldn't find @SerialName for ${this::class}!"
    }
  }
}

Thanks for sharing workarounds, I found more simplier for first case, might be helpful for someone:

    val objName: String by lazy {
        this::class.serializer().descriptor.serialName
    }

since kotlin-serialization anyway generates companion object with serializer() which is we tried getting from kotlinx.serialization.serializer(this::class.starProjectedType) we can omit this step and address it directly

UPD: Also make sure you're using sealed class serializer not its member since it seems like members don't know about discriminator (Good point to make them know about it when planning corresponding feature)

[sandwwraith]

I'm still a little bit puzzled by this feature request. I understand that sometimes you need this:

@Serializable
sealed class Base {
    abstract val type: String
}

@Serializable
@SerialName("BaseImpl")
data class BaseImpl(override val type: String, val value: String): Base()

val input = """{"value":"foo","type":"BaseImpl"}"""
println(Json.decodeFromString<Base>(input)) // BaseImpl(type=BaseImpl, value=foo)

(this already works btw, because conflict check for sealed classes is performed only on encoding).

However, I do not know what is expected on encoding. What if I create BaseImpl("Unrelated", "foo")? Do you expect {"value":"foo","type":"Unrelated"} in the output? In that case it is impossible to decode such data back. Should it be {"value":"foo","type":"BaseImpl"}? Then we overwrite "Unrelated" data.

The same goes for cases with @JsonNames or JsonNamingStrategy, as in this example:

@Serializable
sealed class Base {
  @JsonNames("Type")
  var type: String = ""
}

Json {
  classDiscriminator = "Type"
}

In this case, we can accidentally overwrite data on decoding as well, if Type is present and type is missing (see #2885 for example)

[sandwwraith]

In the case of namingStrategy, it even produces a duplicate key:

@Serializable
sealed class Value {
    var status = "open"
}
val json = Json(default) {
    classDiscriminator = "STATUS"
    encodeDefaults = true 
    namingStrategy = JsonNamingStrategy { _, _, name -> name.uppercase() }
}

val value: Value = ValueImpl("foo")
val str = json.encodeToString(value)
// {"STATUS":"ValueImpl","STATUS":"open","VALUE":"foo"} - ???

[sandwwraith]

I've merged #3105 that allows using type or other identical to discriminator key on deserialization only. With this PR, you can either have val type for hierarchies you're only decoding, or you can have @JsonNames("type") as an alternative deserialization name for one of your properties (be aware that discriminator value may overwrite your defaults, if you're not encoding them) I think this settles this issue for now.