Keep access to plugin generated serializer when mark class with @Serializable(with = MySerializer)

[ikarenkov]

What is your use-case and why do you need this feature?

When we write custom serializer for class, which contains plugin generated serializer inside, we cannot use with parameter of @Serializable because on this case, plugin serializer will not be generated.

In this case we have to manually use custom serializer in all case when we want to use our class serialization, because we cannot use @Serializable(with = MySerializer) instead.

In my case I'm trying to deserialize enum with generated serializer and if I get exception I catch it and return default value. It also can be used for some data transformations during decoding or encoding.

Describe the solution you'd like

It would be nice to have ability to generate and use plugin serializer when use class annotation @Serializable(with = MySerializer). For example, we can add extra parameter generateSerializer for @Serializable or use one more annotation to let plugin know, that we still need serializer. Generated serializer can have another name, for example generatedSerializer().

[bartvanheukelom]

Supposedly, with @Serializer you can generate a separate serializer for any class. Maybe you could use that?

[dimsuz]

I'd like this to be implemented too. I will describe my use case.

I want to deserialize json:

{
   "entryAction": "actionId",
   "customerRefs": "ref1",
   "allRefs": [ "ref1" ]
}

But in some cases it could be an object (don't ask why!) with additional optional fields:

{
   "entryAction": { "id": "actionId", "type": "basic" },
   "customerRefs": { "id": "ref1", "expired": false },
   "allRefs": [ { "id": "ref1", "expired": false } ]
}

What I'd like to do is this:

@Serializable(with = ActionSerializer::class)
data class Action(val id: String, val type: String?)

@Serializable(with = RefSerializer::class)
data class Ref(val id: String, val expired: Boolean?)

class ActionSerializer : KSerializer<Action> {
  fun deserialize(decoder): Action {
    return try { 
      Action(decoder.decodeString())
    } catch (...) {
      Action.DEFAULT_SERIALIZER.deserialize(decoder) // <-- (!!)
    }
  }

// similar implementation for RefSerializer

Notice that in this approach I automatically get deserialization of allRefs property — both in "array of strings" and "array of object cases". But nothing like DEFAULT_SERIALIZER available.

I.e. for each value type ("action" and "ref" above) the cases I have:

• "key": "value"
• "key": { "id": "value", "optionalFields" ... }
• "key": [ "value1", "value2" ]
• "key": [ { "id": "value1", "optionalFields" ... }, { "id": "value1", "optionalFields" ... } ]

If above serializer was possible I would write one serializer per object type and get all 4 cases covered. Currently I have to have:

• Serializer for "single" case where only one value possible in deserialized object: val key: Something
• JsonTransformingSerializer where deserialized object accepts val key: List<Something>

When there are a lot of such props in json it gets tedious having to write similar pairs of serializers for each value type, moreover they share some bits of implementation which makes it error prone to maintain...

[dimsuz]

Oh, wait. @Serializer seems to fit this usecase!

I hope it will make it out of experimental state then :slightly_smiling_face:

[Ayfri]

Any update now ?

[sandwwraith]

@Ayfri No updates, but given the demand for this feature, it's definitely on our table

[ultranity]

temp workaround here: use typealias ref: https://github.com/Kotlin/kotlinx.serialization/issues/840#issuecomment-1674907335

[sschuberth]

@dimsuz, in this scenario

class ActionSerializer : KSerializer<Action> {
  fun deserialize(decoder): Action {
    return try { 
      Action(decoder.decodeString())
    } catch (...) {
      Action.DEFAULT_SERIALIZER.deserialize(decoder) // <-- (!!)
    }
  }

how do you deal with the fact that the first call to decoder.decodeString() already advanced in the input, so the fallback to Action.DEFAULT_SERIALIZER.deserialize(decoder) does not see the full input anymore?

[EarthyOrange]

Hello all, @shanshin, @sandwwraith !

I ran in to the following use case. Please let me know if it fits within this issue or if there already is a solution for it.

JSON Response from a GraphQL endpoint:

{
  "data": {
    "level1": {
      "level2": {
        "level3": {
          "level4": {
            "a": { "a-key-1": "a-value-1", "a-key-2": 250 },
            "b": { "b-key-1": 5.00, "b-key-2": 50.00 }
          }
        }
      }
    }
  }
}

DTO class structure:

@Serializable
data class ResponseDto(val data: T?, val error: Error) // We'll skip Error class definition for brevity

@Serializable(with = Level4Deserializer::class)
data class Level4(val a: A, val b: B) {
  @Serializable data class A(...)
  @Serializable data class B(...)
}

I don't want to write the data classes for level1 to level3 because I am lazy & I think it's a waste of effort. I want to be able to tell the serializer to just ignore the intermediate levels and look at "level4" structure directly. To do that I wrote down Level4Deserializer as follows:

object Level4Deserializer : JsonTransformingSerializer(serializer()) {
  override fun transformDeserialize(element: JsonElement): JsonElement {
        // deserializing the "data" level is done by ResponseDto's deserializer
        return element
            .jsonObject
            .getJsonObject("level1")
            .getJsonObject("level2")
            .getJsonObject("level3")
            .getJsonObject("level4")
    }

    private fun JsonObject.getJsonObject(key: String): JsonObject {
        return if (!this.containsKey(key)) {
            throw Exception("This JsonObject doesn't have any element with the name: $key")
        } else {
            this.getValue(key).jsonObject
        }
    }
}

When this runs the serializer() in class Level4Deserializer : JsonTransformingSerializer<Level4>(serializer()) { throws a null pointer exception (not exactly: the framework invokes the plugin generated serializer method on Level4's companion object which returns null and results in an exception). I think the related issue is https://github.com/Kotlin/kotlinx.serialization/issues/2348 and this current issue also talks about this default serializer being null when the top level data class is annotated with the Serializable(with = KSerializerImplementation::class. Am I understanding it right?

So to overcome this I used https://github.com/Kotlin/kotlinx.serialization/issues/237 as reference to write down the serializer implementation which the plugin should have generated and passed its instance to the JsonTransformingSerializer's constructor. Which works but I am not sure if this is correct; I feel that I shouldn't need to write the "default" serializer for Level4 by hand and it shouldn't be null.

// I shouldn't need to write this
object Level4Serializer : KSerializer {
  descriptor value definition
  deserializer method definition
  blank serializer method // Level4 is only received and never sent.
}

object Level4Deserializer : JsonTransformingSerializer(Level4Serializer) {
...
}

Please let me know if I am missing anything here.

[pdvrieze]

object Level4Deserializer : JsonTransformingSerializer(Level4Serializer) { ... } Please let me know if I am missing anything here.

It is the same problem. Note that the code quoted here can never work because it would have an initialisation loop (and even if that wasn't the issue, it would still not be correct).

The workaround is to create a separate object annotated with @Serializer(forClass=Level4::class) and use that as argument. Note that you also need to adjust the serializer for T (data) to actually handle this.

[EarthyOrange]

Note that the code quoted here can never work

It is working.

even if that wasn't the issue, it would still not be correct

What isn't correct & why?

Note that you also need to adjust the serializer for T (data) to actually handle this.

I don't need to because the Level4 class has been instructed to use the Level4Deserializer.

[sandwwraith]

Expected to be available in Kotlin 2.0.20.

[BenWoodworth]

@shanshin Thanks for implementing this, it looks good! Very simple and straightforward to use with the way it's designed 😊

One quick question / mild concern, would it make sense for the generatedSerializer() function that's created to be internal, instead of public like it is now? I think it would be an implementation detail in many cases, and with library serializers it's not something I'd want exposed in the API. (And if someone does want it exposed, it'd be easy enough to add a function that returns it)

[sandwwraith]

That's an interesting idea. We generally rely on the fact that the serializer has the same visibility as the class, and that plugin can access serializer anywhere where the class is accessible. Yet, generatedSerializer is not accessed by plugin, so this change is possible in theory.

[BenWoodworth]

@sandwwraith It'd be great to see this change implemented! Assuming there really aren't any issues with the generated serializer being internal.

Any chance there's been further thought put towards it, or plans/updates otherwise?

[sandwwraith]

@shanshin PTAL

[abrooksv]

@sandwwraith, what was the reasoning for marking @KeepGeneratedSerializer as an internal API?

Am I overlooking something that lets us enable the feature without using the internal annotation?

Right now it causes me to have to opt-in to it in a lot of places and is raising eyebrows in PRs to order to achieve the following pattern:

@file:OptIn(InternalSerializationApi::class)

object FooSerializer : KSerializer<Foo> by EnumWithFallbackSerializer(Foo.generatedSerializer(), Foo.UNKNOWN)

@Serializable(with = FooSerializer::class)
@KeepGeneratedSerializer
enum class Foo {
...
}

[sandwwraith]

@abrooksv See https://youtrack.jetbrains.com/issue/KT-68519/KotlinX-Serialization-KeepGeneratedSerializer#focus=Comments-27-10079353.0-0 :

@KeepGeneratedSerializer is intended to be a public API (see https://github.com/Kotlin/kotlinx.serialization/issues/1169). The reason for it being internal now is that it doesn't work in 2.0 (as you've noticed) — changes in the plugin required for this feature to work are implemented in the 2.0.20 branch, so when Kotlin 2.0.20 is released, we'll make this API public.