Cannot create a custom CONTEXTUAL serializer without use of InternalSerializationApi

[bartvanheukelom]

I've written a custom serializer which takes Any input and - basically by magic - finds an appropriate KSerializer to then delegate the actual serialization to.

It seemed to me that the most appropriate SerialKind to give to this serializer is CONTEXTUAL, given its description of:

Represents an "unknown" type that will be known only at the moment of the serialization. Effectively it defers the choice of the serializer to a moment of the serialization

In fact, that may have been required to make it work at all. Don't remember any more. In any case, I found no way to do that except by using a function tagged with InternalSerializationApi.

Admittedly, my use case is a bit esoteric, and only done for legacy reasons, but I'm reporting it anyway, since you so explicitly requested that in the InternalSerializationApi KDoc.

Possible improvements could be:

• the ability to declare a CONTEXTUAL SerialKind without having to resort to internal API
• a new SerialKind specifically for use cases like this

Here is the full code for the serializer in question:

/**
 * Yucky serializer for responses whose generic type is not properly known. These include:
 * - legacy RPC responses (partly fixable because return type is known somewhere, but also often something like `Map<String, *>`)
 * - [com.example.ExceptionSerialized.extra]
 *
 * Note that it only supports serialization, not deserialization, but it implements [KSerializer] anyway,
 * because that's what classes like [SerialModule] communicate with.
 */
object RuntimeResponseSerializer : KSerializer<Any> {
  
    override val descriptor =
        @OptIn(InternalSerializationApi::class, ExperimentalSerializationApi::class)
        buildSerialDescriptor(kClass.qualifiedName!!, SerialKind.CONTEXTUAL)

    val nullable = (this as KSerializer<Any>).nullable

    @Suppress("UNCHECKED_CAST")
    fun <T> cast(): KSerializer<T> =
            this as KSerializer<T>

    override fun deserialize(decoder: Decoder): Any =
            throw UnsupportedOperationException("This is the Runtime_RESPONSE_Serializer, see?")

    override fun serialize(encoder: Encoder, value: Any) {
        findRuntimeSerializer(encoder.serializersModule, value)
            .serialize(encoder, value)
    }

    @OptIn(InternalSerializationApi::class, ExperimentalSerializationApi::class)
    @Suppress("UNCHECKED_CAST")
    fun <T : Any> findRuntimeSerializer(context: SerializersModule, v: T): SerializationStrategy<T> =
            try {

                // check some types first because serializerOrNull throws for them
                when {
                    v is Map<*, *> && v.isEmpty() ->
                        MapSerializer(String.serializer(), Unit.serializer()) as SerializationStrategy<T>
                    else -> null
                }

                    ?: v.kClass.takeIf { it.typeParameters.isEmpty() }
                        ?.serializerOrNull()

                    ?: context.getContextual(v.kClass)
                        ?.also {
                            // prevent stack overflow when RRS is used as contextual fallback
                            if (it as KSerializer<*> is RuntimeResponseSerializer) {
                                throw SerializationException("getContextualOrDefault returned RuntimeResponseSerializer")
                            }
                        }

                    ?: when (v) {

                        is TableResult<*> -> TableResult.serializer(nullable)
                        is Map<*, *> -> {
                            val keySer: KSerializer<*> =
                                    if (v.keys.all { it is String }) String.serializer()
                                    else RuntimeResponseSerializer
                            MapSerializer(keySer, nullable)
                        }
                        is List<*> -> {
                            ListSerializer(nullable)
                        }
                        is Set<*> -> SetSerializer(nullable)
                        is ByteArray -> Base64BasicSerializer
                        is Date -> DateSerializer
                        is Instant -> InstantSerializer

                        else -> throw IllegalArgumentException("No compiled, default, contextual or magic serializer exists")

                    } as SerializationStrategy<T>

            } catch (e: Throwable) {
                throw IllegalArgumentException("Could not find a runtime serializer for type ${v.javaClass.kotlin} with value ${v.toString().truncated(16, "...")}: $e", e)
            }
}

[pdvrieze]

When implementing a contextual serializer there are some things to take into account. One, a contextual serializer is not expected to be polymorphic (so the wrong choice here - polymorphic would be). It has to expose the class to be (de)serialized in the context of the descriptor (this uses internal withContext) so you have to create a contextual serializer and then steal the descriptor.

As said, in this case you should be creating a polymorphic serializer. Again, the implementation requires knowledge of the internals and may change in new versions. An important thing to realise with both is that it important that a format can actually inspect the formats to determine the actual serialization to be used.

[bartvanheukelom]

Thanks for the correction. Polymorphic sounds more applicable indeed, I'll see if I can make it one of those. For some reason I was confusing Contextual's feature of looking up a serializer for a known type in the context with Polymorphic's feature of determining the type of a value at runtime.

That said, while it was not the source of my confusion, it seems to me that the statement

Represents an "unknown" type that will be known only at the moment of the serialization.

could very well have done so hypothetically.

An important thing to realise with both is that it important that a format can actually inspect the formats to determine the actual serialization to be used.

Hmm, I don't quite get this. Could you elaborate? Did you mean to write something else instead of "formats"?

[sandwwraith]

Even if Polymorphic kind would be selected instead of Contextual, Internal API still would be used due to buildSerialDescriptor.

[bartvanheukelom]

Looking into it some more, I'm starting to feel like Contextual is the right kind after all. If we look at the descriptor of PolymorphicSerializer, for example:

buildSerialDescriptor("kotlinx.serialization.Polymorphic", PolymorphicKind.OPEN) {
            element("type", String.serializer().descriptor)
            element(
                "value",
                buildSerialDescriptor("kotlinx.serialization.Polymorphic<${baseClass.simpleName}>", SerialKind.CONTEXTUAL)
            )
            annotations = _annotations
        }.withContext(baseClass)

We can see that it statically knows a bit about the serialization format, namely that it includes a type and a value, but that it doesn't know the format of the value, and it describes that as being of a Contextual kind.
This bit is actually what's most equivalent to my custom serializer, since that directly encodes the values it's given, without encoding any type info (that's why it's serialize-only). Like the value in PolymorphicSerializer, it knows absolutely nothing of its format without the having the actual value present.

[pdvrieze]

@bartvanheukelom: Hmm, I don't quite get this. Could you elaborate? Did you mean to write something else instead of "formats"? Kotlin serialization involves 3 components:

• Format: (encoder/decoder and infra). This allow to serialize to a specific format such as
• Serializer: Each type to be serialized has one (or multiple) associated serializers. These serializers provide the knowledge of the type and can read/write instances of those types.
• The library that provides shared api and overall infrastructure.

As to the implementation of the serializer you propose for contextual. You should call encoder.encodeSerializableValue. This provides the format with access to the serializer (and descriptor), so that it can actually handle the delegated type (one of the reasons this is not public api).

[jvgelder]

Not sure if it is entirely related to what was mentioned above. Since my case is rather specific as I wanted to build a serializer for a map that had mixed types and key names that unfortunately up front i cannot predict.

I was trying to extend the internal map serializer but cant because its protected or marked as internal.

I also tried to use the annotation below but since it expects a KClass and not an instance this also wont work.

@Serializable(with=MapSerializer(String.serializer(), MyAnySerializer()))

Update:

The solution for me was the JsonObject