Creating serial descriptor when serializing to heterogeneous lists

[odzhychko]

What is your use-case and why do you need this feature?

I want to serialize a class as a JSON array. The elements can have different types.

For that, I created a custom serializer. When building the corresponding SerialDescriptor I need to use the internal buildSerialDescriptor builder because the public buildClassSerialDescriptor does not cover this use-case.

The following shows a simplified example:

package dev.oleks

import kotlinx.serialization.InternalSerializationApi
import kotlinx.serialization.KSerializer
import kotlinx.serialization.Serializable
import kotlinx.serialization.descriptors.SerialDescriptor
import kotlinx.serialization.descriptors.StructureKind
import kotlinx.serialization.descriptors.buildClassSerialDescriptor
import kotlinx.serialization.descriptors.buildSerialDescriptor
import kotlinx.serialization.encoding.CompositeDecoder
import kotlinx.serialization.encoding.Decoder
import kotlinx.serialization.encoding.Encoder
import kotlinx.serialization.encoding.decodeStructure
import kotlinx.serialization.encoding.encodeCollection
import kotlinx.serialization.json.Json

fun main() {
    val encoded = Json.encodeToString(ListOfThreeElements(1, "aValue1", SomeClass("aValue2")))
    println(encoded) // [1,"aValue1",{"someAttribute":"aValue2"}]
    val decoded: ListOfThreeElements<Int, String, SomeClass> = Json.decodeFromString(encoded)
    println(decoded) // ListOfThreeElements(value1=1, value2=aValue1, value3=SomeClass(someAttribute=aValue2))
}

@Serializable
data class SomeClass(val someAttribute: String)

@Serializable(with = ListOfThreeElementsSerializer::class)
data class ListOfThreeElements<T1 : Any, T2 : Any, T3 : Any>(
    val value1: T1,
    val value2: T2,
    val value3: T3,
)

class ListOfThreeElementsSerializer<T1 : Any, T2 : Any, T3 : Any>(
    private val t1Serializer: KSerializer<T1>,
    private val t2Serializer: KSerializer<T2>,
    private val t3Serializer: KSerializer<T3>,
) : KSerializer<ListOfThreeElements<T1, T2, T3>> {
    override fun deserialize(decoder: Decoder): ListOfThreeElements<T1, T2, T3> {
        var value1: T1? = null
        var value2: T2? = null
        var value3: T3? = null
        decoder.decodeStructure(descriptor) {
            while (true) {
                when (val index = decodeElementIndex(descriptor)) {
                    0 -> value1 = decodeSerializableElement(descriptor, index, t1Serializer)
                    1 -> value2 = decodeSerializableElement(descriptor, index, t2Serializer)
                    2 -> value3 = decodeSerializableElement(descriptor, index, t3Serializer)
                    CompositeDecoder.DECODE_DONE -> break // Input is over
                    else -> error("Unexpected index: $index")
                }
            }
        }
        require(value1 != null && value2 != null && value3 != null)
        return ListOfThreeElements(value1, value2, value3)
    }

    override val descriptor: SerialDescriptor = run {
        val typeParameters = arrayOf(t1Serializer.descriptor, t2Serializer.descriptor, t3Serializer.descriptor)
        @OptIn(InternalSerializationApi::class) (buildSerialDescriptor (
        "dev.oleks.ListOfThreeElements", StructureKind.LIST, *typeParameters
    ) {
        element("0", t1Serializer.descriptor)
        element("1", t1Serializer.descriptor)
        element("2", t1Serializer.descriptor)
    })
    }

    override fun serialize(encoder: Encoder, value: ListOfThreeElements<T1, T2, T3>) {
        encoder.encodeCollection(descriptor, 3) {
            encodeSerializableElement(t1Serializer.descriptor, 0, t1Serializer, value.value1)
            encodeSerializableElement(t2Serializer.descriptor, 1, t2Serializer, value.value2)
            encodeSerializableElement(t3Serializer.descriptor, 2, t3Serializer, value.value3)
        }
    }
}

Describe the solution you'd like

• A function like listSerialDescriptor but for heterogeneous lists,
• making non-internalbuildSerialDescriptor or
• a buildListSerialDescriptor builder similar to buildClassSerialDescriptor and buildSerialDescriptor.

[sandwwraith]

Hm, why do you need exactly LIST and it can't be just some kind of Tuple3 class?

[odzhychko]

Hm, why do you need exactly LIST and it can't be just some kind of Tuple3 class?

I'm not sure, I understood your question correctly.

I want to serialize a ListOfThreeElements to a JSON array (e.g., [1,"aValue1",{"someAttribute":"aValue2"}]) instead of a JSON object (e.g., {"value1":1,"value2":"aValue1", "value3" : { "someAttribute" : "aValue2"}} because that is the data format of the API we decided on in our project. I guess initially, we wanted to achieve smaller responses by avoiding keys.

To serialize/deserialize a JSON array, the SerialDescriptor.kind needs to be set to StructureKind.LIST. Is this a wrong understanding of how Kotlin serialization works? Can an object be serialized to a JSON array when the kind is set to StructureKind.CLASS?

[odzhychko]

I just noticed that I made a mistake in the example. I changed the following in the example just now:

58,59c60,61
<         @OptIn(InternalSerializationApi::class) (buildClassSerialDescriptor (
<         "dev.oleks.ListOfThreeElements", *typeParameters
---
>         @OptIn(InternalSerializationApi::class) (buildSerialDescriptor (
>         "dev.oleks.ListOfThreeElements", StructureKind.LIST, *typeParameters

[sandwwraith]

Thanks for the clarifications. Yes, if you want [ ] brackets in the output, then using StructureKind.LIST is the correct approach. We currently do not support heterogeneous lists natively, so opting-in into internal API is the only way. We'll add this use-case to our list when we'll be designing buildSerialDescriptor for public use.

[pdvrieze]

@odzhychko Note that (in the case of Json) you could use a List<JsonElement> (or even bare JsonElement) as the effective type. It would require you to have the custom serializer/deserializer do the translation to/from JsonElement.

[odzhychko]

@pdvrieze Thanks for the suggestion, but I do not understand how to apply it. Could you perhaps elaborate further?

I found the documentation for JsonTransformingSerializer and the KDoc for StructureKind.LIST. The KDoc mentions a similar case, but I didn't understand what the corresponding code would look like.

[pdvrieze]

@odzhychko For efficiency, there is actually a much simpler solution. Just replace your descriptor with the following:

override val descriptor = ListSerializer(ContextualSerializer(Any::class)).descriptor

This works correctly for the Json format (even when providing the actual descriptors to encodeSerializableElement). If you want it more robust you could use per-element contextual serializers with different fallback serializers (to the proper ones), but for json it is not needed.

The reason to go for Contextual is that this is a special type of serializerKind that indicates that it is resolved at runtime (and can not be cached).

[odzhychko]

That worked. Thanks for the explanation.