Reified types are not inlined into type references of generic classes

[meztihn]

When you create a type reference for a generic class with a reified type parameter, this type parameter is still generic, not the inlined type. That prevents deserialization of properties of the reified type. It might look like very rare case unless you prefer composition over inheritance. Looks like a new typeOf Kotlin function works correctly in this case. Can we use it somehow?

Kotlin version: 1.3.41 Jackson version: 2.9.9

val objectMapper = jacksonObjectMapper()

fun main() {
    val json = """{ "wrapped": { "content": 42 } }"""
    println("objectMapper.readValue<Wrap<Box>>(json): ${objectMapper.readValue<Wrap<Box>>(json)}")

    println("jacksonTypeRef<Wrap<Box>>(): ${jacksonTypeRef<Wrap<Box>>().type}")

    val readWrapped = readWrapped<Box>(json)
    println("readWrapped<Box>(json): $readWrapped")
    println("readWrapped<Box>(json).wrapped::class: ${readWrapped.wrapped::class}")
}

@UseExperimental(ExperimentalStdlibApi::class)
private inline fun <reified R> readWrapped(json: String): Wrap<R> {
    println("jacksonTypeRef<R>().type: ${jacksonTypeRef<R>().type}")
    println("jacksonTypeRef<Wrap<R>>().type: ${jacksonTypeRef<Wrap<R>>().type}")
    println("typeOf<Wrap<R>>(): ${typeOf<Wrap<R>>()}")
    return objectMapper.readValue(json)
}

data class Wrap<V>(val wrapped: V)

data class Box(val content: Int)

Output:

objectMapper.readValue<Wrap<Box>>(json): Wrap(wrapped=Box(content=42))
jacksonTypeRef<Wrap<Box>>(): Wrap<Box>
jacksonTypeRef<R>().type: class Box
jacksonTypeRef<Wrap<R>>().type: Wrap<R>
typeOf<Wrap<R>>(): Wrap<Box>
readWrapped<Box>(json): Wrap(wrapped={content=42})
readWrapped<Box>(json).wrapped::class: class java.util.LinkedHashMap

[apatrida]

Hi @Meztihn Your output is shown but doesn't indicate which thing you consider incorrect, is it the jacksonTypeRef<Wrap<R>>().type: Wrap<R> that you expect to be Wrap<Box> ?

Writing a test case is helpful because that shortens the amount of work to add a fix.

[apatrida]

@Meztihn @cowtowncoder

Type typeOf doesn't help because it creates a KType that cannot be converted in any way to a Java Type

This issue is more clearly stated as, the following two methods of creating a TypeReference return different results:

private inline fun <reified R> subType(): TypeReference<Wrap<R>> {
    return jacksonTypeRef()
}

val typeExplicit = jacksonTypeRef<Wrap<Box>>().type
val typeRefieid =  subType<Box>().type

println("explicit = $typeExplicit") // com.fasterxml.jackson.module.kotlin.test.Wrap<com.fasterxml.jackson.module.kotlin.test.Box>
println("reified  = $typeRefieid") // com.fasterxml.jackson.module.kotlin.test.Wrap<R>

And this is a problem in some use cases.

[apatrida]

Converting typeOf<Wrap<R>>().javaType results in exception:

Exception in thread "main" kotlin.NotImplementedError: An operation is not implemented: Java type is not yet supported for types created with createType

Also this is experimental API and cannot be used in a commonly used library.

[apatrida]

@Meztihn what would help here is if you report in the Kotlin Slack channel something about this, or in the YouTrack that you cannot convert the KType to a Java type even when it might be possible and compatible. See if they come up with any ideas.

@cowtowncoder other than you letting me change parts of databind into Kotlin so that we have a richer super set of Java with more type information, the way to fix this I think would be to have a KType to JavaType converter but I don't really want to use this experimental API typeOf() yet anyways. So this needs to wait for Kotlin to make that standard, and us to be sure people will upgrade to that version of Kotlin, otherwise we aren't backwards compatible.

[cowtowncoder]

@apatrida TypeFactory would be place to do it; and there is already possibility of TypeModifier that allows some level of tweaking (Scala module uses it to infer "Map-like" and "Collection-like" types as Scala collections do not extend java.util.Map / java.util.Collection, but behave very much like those). I am actually not opposed to adding new JavaType subtypes if there is a good conceptual model (I am considering one new type for 3.0 for "Iterable" things) of what extension is needed.

But it may be early to do that.

Other than that, convertor could perhaps construct JavaType programmatically, if building blocks themselves are sufficient. It does allow constructing Java's generic types at least, to the degree Jackson itself needs (which is obviously a small subset of all information types, even in just Java, have).

[LukasForst]

I came across to the issue which is probably related to this one

import com.fasterxml.jackson.module.kotlin.jacksonObjectMapper
import com.fasterxml.jackson.module.kotlin.readValue
import kotlin.test.Test
import kotlin.test.assertEquals
import kotlin.test.assertFailsWith
import kotlin.test.assertTrue

internal class ParsingIssueTest {

    private data class H(val h: String)

    private fun H.toJson() = "[{\"h\": \"$h\"}]"

    private inline fun <reified T> parseJson(json: String) =
        jacksonObjectMapper().readValue<T>(json)

    @Test
    fun `working example`() {
        val expected = H("hello world")
        val actual = parseJson<List<H>>(expected.toJson())

        assertEquals(expected.h, actual.single().h)
    }

    private inline fun <reified T> parseListJson(json: String) =
        jacksonObjectMapper().readValue<List<T>>(json)

    @Test
    fun `failing on class cast exception`() {
        val expected = H("hello again!")
        val actual = parseListJson<H>(expected.toJson())

        val ex = assertFailsWith<ClassCastException> {
            actual.single().h
        }
        assertTrue {
            ex.message?.contains("java.util.LinkedHashMap cannot be cast") ?: false
        }
    }
}

[tomashanley-toast]

Hi,

I think I'm also facing this issue, which is very limiting in terms of what i need to do.

Here's a test case that shows the read value method working normally, and then not working when using a reified type parameter.

As far as I'm aware there should be no type erasure when using a reified inline function, so I don't know how this is happening. Printing the generic type to the console looks correct. Any help or guidance appreciated!

Kotlin version; 1.4.10 Jackson Kotlin Module : 2.9.10

import com.fasterxml.jackson.annotation.JsonProperty
import com.fasterxml.jackson.core.type.TypeReference
import com.fasterxml.jackson.databind.ObjectMapper
import com.fasterxml.jackson.module.kotlin.jacksonObjectMapper
import com.fasterxml.jackson.module.kotlin.readValue
import io.kotest.matchers.shouldBe
import org.junit.jupiter.api.Test

data class Wrapper<T: MarkerInterface>(
    @JsonProperty("value") val value: T
)

interface MarkerInterface

data class InnerType(val value : Int): MarkerInterface


class JacksonIssueTest {

    private val wrapper = Wrapper(InnerType(5))

    private val json = """{"value":{"value":5}}"""

    private val mapper = jacksonObjectMapper()

    @Test
    fun `passing`() {
        mapper.writeValueAsString(wrapper) shouldBe json

        mapper.readValue<Wrapper<InnerType>>(json) shouldBe wrapper
    }

    @Test
    fun `failing`() {
        parseMethod<InnerType>(mapper, """{"value":{"value":"5"}}""") shouldBe wrapper
    }

    @Test
    fun `also failing`() {
        parseMethodUsingTypeReference<InnerType>(mapper, """{"value":{"value":"5"}}""") shouldBe wrapper
    }

    private inline fun <reified T: MarkerInterface> parseMethod(mapper: ObjectMapper, json: String) =
        mapper.readValue<Wrapper<T>>(json)

    private inline fun <reified T: MarkerInterface> parseMethodUsingTypeReference(mapper: ObjectMapper, json: String) =
        mapper.readValue<Wrapper<T>>(json, object : TypeReference<Wrapper<T>>() {})

}

Here is the error:

com.fasterxml.jackson.databind.exc.InvalidDefinitionException: Cannot construct instance of `com.toasttab.test.MarkerInterface` (no Creators, like default construct, exist): abstract types either need to be mapped to concrete types, have custom deserializer, or contain additional type information
 at [Source: (String)"{"value":{"value":"5"}}"; line: 1, column: 10] (through reference chain: com.toasttab.test.Wrapper["value"])

	at com.fasterxml.jackson.databind.exc.InvalidDefinitionException.from(InvalidDefinitionException.java:67)
	at com.fasterxml.jackson.databind.DeserializationContext.reportBadDefinition(DeserializationContext.java:1452)
	at com.fasterxml.jackson.databind.DeserializationContext.handleMissingInstantiator(DeserializationContext.java:1028)
	at com.fasterxml.jackson.databind.deser.AbstractDeserializer.deserialize(AbstractDeserializer.java:265)
	at com.fasterxml.jackson.databind.deser.SettableBeanProperty.deserialize(SettableBeanProperty.java:530)
	at com.fasterxml.jackson.databind.deser.BeanDeserializer._deserializeWithErrorWrapping(BeanDeserializer.java:528)
	at com.fasterxml.jackson.databind.deser.BeanDeserializer._deserializeUsingPropertyBased(BeanDeserializer.java:417)
	at com.fasterxml.jackson.databind.deser.BeanDeserializerBase.deserializeFromObjectUsingNonDefault(BeanDeserializerBase.java:1287)
	at com.fasterxml.jackson.databind.deser.BeanDeserializer.deserializeFromObject(BeanDeserializer.java:326)
	at com.fasterxml.jackson.databind.deser.BeanDeserializer.deserialize(BeanDeserializer.java:159)
	at com.fasterxml.jackson.databind.ObjectMapper._readMapAndClose(ObjectMapper.java:4014)
	at com.fasterxml.jackson.databind.ObjectMapper.readValue(ObjectMapper.java:3024)

[tomashanley-toast]

Ah nevermind, I've found the solution on stackoverflow here.

It turns out there is type erasure with reified, but reified helps to propagate the type's class, so you can to T::class.java.

This works for me:

    private inline fun <reified T: MarkerInterface> parseMethod(mapper: ObjectMapper, json: String): Wrapper<T> {
        val type = mapper.typeFactory.constructParametricType(Wrapper::class.java, T::class.java)
        return mapper.readValue<Wrapper<T>>(json, type)
    }

[fredgalvao]

Stumbled upon this when trying to generalize wrapped HTTP responses. Then, the suggested workaround from @tomashanley-toast to build a parametric type programmatically from two ::class.java didn't quite work, I had to add another step to keep all types safe from erasure.

TL;DR This is the updated solution able to parse nested generic types safely, modified from @tomashanley-toast's snippet:

private inline fun <reified T: MarkerInterface> parseMethod(mapper: ObjectMapper, json: String): Wrapper<T> {
  return mapper.typeFactory
    .let { tf -> tf.constructParametricType(Wrapper::class.java, tf.constructType(jacksonTypeRef<T>())) }
    .let { wrappedType -> mapper.readValue<Wrapper<T>>(json, wrappedType) }
}

Longer details

The thing is, T::class.java can only retain one level of the types involved, and anything deeper than that will be erased. Here's an example:

import com.fasterxml.jackson.databind.ObjectMapper
import com.fasterxml.jackson.databind.type.TypeFactory
import com.fasterxml.jackson.module.kotlin.KotlinModule
import com.fasterxml.jackson.module.kotlin.jacksonTypeRef
import io.vavr.jackson.datatype.VavrModule
import io.vavr.collection.List as VList


val m: ObjectMapper = ObjectMapper().registerModules(KotlinModule(), VavrModule())
val tf: TypeFactory = m.typeFactory

inline fun <reified T> parseWrappedAndUnwrap(json: String): T {
  val typeRefT = jacksonTypeRef<T>()
  val typeRefWT = jacksonTypeRef<Wrapper<T>>()
  val tJavaClass = T::class.java
  val w = tf.constructParametricType(Wrapper::class.java, tJavaClass)
  val ww = tf.constructParametricType(Wrapper::class.java, tf.constructType(typeRefT))

  println("jacksonTypeRef<T>().type:                ${typeRefT.type}")
  println("jacksonTypeRef<SimpleWrapper<T>>().type: ${typeRefWT.type}")
  println("T::class.java:                           $tJavaClass")
  println("tf.cPT(W::class.java, T::class.java):    $w")
  println("tf.cPT(W::class.java, tRef<T>.javaType): $ww")

  val parsed: Wrapper<T> = m.readValue(json, ww)

  return parsed.data
}

data class Wrapper<T>(val data: T)
data class Person<T>(val innerData: T)
data class Address(val street: String)

The only version of m.readValue() that can parse nested generic types safely is with ww. In other words, to retain full type information when going from a reified type T to a wrapped type of X<T>, first we need to convert T to a TypeReference, and then convert that to a JavaType, which then can be used to create the composite type.

Here are the basic tests I ran with that code (I used Vavr collections here because I need to guarantee that Jackson won't use the default interpretation of a JSON array, which is Java's ArrayList):

fun main() {
  val x1s = """{"data": {"innerData": {"street": "here"}}}"""
  val x1p: Person<Address> = parseWrappedAndUnwrap(x1s)
  x1p
    .also { println("x1p:") }
    .also {
      println("  it:                     $it")
      println("  it.javaClass:           ${it.javaClass.name}")
      println("  it.innerData.javaClass: ${it.innerData.javaClass.name}")
    }
    .also { println("-------------") }


  val x2s = """{"data": [[[{"innerData": {"street": "here"}}]]]}"""
  val x2p: VList<VList<VList<Person<Address>>>> = parseWrappedAndUnwrap(x2s)
  x2p
    .also { println("x2p:") }
    .also {
      println("  it:                              $it")
      println("  it.javaClass:                    ${it.javaClass.name}")
      println("  it.javaClass:                    ${it.javaClass.name}")
      println("  it[0].javaClass:                 ${it.first().javaClass.name}")
      println("  it[0][0].javaClass:              ${it.first().first().javaClass.name}")
      println("  it[0][0][0].javaClass:           ${it.first().first().first().javaClass.name}")
      println("  it[0][0][0].innerData.javaClass: ${it.first().first().first().innerData.javaClass.name}")
    }
    .also { println("-------------") }
}

Which gives me the following output:

jacksonTypeRef<T>().type:                Person<Address>
jacksonTypeRef<SimpleWrapper<T>>().type: Wrapper<T>
T::class.java:                           class Person
tf.cPT(W::class.java, T::class.java):    [simple type, class Wrapper<Person>]
tf.cPT(W::class.java, tRef<T>.javaType): [simple type, class Wrapper<Person<Address>>]
x1p:
  it:                     Person(innerData=Address(street=here))
  it.javaClass:           Person
  it.innerData.javaClass: Address
-------------
jacksonTypeRef<T>().type:                io.vavr.collection.List<io.vavr.collection.List<io.vavr.collection.List<Person<Address>>>>
jacksonTypeRef<SimpleWrapper<T>>().type: Wrapper<T>
T::class.java:                           interface io.vavr.collection.List
tf.cPT(W::class.java, T::class.java):    [simple type, class Wrapper<io.vavr.collection.List<java.lang.Object>>]
tf.cPT(W::class.java, tRef<T>.javaType): [simple type, class Wrapper<io.vavr.collection.List<io.vavr.collection.List<io.vavr.collection.List<Person<Address>>>>>]
x2p:
  it:                              List(List(List(Person(innerData=Address(street=here)))))
  it.javaClass:                    io.vavr.collection.List$Cons
  it.javaClass:                    io.vavr.collection.List$Cons
  it[0].javaClass:                 io.vavr.collection.List$Cons
  it[0][0].javaClass:              io.vavr.collection.List$Cons
  it[0][0][0].javaClass:           Person
  it[0][0][0].innerData.javaClass: Address
-------------