Illegal trailing byte

[PinkieSwirl]

Ktor Version

1.1.3

Ktor Engine Used(client or server and name)

jetty server

JVM Version, Operating System and Relevant Context

11, Windows 10

Feedback

I get the following error on a big/long session:

kotlinx.io.charsets.MalformedInputException: Illegal trailing byte
	at kotlinx.coroutines.io.ByteBufferChannel$readUTF8LineToUtf8Suspend$2.invokeSuspend(ByteBufferChannel.kt:2130)
	at kotlinx.coroutines.io.ByteBufferChannel$readUTF8LineToUtf8Suspend$2.invoke(ByteBufferChannel.kt)
	at kotlinx.coroutines.io.ByteBufferChannel.lookAheadSuspend(ByteBufferChannel.kt:1821)
	at kotlinx.coroutines.io.ByteBufferChannel.readUTF8LineToUtf8Suspend(ByteBufferChannel.kt:2113)
	at kotlinx.coroutines.io.ByteBufferChannel.readUTF8LineToAscii(ByteBufferChannel.kt:2053)
	at kotlinx.coroutines.io.ByteBufferChannel.readUTF8LineTo(ByteBufferChannel.kt:2141)
	at kotlinx.coroutines.io.ByteBufferChannel.readUTF8Line(ByteBufferChannel.kt:2145)
	at kotlinx.coroutines.io.ByteReadChannelKt.readUTF8Line(ByteReadChannel.kt:189)
	at io.ktor.sessions.SessionTrackerById$load$2.invokeSuspend(SessionTrackerById.kt:31)
	at io.ktor.sessions.SessionTrackerById$load$2.invoke(SessionTrackerById.kt)
	at io.ktor.sessions.CacheStorage.read(CacheStorage.kt:17)
	at io.ktor.sessions.SessionTrackerById.load(SessionTrackerById.kt:30)
	at io.ktor.sessions.Sessions$Feature$install$1.invokeSuspend(Sessions.kt:45)
	at io.ktor.sessions.Sessions$Feature$install$1.invoke(Sessions.kt)
	at io.ktor.util.pipeline.SuspendFunctionGun.loop(PipelineContext.kt:278)
	at io.ktor.util.pipeline.SuspendFunctionGun.access$loop(PipelineContext.kt:63)
	at io.ktor.util.pipeline.SuspendFunctionGun.proceed(PipelineContext.kt:137)
	at io.ktor.features.StatusPages$interceptCall$2.invokeSuspend(StatusPages.kt:94)
	at io.ktor.features.StatusPages$interceptCall$2.invoke(StatusPages.kt)
	at kotlinx.coroutines.intrinsics.UndispatchedKt.startUndispatchedOrReturn(Undispatched.kt:91)
	at kotlinx.coroutines.CoroutineScopeKt.coroutineScope(CoroutineScope.kt:186)
	at io.ktor.features.StatusPages.interceptCall(StatusPages.kt:93)
	at io.ktor.features.StatusPages$Feature$install$2.invokeSuspend(StatusPages.kt:133)
	at io.ktor.features.StatusPages$Feature$install$2.invoke(StatusPages.kt)
	at io.ktor.util.pipeline.SuspendFunctionGun.loop(PipelineContext.kt:278)
	at io.ktor.util.pipeline.SuspendFunctionGun.access$loop(PipelineContext.kt:63)
	at io.ktor.util.pipeline.SuspendFunctionGun.proceed(PipelineContext.kt:137)
	at io.ktor.features.CallLogging$Feature$install$2.invokeSuspend(CallLogging.kt:130)
	at io.ktor.features.CallLogging$Feature$install$2.invoke(CallLogging.kt)
	at io.ktor.util.pipeline.SuspendFunctionGun.loop(PipelineContext.kt:278)
	at io.ktor.util.pipeline.SuspendFunctionGun.access$loop(PipelineContext.kt:63)
	at io.ktor.util.pipeline.SuspendFunctionGun.proceed(PipelineContext.kt:137)
	at io.ktor.util.pipeline.SuspendFunctionGun.execute(PipelineContext.kt:157)
	at io.ktor.util.pipeline.Pipeline.execute(Pipeline.kt:23)
	at io.ktor.server.engine.DefaultEnginePipelineKt$defaultEnginePipeline$2.invokeSuspend(DefaultEnginePipeline.kt:106)
	at io.ktor.server.engine.DefaultEnginePipelineKt$defaultEnginePipeline$2.invoke(DefaultEnginePipeline.kt)
	at io.ktor.util.pipeline.SuspendFunctionGun.loop(PipelineContext.kt:278)
	at io.ktor.util.pipeline.SuspendFunctionGun.access$loop(PipelineContext.kt:63)
	at io.ktor.util.pipeline.SuspendFunctionGun.proceed(PipelineContext.kt:137)
	at io.ktor.util.pipeline.SuspendFunctionGun.execute(PipelineContext.kt:157)
	at io.ktor.util.pipeline.Pipeline.execute(Pipeline.kt:23)
	at io.ktor.server.jetty.JettyKtorHandler$handle$2.invokeSuspend(JettyKtorHandler.kt:96)
	at kotlin.coroutines.jvm.internal.BaseContinuationImpl.resumeWith(ContinuationImpl.kt:32)
	at kotlinx.coroutines.DispatchedTask.run(Dispatched.kt:233)
	at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515)
	at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264)
	at java.base/java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:304)
	at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
	at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
	at java.base/java.lang.Thread.run(Thread.java:834)

Not sure if this belongs here or to kotlinx.serialization.

[vngantk]

I hit the same problem. I guess the problem is due to inconsistence between the way the serialized session data is written to the session store and the way it is read from the session store. If you look at the code of SessionTrackerById, the serialized session data is written to the session store using channel.writeStringUtf8() without appending any newline at the end. However, the session data is read from session store using channel.readUTF8Line(). According to documentation, readUTF8Line() requires a newline character as terminator to indicate end of read. Another problem is that if there are embedded newline characters appearing in the serialized session data then they could mislead readUTF8Line() to think that it is the end of input resulting in incomplete data read. Please see my comments in the following code fragments of SessionTrackerById:

    override suspend fun load(call: ApplicationCall, transport: String?): Any? {
        val sessionId = transport ?: return null
        call.attributes.put(SessionIdKey, sessionId)
        try {
            return storage.read(sessionId) { channel ->
                // the following read requires a newline in order to work, and it may 
                // also incorrectly consider an embedded newline as terminator:
                val text = channel.readUTF8Line() ?: throw IllegalStateException("Failed to read stored session from $channel")
                serializer.deserialize(text)
            }
        } catch (notFound: NoSuchElementException) {
            call.application.log.debug("Failed to lookup session: $notFound")
        }
        return null
    }

    override suspend fun store(call: ApplicationCall, value: Any): String {
        val sessionId = call.attributes.computeIfAbsent(SessionIdKey, sessionIdProvider)
        val serialized = serializer.serialize(value)
        storage.write(sessionId) { channel ->
            // the following code lacks an explicit writing of a newline at the end for 
            // readUTF8Line() to determine the end of the read operation. On the other 
            // hand, it can also cause problems if there are embedded newline characters
            // inside the serialized data, for example, if the serialized data is a pretty-
            // printed JSON or XML with newline characters at the end of each line:
            channel.writeStringUtf8(serialized)
            channel.close()
        }
        return sessionId
    }

[oleg-larshin]

Please check the following ticket on YouTrack for follow-ups to this issue. GitHub issues will be closed in the coming weeks.