ktorio
Unexpected High Memory Usage When Downloading Large Files Using Ktor HTTP Client
Hi Ktor team,
First of all, thank you for the continued maintenance and regular releases of the Ktor project! It's great to see an actively developed HTTP client in the Kotlin ecosystem.
I’d like to report a potential memory usage issue I observed when using the HttpClient(CIO) to download a large file.
Problem
When downloading a large file (e.g., ~1000MB+), the memory usage spikes dramatically — exceeding 4GB in peak memory — and eventually crashes with an OutOfMemoryError when running with JVM options -Xmx100m -Xms100m.
We're using Ktor 3.1.3 with the following dependencies:
implementation("io.ktor:ktor-client-core:3.1.3")
implementation("io.ktor:ktor-client-cio:3.1.3")
implementation("io.ktor:ktor-client-encoding:3.1.3")
Here's a minimal example that reproduces the issue:
fun testKtorClient() {
val httpClient = HttpClient(CIO) {
install(ContentEncoding) {
deflate()
gzip()
}
followRedirects = true
install(HttpTimeout) {
requestTimeoutMillis = 120_000
connectTimeoutMillis = 10_000
}
defaultRequest {
header(HttpHeaders.UserAgent, "Memory-Usage-Test/1.0 (<email-address>)")
header(HttpHeaders.AcceptEncoding, "gzip,deflate")
header(HttpHeaders.Host, "www.sec.gov")
}
}
runBlocking {
val response = httpClient.get("http://www.sec.gov/Archives/edgar/daily-index/xbrl/companyfacts.zip")
val tempFile = Files.createTempFile("ktor-client-", "")
tempFile.toFile().deleteOnExit()
tempFile.outputStream().buffered().use { fileOutputStream ->
response.bodyAsChannel().copyTo(fileOutputStream)
}
}
}
Comparison
Using OkHttp with a similar logic consumes minimal memory and works fine even under 100MB heap constraints.
We're using OkHttp 4.12.0 with:
implementation("com.squareup.okhttp3:okhttp:4.12.0")
Here's the OkHttp version:
fun testOkHttpClient() {
val httpClient = OkHttpClient.Builder()
.connectTimeout(Duration.ofSeconds(10))
.readTimeout(Duration.ofSeconds(120))
.addInterceptor {
val original = it.request()
val request = original.newBuilder()
.header("User-Agent", "Memory-Usage-Test/1.0 (<email-address>)")
.header("Accept-Encoding", "gzip,deflate")
.header("Host", "www.sec.gov")
.build()
it.proceed(request)
}
.build()
val request = Request.Builder()
.url("http://www.sec.gov/Archives/edgar/daily-index/xbrl/companyfacts.zip")
.build()
httpClient.newCall(request).execute().use { response ->
val tempFile = Files.createTempFile("okhttp-client-", "")
tempFile.toFile().deleteOnExit()
tempFile.sink().buffer().use { sink ->
response.body?.source()?.use { source ->
sink.writeAll(source)
}
}
}
}
Expectations
- I expect Ktor to stream and decompress content in a memory-efficient manner, ideally without loading the entire body into memory.
- I'd appreciate any insights into whether this is an intended behavior, a misconfiguration on my part, or a known issue with CIO or the content decoding pipeline.
Please let me know if you need further details or if I can help with a reproducer project.
Thanks again for your work on Ktor!
Thank you for the detailed issue description!
Could you check if the issue persists if you use .prepareGet("...").execute { response -> ... } instead of .get("...")? This is described in the docs about receiving streaming responses.
client.get("...") saves the entire response body in memory, to be able to release connection and free up resources immediately, while execute doesn't.
Hi @osipxd, thanks a lot for the helpful response!
Yes I can confirm that after replacing the get(...) call with .preparedGet(...).execute { ... }, the memory usage issue is resolved. It now consumes memory comparable to the OkHttpClient.
Here is the updated working example
runBlocking {
val tempFile = Files.createTempFile("ktor-client-", "")
tempFile.toFile().deleteOnExit()
val statement= httpClient.preparedGet("http://www.sec.gov/Archives/edgar/daily-index/xbrl/companyfacts.zip")
statement.execute { response ->
tempFile.outputStream().buffered().use { fileOutputStream ->
response.bodyAsChannel().copyTo(fileOutputStream)
}
}
}
However, one advantage I really enjoy about Ktor is the coroutine-based composing of suspending functions, which helps avoid callback nesting and keeps suspend function composition clean. In this case, it seems that we're forced to use the callback-style execute { ... } to avoid buffering the entire response in memory, and the callback function itself contains two suspend functions.
I also tested the no-parameter version of execute() like this:
runBlocking {
val tempFile = Files.createTempFile("ktor-client-", "")
tempFile.toFile().deleteOnExit()
val response= httpClient.preparedGet("http://www.sec.gov/Archives/edgar/daily-index/xbrl/companyfacts.zip").execute()
tempFile.outputStream().buffered().use { fileOutputStream ->
response.bodyAsChannel().copyTo(fileOutputStream)
}
}
But in this case, it appears the entire response body is still buffered into memory — similar to the .get() behavior.
That behavior is understandable, since suspend function composition is sequential by default. I guess the real issue is that whether using .get() or .prepareGet().execute(), once the suspend function completes, the response body is fully read, and the underlying HTTP connection is already closed.
It would be great if Ktor could decouple response body consumption from connection lifecycle management — for example, allowing users to obtain an OutputStream or ByteReadChannel that streams directly from the connection without forcing full body consumption upfront.
Since the above discussion is purely about code style preferences, which naturally differ from person to person, feel free to mark this high memory usage issue as resolved. We now have a reliable workaround, and it's already documented.
Thanks again for your time and support!
I am trying to download a file close to 1 GB using exactly the same code provided in the documentation. But I am still hitting OOM. Is there any other ways to download such large files?
