Occasional "Channel has been cancelled" when upload large file

[vcanuel]

Describe the bug

I have an error when I upload large file (but not always..)

I thought this was fixed with https://github.com/awslabs/smithy-kotlin/pull/624 but this seems the same issue as https://youtrack.jetbrains.com/issue/KTOR-1054

Expected behavior

No error

Current behavior

My upload is canceled with the following error :

Exception in thread "OkHttp Dispatcher" java.util.concurrent.CancellationException: Channel has been cancelled
	at java.base/jdk.internal.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
	at java.base/jdk.internal.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62)
	at java.base/jdk.internal.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
       .....
	at io.ktor.client.engine.okhttp.StreamRequestBody.writeTo(StreamRequestBody.kt:21)
Caused by: java.util.concurrent.CancellationException: Channel has been cancelled
	at io.ktor.utils.io.ByteBufferChannel.cancel(ByteBufferChannel.kt:180)
	at io.ktor.utils.io.ByteReadChannelKt.cancel(ByteReadChannel.kt:231)
	at io.ktor.utils.io.jvm.javaio.InputAdapter.close(Blocking.kt:73)
	at okio.InputStreamSource.close(JvmOkio.kt:108)
	at kotlin.io.CloseableKt.closeFinally(Closeable.kt:60)
	at io.ktor.client.engine.okhttp.StreamRequestBody.writeTo(StreamRequestBody.kt:20)
	... 15 more

Steps to Reproduce

Upload a large file :

private suspend fun putS3(): UploadResult { 
    val file = File("largeFile.pdf")
    
    val putRequest = PutObjectRequest {
	    bucket = outputS3Bucket
	    key = "$outputS3Prefix/myfile.json"
	    body = ByteStream.fromFile(file)
    }
    
       val uploadResponse = amazonS3ClientKt.putObject(request)
        return  UploadSuccess(bucketName, objectKey, MediaMetadata(fileName, uploadResponse.eTag))
}

Possible Solution

No response

Context

No response

AWS Kotlin SDK version used

0.14.4-beta

Platform (JVM/JS/Native)

JVM

Operating System and version

Mac OS 12.3.1 - Open JDK 17

[aajtodd]

@vcanuel Thanks for reporting this.

Anything else you can share about the operation in question? File size? Any debug/trace logs you may have gathered?

I'll take a look and see if I can reproduce it.

[aajtodd]

I've been able to see a similar exception only a few times in trying to recreate this.

Is this error bubbling up to an exception you see or is it just showing up in your logs?

[aajtodd]

I have been able to reproduce this only a handful of times but not been able to track down any rhyme or reason for it.

Our next short term step here is to enable additional logging to aid in tracking down a root cause. This will be available hopefully in the next release and I'll update instructions here when available.

---

The only issue that looked semi related if you squint hard enough is this okhttp issue.

Indeed several of the times that I was able to reproduce this the underlying cause seemed to be timeout related but there wasn't enough there to discern anything else useful. I'm putting this in the ticket as noteworthy and something to look into closer pending what additional telemetry turns up.

[vcanuel]

Hi, Thanks for your time on this issue. I have no more information than those initially provided, the operation is PutObjectRequest and I have extracted the relevant log. I usually reproduce this issue locally with localstack and a large file (> 1go). In production, we did not notice new occurrence of this bug.

[aajtodd]

Ok thanks for the update. The additional logging should be available in the next release (sometime this week). I'll update here when available. I'll also look into trying to recreate with localstack.

[github-actions[bot]]

⚠️COMMENT VISIBILITY WARNING⚠️

Comments on closed issues are hard for our team to see. If you need more assistance, please either tag a team member or open a new issue that references this one. If you wish to keep having a conversation with other community members under this issue feel free to do so.