servicetalk icon indicating copy to clipboard operation
servicetalk copied to clipboard

Published 20 hours ago • verified publisher icon apple

Flaky Test: ConnectionCloseHeaderHandlingTest

Open idelpivnitskiy opened this issue 3 years ago • 13 comments

https://github.com/apple/servicetalk/runs/4017719358

ConnectionCloseHeaderHandlingTest$PipelinedRequestsTest.3: useUds=false, viaProxy=true, awaitRequestPayload=false

java.util.concurrent.ExecutionException: io.servicetalk.transport.netty.internal.CloseHandler$CloseEventObservedException: PROTOCOL_CLOSING_INBOUND(The application protocol closed the read side of this connection. This maybe the result of sending an HTTP header such as Connection: close.) [id: 0xe60bfed2, L:/127.0.0.1:45560 ! R:127.0.0.1/127.0.0.1:39923]
	at io.servicetalk.concurrent.api.SourceToFuture.reportGet(SourceToFuture.java:121)
	at io.servicetalk.concurrent.api.SourceToFuture.get(SourceToFuture.java:92)
	at io.servicetalk.http.netty.ConnectionCloseHeaderHandlingTest$ConnectionSetup.assertResponsePayloadBody(ConnectionCloseHeaderHandlingTest.java:251)
	at io.servicetalk.http.netty.ConnectionCloseHeaderHandlingTest$PipelinedRequestsTest.serverCloseTwoPipelinedRequestsSentBeforeFirstResponse(ConnectionCloseHeaderHandlingTest.java:371)
	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
...
Caused by: io.servicetalk.transport.netty.internal.CloseHandler$CloseEventObservedException: PROTOCOL_CLOSING_INBOUND(The application protocol closed the read side of this connection. This maybe the result of sending an HTTP header such as Connection: close.) [id: 0xe60bfed2, L:/127.0.0.1:45560 ! R:127.0.0.1/127.0.0.1:39923]
Caused by: io.servicetalk.transport.netty.internal.StacklessClosedChannelException
	at io.servicetalk.transport.netty.internal.DefaultNettyConnection.userEventTriggered(ChannelInputShutdownReadComplete)(Unknown Source)

idelpivnitskiy avatar Oct 27 '21 16:10 idelpivnitskiy

https://github.com/apple/servicetalk/pull/2000/checks?check_run_id=4453922696

ConnectionCloseHeaderHandlingTest$PipelinedRequestsTest.4: useUds=false, viaProxy=true, awaitRequestPayload=true

java.util.concurrent.ExecutionException: io.servicetalk.transport.netty.internal.CloseHandler$CloseEventObservedException: PROTOCOL_CLOSING_OUTBOUND(The application protocol closed the write side of this connection. This maybe the result of sending an HTTP header such as Connection: close.) [id: 0x2d583427, L:/127.0.0.1:43114 - R:127.0.0.1/127.0.0.1:38253]


java.util.concurrent.ExecutionException: io.servicetalk.transport.netty.internal.CloseHandler$CloseEventObservedException: PROTOCOL_CLOSING_OUTBOUND(The application protocol closed the write side of this connection. This maybe the result of sending an HTTP header such as Connection: close.) [id: 0x2d583427, L:/127.0.0.1:43114 - R:127.0.0.1/127.0.0.1:38253]
	at io.servicetalk.concurrent.api.SourceToFuture.reportGet(SourceToFuture.java:121)
	at io.servicetalk.concurrent.api.SourceToFuture.get(SourceToFuture.java:92)
	at io.servicetalk.http.netty.ConnectionCloseHeaderHandlingTest$ConnectionSetup.assertResponsePayloadBody(ConnectionCloseHeaderHandlingTest.java:251)
	at io.servicetalk.http.netty.ConnectionCloseHeaderHandlingTest$PipelinedRequestsTest.clientCloseTwoPipelinedRequestsSentFirstInitiatesClosure(ConnectionCloseHeaderHandlingTest.java:452)
	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.lang.reflect.Method.invoke(Method.java:498)
	at org.junit.platform.commons.util.ReflectionUtils.invokeMethod(ReflectionUtils.java:725)
	at org.junit.jupiter.engine.execution.MethodInvocation.proceed(MethodInvocation.java:60)
	at org.junit.jupiter.engine.execution.InvocationInterceptorChain$ValidatingInvocation.proceed(InvocationInterceptorChain.java:131)
	at org.junit.jupiter.engine.extension.TimeoutInvocation.proceed(TimeoutInvocation.java:46)
	at org.junit.jupiter.engine.extension.TimeoutExtension.intercept(TimeoutExtension.java:149)
	at org.junit.jupiter.engine.extension.TimeoutExtension.interceptTestableMethod(TimeoutExtension.java:140)
	at org.junit.jupiter.engine.extension.TimeoutExtension.interceptTestTemplateMethod(TimeoutExtension.java:92)
	at org.junit.jupiter.engine.execution.ExecutableInvoker$ReflectiveInterceptorCall.lambda$ofVoidMethod$0(ExecutableInvoker.java:115)
	at org.junit.jupiter.engine.execution.ExecutableInvoker.lambda$invoke$0(ExecutableInvoker.java:105)
	at org.junit.jupiter.engine.execution.InvocationInterceptorChain$InterceptedInvocation.proceed(InvocationInterceptorChain.java:106)
	at org.junit.jupiter.engine.execution.InvocationInterceptorChain.proceed(InvocationInterceptorChain.java:64)
	at org.junit.jupiter.engine.execution.InvocationInterceptorChain.chainAndInvoke(InvocationInterceptorChain.java:45)
	at org.junit.jupiter.engine.execution.InvocationInterceptorChain.invoke(InvocationInterceptorChain.java:37)
	at org.junit.jupiter.engine.execution.ExecutableInvoker.invoke(ExecutableInvoker.java:104)
	at org.junit.jupiter.engine.execution.ExecutableInvoker.invoke(ExecutableInvoker.java:98)
	at org.junit.jupiter.engine.descriptor.TestMethodTestDescriptor.lambda$invokeTestMethod$7(TestMethodTestDescriptor.java:214)
	at org.junit.platform.engine.support.hierarchical.ThrowableCollector.execute(ThrowableCollector.java:73)
	at org.junit.jupiter.engine.descriptor.TestMethodTestDescriptor.invokeTestMethod(TestMethodTestDescriptor.java:210)
	at org.junit.jupiter.engine.descriptor.TestMethodTestDescriptor.execute(TestMethodTestDescriptor.java:135)
	at org.junit.jupiter.engine.descriptor.TestMethodTestDescriptor.execute(TestMethodTestDescriptor.java:66)
	at org.junit.platform.engine.support.hierarchical.NodeTestTask.lambda$executeRecursively$6(NodeTestTask.java:151)
	at org.junit.platform.engine.support.hierarchical.ThrowableCollector.execute(ThrowableCollector.java:73)
	at org.junit.platform.engine.support.hierarchical.NodeTestTask.lambda$executeRecursively$8(NodeTestTask.java:141)
	at org.junit.platform.engine.support.hierarchical.Node.around(Node.java:137)
	at org.junit.platform.engine.support.hierarchical.NodeTestTask.lambda$executeRecursively$9(NodeTestTask.java:139)
	at org.junit.platform.engine.support.hierarchical.ThrowableCollector.execute(ThrowableCollector.java:73)
	at org.junit.platform.engine.support.hierarchical.NodeTestTask.executeRecursively(NodeTestTask.java:138)
	at org.junit.platform.engine.support.hierarchical.NodeTestTask.execute(NodeTestTask.java:95)
	at org.junit.platform.engine.support.hierarchical.ForkJoinPoolHierarchicalTestExecutorService$ExclusiveTask.compute(ForkJoinPoolHierarchicalTestExecutorService.java:185)
	at org.junit.platform.engine.support.hierarchical.ForkJoinPoolHierarchicalTestExecutorService.submit(ForkJoinPoolHierarchicalTestExecutorService.java:118)
	at org.junit.platform.engine.support.hierarchical.NodeTestTask$DefaultDynamicTestExecutor.execute(NodeTestTask.java:226)
	at org.junit.platform.engine.support.hierarchical.NodeTestTask$DefaultDynamicTestExecutor.execute(NodeTestTask.java:204)
	at org.junit.jupiter.engine.descriptor.TestTemplateTestDescriptor.execute(TestTemplateTestDescriptor.java:139)
	at org.junit.jupiter.engine.descriptor.TestTemplateTestDescriptor.lambda$execute$2(TestTemplateTestDescriptor.java:107)
	at java.util.stream.ForEachOps$ForEachOp$OfRef.accept(ForEachOps.java:183)
	at java.util.stream.ReferencePipeline$3$1.accept(ReferencePipeline.java:193)
	at java.util.stream.ReferencePipeline$2$1.accept(ReferencePipeline.java:175)
	at java.util.stream.ReferencePipeline$3$1.accept(ReferencePipeline.java:193)
	at java.util.stream.ForEachOps$ForEachOp$OfRef.accept(ForEachOps.java:183)
	at java.util.stream.ReferencePipeline$3$1.accept(ReferencePipeline.java:193)
	at java.util.stream.ReferencePipeline$3$1.accept(ReferencePipeline.java:193)
	at java.util.stream.ReferencePipeline$3$1.accept(ReferencePipeline.java:193)
	at java.util.stream.ForEachOps$ForEachOp$OfRef.accept(ForEachOps.java:183)
	at java.util.stream.ReferencePipeline$3$1.accept(ReferencePipeline.java:193)
	at java.util.Spliterators$ArraySpliterator.forEachRemaining(Spliterators.java:948)
	at java.util.stream.ReferencePipeline$Head.forEach(ReferencePipeline.java:647)
	at java.util.stream.ReferencePipeline$7$1.accept(ReferencePipeline.java:272)
	at java.util.stream.ReferencePipeline$3$1.accept(ReferencePipeline.java:193)
	at java.util.stream.ReferencePipeline$3$1.accept(ReferencePipeline.java:193)
	at java.util.Spliterators$ArraySpliterator.forEachRemaining(Spliterators.java:948)
	at java.util.stream.AbstractPipeline.copyInto(AbstractPipeline.java:482)
	at java.util.stream.AbstractPipeline.wrapAndCopyInto(AbstractPipeline.java:472)
	at java.util.stream.ForEachOps$ForEachOp.evaluateSequential(ForEachOps.java:150)
	at java.util.stream.ForEachOps$ForEachOp$OfRef.evaluateSequential(ForEachOps.java:173)
	at java.util.stream.AbstractPipeline.evaluate(AbstractPipeline.java:234)
	at java.util.stream.ReferencePipeline.forEach(ReferencePipeline.java:485)
	at java.util.stream.ReferencePipeline$7$1.accept(ReferencePipeline.java:272)
	at java.util.stream.ReferencePipeline$3$1.accept(ReferencePipeline.java:193)
	at java.util.stream.ReferencePipeline$3$1.accept(ReferencePipeline.java:193)
	at java.util.stream.ReferencePipeline$3$1.accept(ReferencePipeline.java:193)
	at java.util.ArrayList$ArrayListSpliterator.forEachRemaining(ArrayList.java:1384)
	at java.util.stream.AbstractPipeline.copyInto(AbstractPipeline.java:482)
	at java.util.stream.AbstractPipeline.wrapAndCopyInto(AbstractPipeline.java:472)
	at java.util.stream.ForEachOps$ForEachOp.evaluateSequential(ForEachOps.java:150)
	at java.util.stream.ForEachOps$ForEachOp$OfRef.evaluateSequential(ForEachOps.java:173)
	at java.util.stream.AbstractPipeline.evaluate(AbstractPipeline.java:234)
	at java.util.stream.ReferencePipeline.forEach(ReferencePipeline.java:485)
	at java.util.stream.ReferencePipeline$7$1.accept(ReferencePipeline.java:272)
	at java.util.ArrayList$ArrayListSpliterator.forEachRemaining(ArrayList.java:1384)
	at java.util.stream.AbstractPipeline.copyInto(AbstractPipeline.java:482)
	at java.util.stream.AbstractPipeline.wrapAndCopyInto(AbstractPipeline.java:472)
	at java.util.stream.ForEachOps$ForEachOp.evaluateSequential(ForEachOps.java:150)
	at java.util.stream.ForEachOps$ForEachOp$OfRef.evaluateSequential(ForEachOps.java:173)
	at java.util.stream.AbstractPipeline.evaluate(AbstractPipeline.java:234)
	at java.util.stream.ReferencePipeline.forEach(ReferencePipeline.java:485)
	at org.junit.jupiter.engine.descriptor.TestTemplateTestDescriptor.execute(TestTemplateTestDescriptor.java:107)
	at org.junit.jupiter.engine.descriptor.TestTemplateTestDescriptor.execute(TestTemplateTestDescriptor.java:42)
	at org.junit.platform.engine.support.hierarchical.NodeTestTask.lambda$executeRecursively$6(NodeTestTask.java:151)
	at org.junit.platform.engine.support.hierarchical.ThrowableCollector.execute(ThrowableCollector.java:73)
	at org.junit.platform.engine.support.hierarchical.NodeTestTask.lambda$executeRecursively$8(NodeTestTask.java:141)
	at org.junit.platform.engine.support.hierarchical.Node.around(Node.java:137)
	at org.junit.platform.engine.support.hierarchical.NodeTestTask.lambda$executeRecursively$9(NodeTestTask.java:139)
	at org.junit.platform.engine.support.hierarchical.ThrowableCollector.execute(ThrowableCollector.java:73)
	at org.junit.platform.engine.support.hierarchical.NodeTestTask.executeRecursively(NodeTestTask.java:138)
	at org.junit.platform.engine.support.hierarchical.NodeTestTask.execute(NodeTestTask.java:95)
	at org.junit.platform.engine.support.hierarchical.ForkJoinPoolHierarchicalTestExecutorService$ExclusiveTask.compute(ForkJoinPoolHierarchicalTestExecutorService.java:185)
	at org.junit.platform.engine.support.hierarchical.ForkJoinPoolHierarchicalTestExecutorService.executeNonConcurrentTasks(ForkJoinPoolHierarchicalTestExecutorService.java:155)
	at org.junit.platform.engine.support.hierarchical.ForkJoinPoolHierarchicalTestExecutorService.invokeAll(ForkJoinPoolHierarchicalTestExecutorService.java:135)
	at org.junit.platform.engine.support.hierarchical.NodeTestTask.lambda$executeRecursively$6(NodeTestTask.java:155)
	at org.junit.platform.engine.support.hierarchical.ThrowableCollector.execute(ThrowableCollector.java:73)
	at org.junit.platform.engine.support.hierarchical.NodeTestTask.lambda$executeRecursively$8(NodeTestTask.java:141)
	at org.junit.platform.engine.support.hierarchical.Node.around(Node.java:137)
	at org.junit.platform.engine.support.hierarchical.NodeTestTask.lambda$executeRecursively$9(NodeTestTask.java:139)
	at org.junit.platform.engine.support.hierarchical.ThrowableCollector.execute(ThrowableCollector.java:73)
	at org.junit.platform.engine.support.hierarchical.NodeTestTask.executeRecursively(NodeTestTask.java:138)
	at org.junit.platform.engine.support.hierarchical.NodeTestTask.execute(NodeTestTask.java:95)
	at org.junit.platform.engine.support.hierarchical.ForkJoinPoolHierarchicalTestExecutorService$ExclusiveTask.compute(ForkJoinPoolHierarchicalTestExecutorService.java:185)
	at org.junit.platform.engine.support.hierarchical.ForkJoinPoolHierarchicalTestExecutorService.executeNonConcurrentTasks(ForkJoinPoolHierarchicalTestExecutorService.java:155)
	at org.junit.platform.engine.support.hierarchical.ForkJoinPoolHierarchicalTestExecutorService.invokeAll(ForkJoinPoolHierarchicalTestExecutorService.java:135)
	at org.junit.platform.engine.support.hierarchical.NodeTestTask.lambda$executeRecursively$6(NodeTestTask.java:155)
	at org.junit.platform.engine.support.hierarchical.ThrowableCollector.execute(ThrowableCollector.java:73)
	at org.junit.platform.engine.support.hierarchical.NodeTestTask.lambda$executeRecursively$8(NodeTestTask.java:141)
	at org.junit.platform.engine.support.hierarchical.Node.around(Node.java:137)
	at org.junit.platform.engine.support.hierarchical.NodeTestTask.lambda$executeRecursively$9(NodeTestTask.java:139)
	at org.junit.platform.engine.support.hierarchical.ThrowableCollector.execute(ThrowableCollector.java:73)
	at org.junit.platform.engine.support.hierarchical.NodeTestTask.executeRecursively(NodeTestTask.java:138)
	at org.junit.platform.engine.support.hierarchical.NodeTestTask.execute(NodeTestTask.java:95)
	at org.junit.platform.engine.support.hierarchical.ForkJoinPoolHierarchicalTestExecutorService$ExclusiveTask.compute(ForkJoinPoolHierarchicalTestExecutorService.java:185)
	at org.junit.platform.engine.support.hierarchical.ForkJoinPoolHierarchicalTestExecutorService.executeNonConcurrentTasks(ForkJoinPoolHierarchicalTestExecutorService.java:155)
	at org.junit.platform.engine.support.hierarchical.ForkJoinPoolHierarchicalTestExecutorService.invokeAll(ForkJoinPoolHierarchicalTestExecutorService.java:135)
	at org.junit.platform.engine.support.hierarchical.NodeTestTask.lambda$executeRecursively$6(NodeTestTask.java:155)
	at org.junit.platform.engine.support.hierarchical.ThrowableCollector.execute(ThrowableCollector.java:73)
	at org.junit.platform.engine.support.hierarchical.NodeTestTask.lambda$executeRecursively$8(NodeTestTask.java:141)
	at org.junit.platform.engine.support.hierarchical.Node.around(Node.java:137)
	at org.junit.platform.engine.support.hierarchical.NodeTestTask.lambda$executeRecursively$9(NodeTestTask.java:139)
	at org.junit.platform.engine.support.hierarchical.ThrowableCollector.execute(ThrowableCollector.java:73)
	at org.junit.platform.engine.support.hierarchical.NodeTestTask.executeRecursively(NodeTestTask.java:138)
	at org.junit.platform.engine.support.hierarchical.NodeTestTask.execute(NodeTestTask.java:95)
	at org.junit.platform.engine.support.hierarchical.ForkJoinPoolHierarchicalTestExecutorService$ExclusiveTask.compute(ForkJoinPoolHierarchicalTestExecutorService.java:185)
	at java.util.concurrent.RecursiveAction.exec(RecursiveAction.java:189)
	at java.util.concurrent.ForkJoinTask.doExec(ForkJoinTask.java:289)
	at java.util.concurrent.ForkJoinPool$WorkQueue.runTask(ForkJoinPool.java:1056)
	at java.util.concurrent.ForkJoinPool.runWorker(ForkJoinPool.java:1692)
	at java.util.concurrent.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:175)
Caused by: io.servicetalk.transport.netty.internal.CloseHandler$CloseEventObservedException: PROTOCOL_CLOSING_OUTBOUND(The application protocol closed the write side of this connection. This maybe the result of sending an HTTP header such as Connection: close.) [id: 0x2d583427, L:/127.0.0.1:43114 - R:127.0.0.1/127.0.0.1:38253]
Caused by: io.netty.channel.unix.Errors$NativeIoException: readAddress(..) failed: Connection reset by peer

idelpivnitskiy avatar Dec 08 '21 07:12 idelpivnitskiy

https://github.com/apple/servicetalk/pull/2015/checks?check_run_id=4514556479

ConnectionCloseHeaderHandlingTest$PipelinedRequestsTest.4: useUds=false, viaProxy=true, awaitRequestPayload=true

java.util.concurrent.ExecutionException: io.servicetalk.transport.netty.internal.CloseHandler$CloseEventObservedException: PROTOCOL_CLOSING_INBOUND(The application protocol closed the read side of this connection. This maybe the result of sending an HTTP header such as Connection: close.) [id: 0xada435da, L:/127.0.0.1:33570 - R:127.0.0.1/127.0.0.1:43627]
 Raw output

idelpivnitskiy avatar Dec 14 '21 01:12 idelpivnitskiy

https://github.com/apple/servicetalk/runs/4719693892?check_suite_focus=true

ConnectionCloseHeaderHandlingTest > PipelinedRequestsTest > clientCloseTwoPipelinedRequestsSentFirstInitiatesClosure(boolean, boolean, boolean) > io.servicetalk.http.netty.ConnectionCloseHeaderHandlingTest$PipelinedRequestsTest.clientCloseTwoPipelinedRequestsSentFirstInitiatesClosure(boolean, boolean, boolean)[4] FAILED

bondolo avatar Jan 05 '22 23:01 bondolo

https://github.com/apple/servicetalk/pull/2050/checks?check_run_id=4796598740

ConnectionCloseHeaderHandlingTest$PipelinedRequestsTest.4: useUds=false, viaProxy=true, awaitRequestPayload=true

java.util.concurrent.ExecutionException: io.servicetalk.transport.netty.internal.CloseHandler$CloseEventObservedException: PROTOCOL_CLOSING_INBOUND(The application protocol closed the read side of this connection. This maybe the result of sending an HTTP header such as Connection: close.) [id: 0x0feca103, L:/127.0.0.1:45466 ! R:127.0.0.1/127.0.0.1:45899]
	at io.servicetalk.concurrent.api.SourceToFuture.reportGet(SourceToFuture.java:121)
	at io.servicetalk.concurrent.api.SourceToFuture.get(SourceToFuture.java:92)
	at io.servicetalk.http.netty.ConnectionCloseHeaderHandlingTest$ConnectionSetup.assertResponsePayloadBody(ConnectionCloseHeaderHandlingTest.java:251)
	at io.servicetalk.http.netty.ConnectionCloseHeaderHandlingTest$PipelinedRequestsTest.serverCloseTwoPipelinedRequestsSentBeforeFirstResponse(ConnectionCloseHeaderHandlingTest.java:371)
	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
...
Caused by: io.servicetalk.transport.netty.internal.CloseHandler$CloseEventObservedException: PROTOCOL_CLOSING_INBOUND(The application protocol closed the read side of this connection. This maybe the result of sending an HTTP header such as Connection: close.) [id: 0x0feca103, L:/127.0.0.1:45466 ! R:127.0.0.1/127.0.0.1:45899]
Caused by: io.servicetalk.transport.netty.internal.StacklessClosedChannelException
	at io.servicetalk.transport.netty.internal.DefaultNettyConnection.userEventTriggered(ChannelInputShutdownReadComplete)(Unknown Source)

idelpivnitskiy avatar Jan 12 '22 23:01 idelpivnitskiy

https://github.com/apple/servicetalk/pull/2059/checks?check_run_id=4889058015

ConnectionCloseHeaderHandlingTest$PipelinedRequestsTest.4: useUds=false, viaProxy=true, awaitRequestPayload=true
java.util.concurrent.ExecutionException: io.servicetalk.transport.netty.internal.CloseHandler$CloseEventObservedException: PROTOCOL_CLOSING_OUTBOUND(The application protocol closed the write side of this connection. This maybe the result of sending an HTTP header such as Connection: close.) [id: 0x6ee25c8c, L:/127.0.0.1:58356 - R:127.0.0.1/127.0.0.1:37967]

java.util.concurrent.ExecutionException: io.servicetalk.transport.netty.internal.CloseHandler$CloseEventObservedException: PROTOCOL_CLOSING_OUTBOUND(The application protocol closed the write side of this connection. This maybe the result of sending an HTTP header such as Connection: close.) [id: 0x6ee25c8c, L:/127.0.0.1:58356 - R:127.0.0.1/127.0.0.1:37967]
	at io.servicetalk.concurrent.api.SourceToFuture.reportGet(SourceToFuture.java:121)
	at io.servicetalk.concurrent.api.SourceToFuture.get(SourceToFuture.java:92)
	at io.servicetalk.http.netty.ConnectionCloseHeaderHandlingTest$ConnectionSetup.assertResponsePayloadBody(ConnectionCloseHeaderHandlingTest.java:251)
	at io.servicetalk.http.netty.ConnectionCloseHeaderHandlingTest$PipelinedRequestsTest.clientCloseTwoPipelinedRequestsSentFirstInitiatesClosure(ConnectionCloseHeaderHandlingTest.java:452)
	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
     …
Caused by: io.servicetalk.transport.netty.internal.CloseHandler$CloseEventObservedException: PROTOCOL_CLOSING_OUTBOUND(The application protocol closed the write side of this connection. This maybe the result of sending an HTTP header such as Connection: close.) [id: 0x6ee25c8c, L:/127.0.0.1:58356 - R:127.0.0.1/127.0.0.1:37967]
Caused by: io.netty.channel.unix.Errors$NativeIoException: readAddress(..) failed: Connection reset by peer

bondolo avatar Jan 21 '22 16:01 bondolo

https://github.com/apple/servicetalk/pull/2037/checks?check_run_id=5121085208

ConnectionCloseHeaderHandlingTest$PipelinedRequestsTest.3: useUds=false, viaProxy=true, awaitRequestPayload=false
java.util.concurrent.ExecutionException: io.servicetalk.transport.netty.internal.CloseHandler$CloseEventObservedException: PROTOCOL_CLOSING_INBOUND(The application protocol closed the read side of this connection. This maybe the result of sending an HTTP header such as Connection: close.) [id: 0x920fc2c1, L:/127.0.0.1:53798 - R:127.0.0.1/127.0.0.1:35429]

java.util.concurrent.ExecutionException: io.servicetalk.transport.netty.internal.CloseHandler$CloseEventObservedException: PROTOCOL_CLOSING_INBOUND(The application protocol closed the read side of this connection. This maybe the result of sending an HTTP header such as Connection: close.) [id: 0x920fc2c1, L:/127.0.0.1:53798 - R:127.0.0.1/127.0.0.1:35429]
	at io.servicetalk.concurrent.api.SourceToFuture.reportGet(SourceToFuture.java:121)
	at io.servicetalk.concurrent.api.SourceToFuture.get(SourceToFuture.java:92)
	at io.servicetalk.http.netty.ConnectionCloseHeaderHandlingTest$ConnectionSetup.assertResponsePayloadBody(ConnectionCloseHeaderHandlingTest.java:251)
	at io.servicetalk.http.netty.ConnectionCloseHeaderHandlingTest$PipelinedRequestsTest.serverCloseSecondPipelinedRequestWriteAborted(ConnectionCloseHeaderHandlingTest.java:405)
	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
...
Caused by: io.servicetalk.transport.netty.internal.CloseHandler$CloseEventObservedException: PROTOCOL_CLOSING_INBOUND(The application protocol closed the read side of this connection. This maybe the result of sending an HTTP header such as Connection: close.) [id: 0x920fc2c1, L:/127.0.0.1:53798 - R:127.0.0.1/127.0.0.1:35429]
Caused by: io.netty.channel.unix.Errors$NativeIoException: readAddress(..) failed: Connection reset by peer

idelpivnitskiy avatar Feb 09 '22 06:02 idelpivnitskiy

https://github.com/apple/servicetalk/runs/5207931543?check_suite_focus=true

on snapshot build:

ConnectionCloseHeaderHandlingTest$PipelinedRequestsTest.3: useUds=false, viaProxy=true, awaitRequestPayload=false


java.util.concurrent.ExecutionException: io.servicetalk.transport.netty.internal.CloseHandler$CloseEventObservedException: PROTOCOL_CLOSING_INBOUND(The application protocol closed the read side of this connection. This maybe the result of sending an HTTP header such as Connection: close.) [id: 0x8918cabc, L:/127.0.0.1:42526 - R:127.0.0.1/127.0.0.1:36287]
	at io.servicetalk.concurrent.api.SourceToFuture.reportGet(SourceToFuture.java:121)
	at io.servicetalk.concurrent.api.SourceToFuture.get(SourceToFuture.java:92)
	at io.servicetalk.http.netty.ConnectionCloseHeaderHandlingTest$ConnectionSetup.assertResponsePayloadBody(ConnectionCloseHeaderHandlingTest.java:251)
	at io.servicetalk.http.netty.ConnectionCloseHeaderHandlingTest$PipelinedRequestsTest.serverCloseSecondPipelinedRequestWriteAborted(ConnectionCloseHeaderHandlingTest.java:405)
	
...

Caused by: io.servicetalk.transport.netty.internal.CloseHandler$CloseEventObservedException: PROTOCOL_CLOSING_INBOUND(The application protocol closed the read side of this connection. This maybe the result of sending an HTTP header such as Connection: close.) [id: 0x8918cabc, L:/127.0.0.1:42526 - R:127.0.0.1/127.0.0.1:36287]
Caused by: io.netty.channel.unix.Errors$NativeIoException: readAddress(..) failed: Connection reset by peer

Scottmitch avatar Feb 22 '22 02:02 Scottmitch 

ConnectionCloseHeaderHandlingTest$PipelinedRequestsTest.4: useUds=false, viaProxy=true, awaitRequestPayload=true

java.util.concurrent.ExecutionException: io.servicetalk.transport.netty.internal.CloseHandler$CloseEventObservedException: PROTOCOL_CLOSING_OUTBOUND(The application protocol closed the write side of this connection. This maybe the result of sending an HTTP header such as Connection: close.) [id: 0x6d36f538, L:/127.0.0.1:53976 - R:127.0.0.1/127.0.0.1:37059]
	at io.servicetalk.concurrent.api.SourceToFuture.reportGet(SourceToFuture.java:121)
	at io.servicetalk.concurrent.api.SourceToFuture.get(SourceToFuture.java:92)
	at io.servicetalk.http.netty.ConnectionCloseHeaderHandlingTest$ConnectionSetup.assertResponsePayloadBody(ConnectionCloseHeaderHandlingTest.java:251)
	at io.servicetalk.http.netty.ConnectionCloseHeaderHandlingTest$PipelinedRequestsTest.clientCloseTwoPipelinedRequestsSentFirstInitiatesClosure(ConnectionCloseHeaderHandlingTest.java:452)


Caused by: io.servicetalk.transport.netty.internal.CloseHandler$CloseEventObservedException: PROTOCOL_CLOSING_OUTBOUND(The application protocol closed the write side of this connection. This maybe the result of sending an HTTP header such as Connection: close.) [id: 0x6d36f538, L:/127.0.0.1:53976 - R:127.0.0.1/127.0.0.1:37059]
Caused by: io.netty.channel.unix.Errors$NativeIoException: readAddress(..) failed: Connection reset by peer

Scottmitch avatar Feb 27 '22 22:02 Scottmitch

https://github.com/apple/servicetalk/runs/5546627377

ConnectionCloseHeaderHandlingTest$NonPipelinedRequestsTest.27: useUds=false, viaProxy=true, awaitRequestPayload=false, requestInitiatesClosure=true, noRequestContent=false, noResponseContent=true
java.lang.AssertionError: 
Expected: "0"
     but: was "15"

at org.hamcrest.MatcherAssert.assertThat(MatcherAssert.java:20)
	at org.hamcrest.MatcherAssert.assertThat(MatcherAssert.java:6)
	at io.servicetalk.http.netty.ConnectionCloseHeaderHandlingTest$NonPipelinedRequestsTest.testConnectionClosure(ConnectionCloseHeaderHandlingTest.java:322)

Scottmitch avatar Mar 15 '22 00:03 Scottmitch

https://github.com/apple/servicetalk/runs/6014945862?check_suite_focus=true

ConnectionCloseHeaderHandlingTest > NonPipelinedRequestsTest > testConnectionClosure(boolean, boolean, boolean, boolean, boolean, boolean) > io.servicetalk.http.netty.ConnectionCloseHeaderHandlingTest$NonPipelinedRequestsTest.testConnectionClosure(boolean, boolean, boolean, boolean, boolean, boolean)[28] FAILED
    java.util.concurrent.ExecutionException: io.servicetalk.transport.netty.internal.CloseHandler$CloseEventObservedException: PROTOCOL_CLOSING_OUTBOUND(The application protocol closed the write side of this connection. This maybe the result of sending an HTTP header such as Connection: close.) [id: 0xc7026128, L:/127.0.0.1:48366 - R:127.0.0.1/127.0.0.1:39247]
        at io.servicetalk.concurrent.api.SourceToFuture.reportGet(SourceToFuture.java:121)
        at io.servicetalk.concurrent.api.SourceToFuture.get(SourceToFuture.java:92)
        at io.servicetalk.http.netty.ConnectionCloseHeaderHandlingTest$ConnectionSetup.assertResponsePayloadBody(ConnectionCloseHeaderHandlingTest.java:248)
        at io.servicetalk.http.netty.ConnectionCloseHeaderHandlingTest$NonPipelinedRequestsTest.testConnectionClosure(ConnectionCloseHeaderHandlingTest.java:319)

        Caused by:
        io.servicetalk.transport.netty.internal.CloseHandler$CloseEventObservedException: PROTOCOL_CLOSING_OUTBOUND(The application protocol closed the write side of this connection. This maybe the result of sending an HTTP header such as Connection: close.) [id: 0xc7026128, L:/127.0.0.1:48366 - R:127.0.0.1/127.0.0.1:39247]

            Caused by:
            io.netty.channel.unix.Errors$NativeIoException: readAddress(..) failed: Connection reset by peer

bondolo avatar Apr 13 '22 21:04 bondolo

https://github.com/apple/servicetalk/pull/2166/checks?check_run_id=6042825262

ConnectionCloseHeaderHandlingTest$PipelinedRequestsTest.4: useUds=false, viaProxy=true, awaitRequestPayload=true
java.util.concurrent.ExecutionException: io.servicetalk.transport.netty.internal.CloseHandler$CloseEventObservedException: PROTOCOL_CLOSING_OUTBOUND(The application protocol closed the write side of this connection. This maybe the result of sending an HTTP header such as Connection: close.) [id: 0xd92ebdb6, L:/127.0.0.1:36204 - R:127.0.0.1/127.0.0.1:39121]
java.util.concurrent.ExecutionException: io.servicetalk.transport.netty.internal.CloseHandler$CloseEventObservedException: PROTOCOL_CLOSING_OUTBOUND(The application protocol closed the write side of this connection. This maybe the result of sending an HTTP header such as Connection: close.) [id: 0xd92ebdb6, L:/127.0.0.1:36204 - R:127.0.0.1/127.0.0.1:39121]
	at io.servicetalk.concurrent.api.SourceToFuture.reportGet(SourceToFuture.java:121)
	at io.servicetalk.concurrent.api.SourceToFuture.get(SourceToFuture.java:94)
	at io.servicetalk.http.netty.ConnectionCloseHeaderHandlingTest$ConnectionSetup.assertResponsePayloadBody(ConnectionCloseHeaderHandlingTest.java:248)
	at io.servicetalk.http.netty.ConnectionCloseHeaderHandlingTest$PipelinedRequestsTest.clientCloseTwoPipelinedRequestsSentSecondInitiatesClosure(ConnectionCloseHeaderHandlingTest.java:479)
	…
Caused by: io.servicetalk.transport.netty.internal.CloseHandler$CloseEventObservedException: PROTOCOL_CLOSING_OUTBOUND(The application protocol closed the write side of this connection. This maybe the result of sending an HTTP header such as Connection: close.) [id: 0xd92ebdb6, L:/127.0.0.1:36204 - R:127.0.0.1/127.0.0.1:39121]
Caused by: io.netty.channel.unix.Errors$NativeIoException: readAddress(..) failed: Connection reset by peer

bondolo avatar Apr 15 '22 21:04 bondolo

snapshot build https://github.com/apple/servicetalk/runs/6330132708

ConnectionCloseHeaderHandlingTest$PipelinedRequestsTest.4: useUds=false, viaProxy=true, awaitRequestPayload=true

java.util.concurrent.ExecutionException: io.servicetalk.transport.netty.internal.CloseHandler$CloseEventObservedException: PROTOCOL_CLOSING_INBOUND(The application protocol closed the read side of this connection. This maybe the result of sending an HTTP header such as Connection: close.) [id: 0xb30d5393, L:/127.0.0.1:38054 - R:127.0.0.1/127.0.0.1:41383]
	at io.servicetalk.concurrent.api.SourceToFuture.reportGet(SourceToFuture.java:121)
	at io.servicetalk.concurrent.api.SourceToFuture.get(SourceToFuture.java:92)
	at io.servicetalk.http.netty.ConnectionCloseHeaderHandlingTest$ConnectionSetup.assertResponsePayloadBody(ConnectionCloseHeaderHandlingTest.java:248)
	at io.servicetalk.http.netty.ConnectionCloseHeaderHandlingTest$PipelinedRequestsTest.serverCloseTwoPipelinedRequestsInSequence(ConnectionCloseHeaderHandlingTest.java:425)

Caused by: io.servicetalk.transport.netty.internal.CloseHandler$CloseEventObservedException: PROTOCOL_CLOSING_INBOUND(The application protocol closed the read side of this connection. This maybe the result of sending an HTTP header such as Connection: close.) [id: 0xb30d5393, L:/127.0.0.1:38054 - R:127.0.0.1/127.0.0.1:41383]
Caused by: io.netty.channel.unix.Errors$NativeIoException: readAddress(..) failed: Connection reset by peer

Scottmitch avatar May 06 '22 22:05 Scottmitch

https://github.com/apple/servicetalk/pull/2241/checks?check_run_id=6782920058

ConnectionCloseHeaderHandlingTest$NonPipelinedRequestsTest.26: useUds=false, viaProxy=true, awaitRequestPayload=false, requestInitiatesClosure=true, noRequestContent=true, noResponseContent=false
java.util.concurrent.ExecutionException: io.servicetalk.transport.netty.internal.CloseHandler$CloseEventObservedException: PROTOCOL_CLOSING_OUTBOUND(The application protocol closed the write side of this connection. This maybe the result of sending an HTTP header such as Connection: close.) [id: 0x7151d3cd, L:/127.0.0.1:58796 - R:127.0.0.1/127.0.0.1:35507]

java.util.concurrent.ExecutionException: io.servicetalk.transport.netty.internal.CloseHandler$CloseEventObservedException: PROTOCOL_CLOSING_OUTBOUND(The application protocol closed the write side of this connection. This maybe the result of sending an HTTP header such as Connection: close.) [id: 0x7151d3cd, L:/127.0.0.1:58796 - R:127.0.0.1/127.0.0.1:35507]
	at io.servicetalk.concurrent.api.SourceToFuture.reportGet(SourceToFuture.java:121)
	at io.servicetalk.concurrent.api.SourceToFuture.get(SourceToFuture.java:92)
	at io.servicetalk.http.netty.ConnectionCloseHeaderHandlingTest$ConnectionSetup.assertResponsePayloadBody(ConnectionCloseHeaderHandlingTest.java:248)
	at io.servicetalk.http.netty.ConnectionCloseHeaderHandlingTest$NonPipelinedRequestsTest.testConnectionClosure(ConnectionCloseHeaderHandlingTest.java:319)
	at sun.reflect.GeneratedMethodAccessor42.invoke(Unknown Source)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.lang.reflect.Method.invoke(Method.java:498)
	at org.junit.platform.commons.util.ReflectionUtils.invokeMethod(ReflectionUtils.java:725)
	at org.junit.jupiter.engine.execution.MethodInvocation.proceed(MethodInvocation.java:60)
	at org.junit.jupiter.engine.execution.InvocationInterceptorChain$ValidatingInvocation.proceed(InvocationInterceptorChain.java:131)
	at org.junit.jupiter.engine.extension.TimeoutInvocation.proceed(TimeoutInvocation.java:46)

bondolo avatar Jun 07 '22 21:06 bondolo

https://github.com/apple/servicetalk/pull/2391/checks?check_run_id=8833884743

ConnectionCloseHeaderHandlingTest$PipelinedRequestsTest.4: useUds=false, viaProxy=true, awaitRequestPayload=true
java.util.concurrent.ExecutionException: io.servicetalk.transport.netty.internal.CloseHandler$CloseEventObservedException: PROTOCOL_CLOSING_INBOUND(The application protocol closed the read side of this connection. This maybe the result of sending an HTTP header such as Connection: close.) [id: 0x008b61af, L:/127.0.0.1:38552 ! R:127.0.0.1/127.0.0.1:33585]
	at io.servicetalk.concurrent.api.SourceToFuture.reportGet(SourceToFuture.java:121)
	at io.servicetalk.concurrent.api.SourceToFuture.get(SourceToFuture.java:92)
	at io.servicetalk.http.netty.ConnectionCloseHeaderHandlingTest$ConnectionSetup.assertResponsePayloadBody(ConnectionCloseHeaderHandlingTest.java:248)
	at io.servicetalk.http.netty.ConnectionCloseHeaderHandlingTest$PipelinedRequestsTest.serverCloseTwoPipelinedRequestsSentBeforeFirstResponse(ConnectionCloseHeaderHandlingTest.java:368)
	at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
...
Caused by: io.servicetalk.transport.netty.internal.CloseHandler$CloseEventObservedException: PROTOCOL_CLOSING_INBOUND(The application protocol closed the read side of this connection. This maybe the result of sending an HTTP header such as Connection: close.) [id: 0x008b61af, L:/127.0.0.1:38552 ! R:127.0.0.1/127.0.0.1:33585]
Caused by: io.servicetalk.transport.netty.internal.StacklessClosedChannelException
	at io.servicetalk.transport.netty.internal.DefaultNettyConnection.userEventTriggered(ChannelInputShutdownReadComplete)(Unknown Source)

idelpivnitskiy avatar Oct 11 '22 23:10 idelpivnitskiy

This was not flaky for more than a year. Reopen if it happens again.

idelpivnitskiy avatar Oct 25 '23 22:10 idelpivnitskiy