Tom Kaitchuck

`testEndToEndWithChunkedStorage` is similarly affected. > io.pravega.segmentstore.server.host.HDFSIntegrationTest.testEndToEndWithChunkedStorage FAILED. Output (Last 300 lines): > at io.pravega.segmentstore.server.containers.StreamSegmentContainer.createStreamSegment(StreamSegmentContainer.java:518) > at io.pravega.segmentstore.server.containers.ContainerEventProcessorImpl.tryCreateAndPinInternalSegment(ContainerEventProcessorImpl.java:151) > at io.pravega.segmentstore.server.containers.ContainerEventProcessorImpl.lambda$getOrCreateInternalSegment$1(ContainerEventProcessorImpl.java:138) > at io.pravega.segmentstore.server.containers.ContainerEventProcessorImpl$EventProcessorImpl.refreshSegment(ContainerEventProcessorImpl.java:482) > at io.pravega.segmentstore.server.containers.ContainerEventProcessorImpl$EventProcessorImpl.processEvents(ContainerEventProcessorImpl.java:452) > at io.pravega.segmentstore.server.containers.ContainerEventProcessorImpl$EventProcessorImpl.lambda$doRun$4(ContainerEventProcessorImpl.java:419) >...

This test also sometime times out. Possibly due to this error: ``` java.lang.RuntimeException: Failed to start SegmentContainerManager at io.pravega.segmentstore.server.mocks.LocalSegmentContainerManager.initialize(LocalSegmentContainerManager.java:121) at io.pravega.segmentstore.server.store.ServiceBuilder.initialize(ServiceBuilder.java:247) at io.pravega.segmentstore.server.store.StreamSegmentStoreTestBase.createBuilder(StreamSegmentStoreTestBase.java:513) at io.pravega.segmentstore.server.store.StreamSegmentStoreTestBase.testSegmentRestoration(StreamSegmentStoreTestBase.java:198) at io.pravega.segmentstore.server.host.HDFSIntegrationTest.testDataRecovery(HDFSIntegrationTest.java:100) at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)...

More up to date stack trace: ``` io.pravega.segmentstore.server.host.HDFSIntegrationTest > testDataRecovery FAILED java.lang.NullPointerException at io.pravega.segmentstore.server.containers.ContainerRecoveryUtils.recoverAllSegments(ContainerRecoveryUtils.java:112) at io.pravega.segmentstore.server.store.StreamSegmentStoreTestBase.testSegmentRestoration(StreamSegmentStoreTestBase.java:245) at io.pravega.segmentstore.server.host.HDFSIntegrationTest.testDataRecovery(HDFSIntegrationTest.java:106) ```

Another failure: > java.util.concurrent.CompletionException: io.pravega.common.ObjectClosedException: Object 'StreamSegmentContainer' has been closed and cannot be accessed anymore. > at java.base/java.util.concurrent.CompletableFuture.encodeThrowable(CompletableFuture.java:314) > at java.base/java.util.concurrent.CompletableFuture.uniComposeStage(CompletableFuture.java:1113) > at java.base/java.util.concurrent.CompletableFuture.thenCompose(CompletableFuture.java:2235) > at io.pravega.segmentstore.server.containers.ContainerEventProcessorImpl.lambda$getOrCreateInternalSegment$1(ContainerEventProcessorImpl.java:131) > at io.pravega.segmentstore.server.containers.ContainerEventProcessorImpl$EventProcessorImpl.refreshSegment(ContainerEventProcessorImpl.java:449)...

It could also be cached between subsequent calls so that the lengths of sealed segments doesn't need to be re-queried.

An example of this is here: https://github.com/pravega/pravega-client-rust/runs/5636969184?check_suite_focus=true

This is not possible if a transaction is to preserve exactly once semantics unless the server is going to call back into the controller for each segment creation, which would...

This is still occuring: ``` io.pravega.segmentstore.server.containers.StreamSegmentContainerTests > testAppendWithOffset FAILED java.util.concurrent.TimeoutException: Expected value: 0 observed: 1 at io.pravega.test.common.AssertExtensions.assertEventuallyEquals(AssertExtensions.java:76) at io.pravega.test.common.AssertExtensions.assertEventuallyEquals(AssertExtensions.java:53) at io.pravega.segmentstore.server.containers.StreamSegmentContainerTests.checkAppendLeaks(StreamSegmentContainerTests.java:2713) at io.pravega.segmentstore.server.containers.StreamSegmentContainerTests.testAppendWithOffset(StreamSegmentContainerTests.java:1046) ```

The absence of an event retrieved from a reader does not mean that no events exist, merely that one was not obtained before the provided timeout. To check for the...

Ok. That leaves two possibilities, ether the issue is on the server or the client. The metrics produced by the server should tell what it perceives the latency as, and...