NettyHttpServer control flow, internal state, test failures

[Scottmitch]

trafficstars

The control flow for NettyHttpServer uses a series of operators and streaming transformations in order to orchestrate handling requests/responses. The stream is then repeated in order to facilitate handling multiple requests on the same stream. Our internal utilities (e.g. CloseHandler, WriteStreamSubscriber, FlushStrategy) are geared toward control flow which has an independent stream for each operation, and have been adapted to propagate state as necessary (e.g. connection level flushing). However in some cases this may lead toward internal state not always being consistent which may lead to undesirable results [1].

There may also be an opportunity to simplify request-based state management (e.g. FlushStrategy) while addressing this issue.

[1] close handler not progressing server connection to closed during graceful closure

• the close handler is waiting for the write to complete successfully (https://github.com/apple/servicetalk/blob/master/servicetalk-http-netty/src/main/java/io/servicetalk/http/netty/HttpObjectEncoder.java#L190-L193)
• netty notifies the write future for the trailers but AllWritesPromise doesn't notify because the subscriber hasn't terminated (https://github.com/apple/servicetalk/blob/master/servicetalk-transport-netty-internal/src/main/java/io/servicetalk/transport/netty/internal/WriteStreamSubscriber.java#L356)
• the WriteStreamSubscriber doesn't terminate because of the control flow on the server has the repeat (https://github.com/apple/servicetalk/blob/master/servicetalk-http-netty/src/main/java/io/servicetalk/http/netty/NettyHttpServer.java#L361)

[Scottmitch]

Related items for consideration:

• https://github.com/apple/servicetalk/pull/980#discussion_r396835362
• https://github.com/apple/servicetalk/pull/980#discussion_r396835687

[idelpivnitskiy]

Resolved by #2367, #2423, #2471