[tcp] Part of Transition from SYN_RCVD to ESTABLISHED is Delayed, Causing Packet Loss

[BrianZill]

Description

We currently have 3 different Hash Maps for routing incoming packets to their correct connections. One each for active-open (SYN_SENT), passive-open (LISTEN, SYN_RCVD), and established. When we are in passive-open LISTEN and receive a SYN packet we go to SYN_RCVD and send a SYN-ACK packet. When we receive an ACK for our SYN-ACK we should go to ESTABLISHED, and we do (sort-of). The problem is that we should create the right entry in the established Hash Map immediately but we don't. The Hash Map doesn't get updated until the user's Accept completes.

There is a major problem with this: Any packets arriving for this connection prior to the user's accept() completing will go to the wrong place and be lost. Since it is a fairly common place communication pattern for a client connecting to a server to immediately send data to the server as soon as the connection is completed, this bug will be hit a lot (and ruin our RPC performance numbers).

There is also a related bug. Any data that accompanies the ACK for the SYN-ACK will also be lost, because even though we create the established Control Block when we transition to ESTABLISHED state, we don't check for data accompanying the ACK when we do so.