Example: read-after-send, APM style

[osalyk]

This example will show it is possible to make data persistent single-sidedly using RECV buffers carved out from PMEM and following rpma_send() with rpma_flush().

SERVER

prepare memory (recv_mr) 
create a new peer 
listening endpoint 
register memory (rpma_mr_reg(USAGE_RECV && USAGE_FLUSH)) 
receive an incoming connection request 
initial receive to be prepared for the first message (rpma_conn_req_recv) 
prepare private_data (recv_mr) 
accept the connection 
wait for the connection to be established 
while (1) { 
        do { 
                prepare completions, get one and validate it 
        } while (!recv_cmpl); 
        if (*recv == I_M_DONE) 
                break; 
        rpma_recv 
} (!recv_cmpl); 
disconnect 

CLIENT

prepare memory (send_mr) 
register memory(RPMA_MR_USAGE_SEND) 
establish a new connection to a server listening at addr:port 
unpack private date 
while (--rounds) { //rounds -argument provided by user 
        rpma_send 
        rpma_flush (mr_region from private_data) 
        do { 
                prepare completions, get one and validate it 
        } while (!send_cmpl ); 
} 
rpma_send (I_M_DONE) 
disconnect 

[ldorau]

LGTM :+1: