transactions lost when stop 3 of 4 peers, invoke, restart all 3 peers

[scottz64]

Description

The network loses any Invoke requests sent to the single running peer node, after stopping the other 3 peers of a 4 node network. The problem is observed in both testcase variations when the VP0 is the only running node, and when VP3 is the only running node.

The problem is observed ONLY when we restart the other 3 nodes ALL AT THE SAME TIME. If we stop 3, send invokes, and restart only 2 (so there are exactly 3 peers running), then the transactions are processed ok (as expected).

Describe How to Reproduce

Set up a 4 peer network, pbft batch (batchsize 2) with security using docker containers in vagrant environment, running chaincode-example02. Testcase CAT_25_S0S1S2_IQ_R0R1R2_IQ.go was executed using IMAGE=5efed61. See attachments for logs and details. lose_invokes_when_stop012_invoke_restart012.tar.gz

1. Start 4 peers running
2. invokes
3. query
4. stop 3 peers
5. invokes 50 to the only running peer
6. restart 3 peers, so all 4 are running again
7. invokes, queries, sleeps
8. confirm the 50 invokes are never processed by examining chainheight and query results from all 4 peers

[scottz64]

The issue also can be reproduced using a later image, from July 19th: IMAGE=3e0e80a. Both this and the image mentioned previously are on the branch v0.5-developer-preview.

[corecode]

Transactions can be lost.

[scottz64]

Running a similar testcase, CAT_24_S0S1S2_IQ_R0R1_IQIQ.go, was successful, with no transactions lost. In that one, I restarted only 2 peers instead of all three at once.

[christo4ferris]

@scottz64 we've migrated to using Jira for issue tracking, please open an issue there and close this one. You'll need a Linux Foundation ID to login. Thanks