[Improvement] Disallow sendShuffleData if requireBufferId expired

[xianjingfeng]

We found shuffle server which under high load is easy encounter java.lang.OutOfMemoryError: Java heap space even we allocate more jvm heap memory and less rss.server.buffer.capacity

The steps for the exception above:

1. When shuffle server under high load, requireBufferId is easy to expire, and Shuffle server release usedMemory
2. Client sendShuffleData using a expired requireBufferId,
3. Shuffle server recive shuffle data and store in rpc queue(this part of memory usage was not be added to usedMemory)
4. Other clients requireBuffer success because usedMemory is enough

[jerqi]

Do you have the solution of the problem?

[xianjingfeng]

Yes, it is be testing in our production environment. I will watch it for a while. If it's OK, I will create a pr

[jerqi]

Could you share your solution? We can discuss first.

[xianjingfeng]

Could you share your solution? We can discuss first.

1. In server side, if requireBufferId not found when send data, thrown an exception.
2. In client side, if fail to send data, require buffer again.

[jerqi]

cc @colinmjj . There seems not be cases in our production environment. But I think the analysis is correct. What do you think?

[colinmjj]

I think @xianjingfeng is right, with current implementation, OOM will happen if requireBufferId was expired in Shuffle Server already, this maybe caused by GC, network problem, high workload in shuffle server etc. It's better to have the limitation to accept the data with requireBufferId only to avoid such problem.

[jerqi]

closed by #157