RATIS-1291. Send heartbeat when there is no reply

[runzhiwang]

What changes were proposed in this pull request?

Currently, when there always exist log, leader only send log, and will not send heartbeat. The problem is if follower need a long time to process log, leader maybe receives response from follower after a long time.

  default long getHeartbeatRemainingTimeMs() {
    return getServer().properties().minRpcTimeoutMs()/2 - getFollower().getLastRpcTime().elapsedTimeMs();
  }

In this pr, I record lastRpcSendTimeWithResponse, i.e. when leader receive response of request1, leader record the send time of request1. Besides, I record lastHeartBeatSendTime, i.e. the send time of heartbeat.

Use this two timestamp, leader send heartbeat when can not receive response after minRpcTimeoutMs()/2. If log cost a short time from request to reply, leader also does not need to send heartbeat.

  default long getHeartbeatRemainingTimeMs() {
    return getServer().properties().minRpcTimeoutMs()/2 -
       Math.min(getFollower().getLastRpcSendTimeWithResponse().elapsedTimeMs(),
           getFollower().getLastHeartBeatSendTime().elapsedTimeMs());
  }

What is the link to the Apache JIRA

https://issues.apache.org/jira/browse/RATIS-1291

How was this patch tested?

no need new ut

[runzhiwang]

will fix the failed ut.

[amaliujia]

cc @amaliujia I can help review when the UT is fixed.