BlobStore may not be respecting size limit in findEntriesSince()

[lightningrob]

The heap dump of an OOM storage node revealed a log message from AmbryRequests.handleReplicaMetadataRequest() that was over 1GB. The log line in question is: publicAccessLogger.info("{} {} processingTime {}", replicaMetadataRequest, response, processingTime); And the ReplicaMetadataResponse object apparently has a huge list.

We will add some logging/metrics to track when this happens and debug it.

java.lang.OutOfMemoryError: Java heap space
    at sun.nio.cs.StreamEncoder.write(StreamEncoder.java:133)
    at java.io.OutputStreamWriter.write(OutputStreamWriter.java:220)
    at java.io.Writer.write(Writer.java:157)
    at org.apache.log4j.helpers.QuietWriter.write(QuietWriter.java:48)
    at org.apache.log4j.WriterAppender.subAppend(WriterAppender.java:310)
    at org.apache.log4j.DailyRollingFileAppender.subAppend(DailyRollingFileAppender.java:369)
    at org.apache.log4j.WriterAppender.append(WriterAppender.java:162)
    at org.apache.log4j.AppenderSkeleton.doAppend(AppenderSkeleton.java:251)
    at org.apache.log4j.helpers.AppenderAttachableImpl.appendLoopOnAppenders(AppenderAttachableImpl.java:66)
    at org.apache.log4j.Category.callAppenders(Category.java:206)
    at org.apache.log4j.Category.forcedLog(Category.java:391)
    at org.apache.log4j.Category.log(Category.java:856)
    at org.slf4j.impl.Log4jLoggerAdapter.info(Log4jLoggerAdapter.java:363)
    at com.github.ambry.server.AmbryRequests.handleReplicaMetadataRequest(AmbryRequests.java:647)
    at com.github.ambry.server.AmbryRequests.handleRequests(AmbryRequests.java:169)
    at com.github.ambry.server.RequestHandler.run(RequestHandler.java:46)
    at java.lang.Thread.run(Thread.java:745)```