ApacheHttp2Transport connection did not return to the pool when it is failed

[meor9zcu12]

Firebase Admin Java SDK v9.6.0

FirebaseOptions.Builder builder = FirebaseOptions.builder()
...
.setHttpTransport(new ApacheHttp2Transport(httpClient));

ApiFuture<BatchResponse> futures = firebaseMessaging.sendEachAsync(messages, dryRun);
BatchResponse response;
try {
      response = futures.get(TIME_OUT_MS, TimeUnit.MILLISECONDS);
 } catch (TimeoutException e) {
   ...
}

final PoolStats total = poolingAsyncClientConnectionManager.getTotalStats();
return String.format("Available=%d, leased=%d, pending=%d, max=%d",
                total.getAvailable(),
                total.getLeased(),
                total.getPending(),
                total.getMax());

Observation: Under very busy network, FCM backend may return Unknown error while making a remote service call: Write Timeout or other errors.

Log writing (successful or failed response) become slow and slower. Afterward, all pending requests were timed out at response = futures.get(TIME_OUT_MS, TimeUnit.MILLISECONDS);

When the program just started:
Available=95, leased=5, pending=0, max=200

After some times:
Available=0, leased=0, pending=0, max=200

---

Update:

When using Google APIs Transports: There is disconnect implementation in google-http-java-client

However, when using ApacheHttp2Transport: There is no such implementation in ApacheHttp2Response when it is disconnected, no matter the response is successfully or failed.

[jonathanedey]

Hi @meor9zcu12, I wasn't able to reproduce this on my end.

Based on my understanding, the ApacheHttp2Response doesn't require a disconnect configuration since the connections are automatically released by the PoolingAsyncClientConnectionManager. The pool stats you provided seem to suggest that the connections are being closed or deleted and are not replaced as expected. I believe that if the connections were being held by the missing disconnect implementation in some way then these would be counted as still leased.

Could you provide some additional context to help debug this issue:

1. Are you using either a custom CloseableHttpAsyncClient implementation or PoolingAsyncClientConnectionManager config?
2. Do you have read, write or connection timeouts set on FirebaseOptions?
3. How many messages are being sent when connections begin to disappear?

Additionally, if possible could you provide a minimum reproducible snippet that we could use to recreate this bug?