Unexpected IllegalArgumentException while sending multicast message

[ayhankaraa]

Environment

• Operating System version: *
• Firebase SDK version: 7.1.0
• Library version: *
• Firebase Product: Firebase Cloud Messaging

[REQUIRED] Step 3: Describe the problem

While using sendMulticast method, it fails with unexpected IllegalArgumentException.

Steps to reproduce:

Cannot reproduce the same error in my local environment while sending exactly same messages which failed with IllegalArgumentException. I've used dryRun true option while sending, and got a successful response.

After restarting the server everything looks OK for some time (maybe days). These made me think there might be an issue about thread safety in SDK. There is also another issue (#506 ) mentioning errors stopped after restarting.

Here is the stack trace:

"stack_trace": "java.lang.IllegalArgumentException: expected primitive class, but got: java.util.List<java.lang.String>\n\tat com.google.api.client.util.Data.parsePrimitiveValue(Data.java:467)\n\tat com.google.api.client.http.HttpHeaders.parseValue(HttpHeaders.java:1153)\n\tat com.google.api.client.http.HttpHeaders.parseHeader(HttpHeaders.java:1137)\n\tat com.google.api.client.http.HttpHeaders.fromHttpResponse(HttpHeaders.java:964)\n\tat com.google.api.client.http.HttpResponse.(HttpResponse.java:144)\n\tat com.google.api.client.http.HttpRequest.execute(HttpRequest.java:1019)\n\tat c.google.api.client.googleapis.batch.BatchRequest.execute(BatchRequest.java:257)\n\tat c.g.firebase.messaging.FirebaseMessagingClientImpl.sendBatchRequest(FirebaseMessagingClientImpl.java:138)\n\tat c.g.firebase.messaging.FirebaseMessagingClientImpl.sendAll(FirebaseMessagingClientImpl.java:118)\n\tat com.google.firebase.messaging.FirebaseMessaging$2.execute(FirebaseMessaging.java:289)\n\tat com.google.firebase.messaging.FirebaseMessaging$2.execute(FirebaseMessaging.java:286)\n\tat com.google.firebase.internal.CallableOperation.call(CallableOperation.java:36)\n\tat com.google.firebase.messaging.FirebaseMessaging.sendAll(FirebaseMessaging.java:179)\n\tat com.google.firebase.messaging.FirebaseMessaging.sendMulticast(FirebaseMessaging.java:248)\n\tat com.google.firebase.messaging.FirebaseMessaging.sendMulticast(FirebaseMessaging.java:223)\n\tat

Relevant Code:

This is the request I've used while trying to reproduce the problem (I have masked the data for security reasons):

    MulticastMessage.builder()
                    .putAllData(Map.of("key1", "value1",
                                       "key2", "value2",
                                       "key3", "value3",
                                       "key4", "value4", "key5", "value5"))
                    .setNotification(Notification.builder().setTitle("title").setBody("body").build())
                    .addAllTokens(List.of("token1", "token2", "token3", "token4", "token5"))
                    .build();
    FirebaseMessaging.getInstance().sendMulticast(message, true);

Response while retrying:

{
  "responses": [
    {
      "messageId": "projects/***/messages/fake_message_id",
      "exception": null,
      "successful": true
    },
    {
      "messageId": "projects/***/messages/fake_message_id",
      "exception": null,
      "successful": true
    },
    {
      "messageId": "projects/***/messages/fake_message_id",
      "exception": null,
      "successful": true
    },
    {
      "messageId": "projects/***/messages/fake_message_id",
      "exception": null,
      "successful": true
    },
    {
      "messageId": "projects/***/messages/fake_message_id",
      "exception": null,
      "successful": true
    }
  ],
  "successCount": 5,
  "failureCount": 0
}

[google-oss-bot]

I found a few problems with this issue:

• I couldn't figure out how to label this issue, so I've labeled it for a human to triage. Hang tight.
• This issue does not seem to follow the issue template. Make sure you provide all the required information.

[hiranya911]

Your code is single-threaded (and the SDK doesn't spawn other threads unless you use the async APIs). It seems to be a problem with parsing an HTTP header. That's quite different from #506, which was due to an unexpected error body. Also in that case the problem was in our code, whereas here the problem occurs in the google-http-java-client library.

@chong-shao any idea what can be causing this?

[Nohus]

I'm also experiencing this issue in production, but cannot reproduce locally. I'm hitting this exception every few days. For me it happened 29 times since 4 months ago (across millions of sent messages). Stack trace is the same as in the original report.

[chong-shao]

Hi @Nohus @ayhankarasm could you provide debugging logs on your side? In addition to the stacktrace, it would be great if we can get to know what's the input that triggers the error in google-http-java-client library. Thanks!

[Nohus]

@chong-shao Happy to do so, but how to gather the input? Is there a way enable some verbose logging in the library?

[chong-shao]

Hi @Nohus this blog post on logging https://hiranya911.medium.com/logging-in-java-libraries-for-firebase-and-google-cloud-platform-f8742493b73f would be helpful. The recommendation is to use SLF4J.

[Nohus]

@chong-shao I've managed to capture logs from the google-http-java-client library. Here are the last 200 log messages when the exception was thrown:

https://gist.github.com/Nohus/0d7383f97b371a2357b821fec4298792

Many requests are sent concurrently, so a given request/response pair may have other requests/responses in-between, and some requests/responses may have been logged in the few milliseconds between the exception was caught and the logs saved (the bad response may not necessarily be the last one).

[Nohus]

I am still hitting getting this failures.

[lahirumaramba]

Closing due to inactivity. We can reopen later if needed.