ExternalAccountCredentials.fromStream() throws exception on optional fields with null value

[clementdenis]

Steps to reproduce

1. Try to load the following JSON configuration using ExternalAccountCredentials.fromStream()

{
  "service_account_impersonation_url": null, //optional field, would be similar with any other
  "audience": "audience",
  "subject_token_type": "subjectTokenType",
  "credential_source": {"file":"file"}
}

Code example

  ExternalAccountCredentials credentials = ExternalAccountCredentials.fromStream(
          new ByteArrayInputStream(("""
                  {\
                  "service_account_impersonation_url": null,\
                  "audience": "audience",\
                  "subject_token_type": "subjectTokenType",\
                  "credential_source": {"file":"file"}\
                  }""").getBytes()));

Stack trace

com.google.auth.oauth2.CredentialFormatException: An invalid input stream was provided.

	at com.google.auth.oauth2.ExternalAccountCredentials.fromStream(ExternalAccountCredentials.java:398)
	at com.google.auth.oauth2.ExternalAccountCredentials.fromStream(ExternalAccountCredentials.java:366)
	at com.google.auth.oauth2.ExternalAccountCredentialsTest.fromStream_nullOptionalField(ExternalAccountCredentialsTest.java:153)
	at java.base/jdk.internal.reflect.DirectMethodHandleAccessor.invoke(DirectMethodHandleAccessor.java:103)
	at java.base/java.lang.reflect.Method.invoke(Method.java:580)
	at org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:59)
	at org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12)
	at org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:56)
	at org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17)
	at org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:26)
	at org.junit.runners.ParentRunner$3.evaluate(ParentRunner.java:306)
	at org.junit.runners.BlockJUnit4ClassRunner$1.evaluate(BlockJUnit4ClassRunner.java:100)
	at org.junit.runners.ParentRunner.runLeaf(ParentRunner.java:366)
	at org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:103)
	at org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:63)
	at org.junit.runners.ParentRunner$4.run(ParentRunner.java:331)
	at org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:79)
	at org.junit.runners.ParentRunner.runChildren(ParentRunner.java:329)
	at org.junit.runners.ParentRunner.access$100(ParentRunner.java:66)
	at org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:293)
	at org.junit.runners.ParentRunner$3.evaluate(ParentRunner.java:306)
	at org.junit.runners.ParentRunner.run(ParentRunner.java:413)
	at org.junit.runner.JUnitCore.run(JUnitCore.java:137)
	at com.intellij.junit4.JUnit4IdeaTestRunner.startRunnerWithArgs(JUnit4IdeaTestRunner.java:69)
	at com.intellij.rt.junit.IdeaTestRunner$Repeater$1.execute(IdeaTestRunner.java:38)
	at com.intellij.rt.execution.junit.TestsRepeater.repeat(TestsRepeater.java:11)
	at com.intellij.rt.junit.IdeaTestRunner$Repeater.startRunnerWithArgs(IdeaTestRunner.java:35)
	at com.intellij.rt.junit.JUnitStarter.prepareStreamsAndStart(JUnitStarter.java:232)
	at com.intellij.rt.junit.JUnitStarter.main(JUnitStarter.java:55)
Caused by: java.lang.ClassCastException: class java.lang.Object cannot be cast to class java.lang.String (java.lang.Object and java.lang.String are in module java.base of loader 'bootstrap')
	at com.google.auth.oauth2.ExternalAccountCredentials.fromJson(ExternalAccountCredentials.java:422)
	at com.google.auth.oauth2.ExternalAccountCredentials.fromStream(ExternalAccountCredentials.java:396)
	... 28 more

Any additional information below

This is caused by com.google.api.client.json.JsonParser used to parse the JSON payload. This implementation does not deserializes null JSON value to null reference in Java, instead using magic null objects from com.google.api.client.util.Data => this causes this ClassCastException when trying to deserialize, as the magic object can't be cast to the expected field type.

The current behavior does not follow the principle of least astonishment, that be that a null value for a specific optional field in JSON configuration is just considered as missing.

This issue is likely to happen for other types of JSON credential configuration that have optional values.

[lqiu96]

Thanks for raising this issue. I do see that from our docs that we do have a few optional parameters. For my understanding, would you help clarify your use case/ scenario where certain optional fields (i.e. service_account_impersonation_url) come in as null as opposed to being omitted in the file/ stream?

I'm looking at the current implementation and I think we have previously made the assumption that these values would have been omitted in the file.

[clementdenis]

The use case is to have more generic IaC code, regardless of the presence / absence of optional fields: as Terraform does not support leaving out null field when serializing to JSON, the optional fields will be present with null values when we write the credential files.