Potential data corruption issue for OCI Java SDK on binary data upload with RefreshableOnNotAuthenticatedProvider

[mricken]

Update 2020-10-28: We believe we have fixed this issue in version 1.25.2. If you are using a RefreshableOnNotAuthenticatedProvider (Instance Principals or Resource Principals, for example) and are uploading streams of binary data, we urge you to update to version 1.25.2.

---

If you are using any of the OCI Java SDK clients that upload streams of data, e.g. ObjectStorageClient or FunctionsInvokeClient (both synchronous and asynchronous), and you use a RefreshableOnNotAuthenticatedProvider (e.g. for Resource Principals or Instance Principals) you may be affected by silent data corruption.

Description

When using a RefreshableOnNotAuthenticatedProvider, the OCI Java SDK will attempt to acquire a new security token when receiving a 401 response, which indicates authentication problems, and then automatically retry the request. In this situation, the clients fail to reset the stream position for requests that upload streams. As a result, an empty stream is successfully uploaded, and success may be returned, even though the actual stream content was not uploaded.

Affected and Not Affected Code

This happens both for synchronous and asynchronous clients, e.g.

• Affected: ObjectStorageClient
• Affected: FunctionsInvokeClient
• Affected: ObjectStorageAsyncClient
• Affected: FunctionsInvokeAsyncClient

It only happens when an authentication details provider that implements RefreshableOnNotAuthenticatedProvider is used, e.g.:

• Affected: InstancePrincipalsAuthenticationDetailsProvider
• Affected: ResourcePrincipalAuthenticationDetailsProvider

It does not happen for other authentication details provider that do not implement said interface, e.g.:

• Not affected: SimpleAuthenticationDetailsProvider
• Not affected: ConfigFileAuthenticationDetailsProvider

It only affects operations that upload streams, e.g.

• ObjectStorageClient.putObject
• FunctionsInvokeAsyncClient.invokeFunction

Operations that do not upload streams are not affected.

The ObjectStorage UploadManager is may be affected, since it uses the synchronous ObjectStorage client.

If the contentLength is set in PutObjectRequest, uploads using ObjectStorageAsyncClient.putObject are not affected.

To summarize, you are affected if you

1. upload streams,
2. and use a RefreshableOnNotAuthenticatedProvider,
3. and do not set the contentLength.

Workarounds

Fixing this issue is our highest priority, and we are actively working on it. In the meantime, here are some possible workarounds:

• Set the contentLength, if the operation supports it (e.g. ObjectStorageClient.putObject). If the client fails to write the exact number of bytes, the operation will fail and data corruption can be avoided.
• Set the contentMD5, if you can compute it and the operation supports it (e.g. ObjectStorageClient.putObject). Then compare the MD5 hash returned in the response to your pre-computed MD5 hash.
• After you successfully upload a stream, download the stream to validate it. If the content is not as expected, repeat the upload. This is only an option for services that provide such a feature, e.g. Object Storage, but not for operations like FunctionsInvokeClient.invokeFunction.
• Immediately before making a request that uploads a stream, call refresh() on the authentication details provider. This will force a refresh of the token, making it nearly certain the next request will succeed (depending on the lifetime of the token; normally, on the order of minutes).

We apologize for any inconvenience this issue may cause you. We are working to promptly resolve this issue. Please watch this GitHub issue for updates.

[mricken]

We believe we have fixed this issue in version 1.25.2: https://github.com/oracle/oci-java-sdk/releases/tag/v1.25.2

If you are using a RefreshableOnNotAuthenticatedProvider (Instance Principals or Resource Principals, for example) and are uploading streams of binary data, we urge you to update to version 1.25.2.