ArrayList payloads lose their elements

[carolmorneau]

Describe the issue Message payloads of type ArrayList lose their elements as they go through the Spring Cloud Stream framework.

For instance, an application configured with:

spring:
  cloud:
    function:
      definition: myFunction
    stream:
      default:
        producer:
          use-native-encoding: true

and a function defined as:

	@Bean
	public Function<Message<?>, Message<?>> myFunction() {
		return msg -> msg;
	}

the output destination receives the Message with an ArrayList payload, however, all elements from the ArrayList have vanished.

It appears that this issue was introduced as part of the following changes in Spring Cloud Function. When I undo the changes from that PR, the issue goes away.

To Reproduce I have uploaded a minimal application reproducing the issue here. Run the unit test called testSpringCloudFunctionIssueWhereArrayListPayloadsLoseTheirElements to reproduce the issue.

Version of the framework Using version 4.1.0 of Spring Cloud Stream and Spring Cloud Function

[INFO] +- org.springframework.cloud:spring-cloud-stream:jar:4.1.0:compile
[ ... TRUNCATED ]
[INFO] |  +- org.springframework.cloud:spring-cloud-function-context:jar:4.1.0:compile
[INFO] |  |  +- net.jodah:typetools:jar:0.6.2:compile
[INFO] |  |  +- org.springframework.boot:spring-boot-autoconfigure:jar:3.2.2:compile
[INFO] |  |  +- org.springframework.cloud:spring-cloud-function-core:jar:4.1.0:compile

Expected behavior With the above function, I would expect to receive the message with complete arrayList payload on the output destination.

[sobychacko]

@carolmorneau When specifying collection types (batch mode), you must use a concrete type in the function signature. You cannot use generics or wildcard types when running in batch mode. If you change your function signature to the following, it will work.

@Bean
public Function<Message<List<String>, Message<List<String>> myFunction() {
  return msg -> msg;
}

You must also enable batch mode on the consumer binding when running with a binder that supports the batch mode (for example, the Kafka binder). You can enable batch-mode using the property - spring.cloud.stream.bindings.myFunction-in-0.consumer.batch-mode and set this to true. You don't need to set this, however, for your unit test.

[carolmorneau]

Thank you @sobychacko for looking into the issue. The issue is not related to Batch Mode, we are processing messages individually.

In our use case, the Spring Message may have different payload types which is why we use Message<?>.

The issue occurs specifically when the Spring Message has a payload of type ArrayList. For instance:

List<String> payload = List.of("aSimpleStringElement");
Message<List<String>> msgWithArrayListPayload = MessageBuilder.withPayload(payload).build();

The above aSimpleStringElement will be lost during this payload conversion.

I was able to pinpoint when the issue was introduced. It appears to be with this change

[sobychacko]

Ok, that appears to be a regression issue in Spring Cloud Function. Transferring the issue over there. cc @olegz

[carolmorneau]

FYI, since this issue was opened, we found two workarounds in order to avoid loosing payload elements:

• Configure consumer's user-native-decoding and producer's use-native-encoding to true. However, this currently only works when function composition isn't used. This essentially does not attempt payload conversion and avoids the issue.
• Provide a user-defined message converter capable of handling payloads of type ArrayList. This ensures that at least one MessageConverter can handle ArrayList payloads and avoids loosing payloads.

Also, I created another reproduction sample app based on the function-composition-rabbit sample to showcase the issue.

Reproduction steps:

• Run the sample app with the issue1112 profile
• Send some data to the RabbitMQ exchange (at least 3 messages since batch-size=3)
• Notice that payloads elements are lost

As a side note, for our specific use-case, https://github.com/spring-cloud/spring-cloud-function/issues/1144 is deemed more important than this issue.

[olegz]

I believe this issue has been addressed with https://github.com/spring-cloud/spring-cloud-function/issues/1145, but i will recheck

[olegz]

I confirm on my end that it was fixed with https://github.com/spring-cloud/spring-cloud-function/issues/1145, but awaiting for your confirmation

[carolmorneau]

The fix on 4.1.2-SNAPSHOT looks good to me.

Thank you @olegz for the fix