No serializer found for class java.io.ByteArrayInputStream on OpenAiAudioTranscriptionModel.call

[clivelewis]

Bug description Exception is thrown when attempting to send a request to OpenAI transcription service. Please note that the same logic was working approximately 1-2 months ago.

Environment Kotlin + Java 21 Spring Boot 3.3.1 Spring AI 1.0.0-M1 (Also tried on 1.0.0-SNAPSHOT)

Steps to reproduce Simply execute .call method from OpenAiAudioTranscriptionModel with a ByteArrayResource set in the Prompt. Here's my method:

fun transcribeAudio(audio: ByteArray): String {

        val resource = ByteArrayResource(audio)
        val options = OpenAiAudioTranscriptionOptions.builder()
            .withResponseFormat(OpenAiAudioApi.TranscriptResponseFormat.TEXT)
            .build()
        val prompt = AudioTranscriptionPrompt(resource, options)
        val response = client.call(prompt)
        val output = response.result.output
        log.info { "[Transcription] Result - ${response.result}" }
        return output
    }

Expected behavior No exception. Executed method returns transcribed audio as text.

Some of the things that I tried to fix the issue

• Removed all custom beans that could affect the serialization - ObjectMapper and MappingJackson2HttpMessageConverter
• Custom ObjectMapper bean with disabled SerializationFeature.FAIL_ON_EMPTY_BEANS

@Bean
    fun objectMapper(): ObjectMapper {
        return ObjectMapper().apply {
            disable(SerializationFeature.FAIL_ON_EMPTY_BEANS);
        }
    }

• Custom MappingJackson2HttpMessageConverter bean

    @Bean
    fun mappingJackson2HttpMessageConverter(objectMapper: ObjectMapper): MappingJackson2HttpMessageConverter {
        objectMapper.configure(SerializationFeature.FAIL_ON_EMPTY_BEANS, false)
        val converter = MappingJackson2HttpMessageConverter(objectMapper)
        return converter
    }

Stack trace

org.springframework.http.converter.HttpMessageConversionException: Type definition error: [simple type, class java.io.ByteArrayInputStream]
	at org.springframework.http.converter.json.AbstractJackson2HttpMessageConverter.writeInternal(AbstractJackson2HttpMessageConverter.java:489)
	at org.springframework.http.converter.AbstractGenericHttpMessageConverter$1.writeTo(AbstractGenericHttpMessageConverter.java:94)
	at org.springframework.http.client.HttpComponentsClientHttpRequest$BodyEntity.writeTo(HttpComponentsClientHttpRequest.java:155)
	at org.apache.hc.core5.http.impl.io.DefaultBHttpClientConnection.sendRequestEntity(DefaultBHttpClientConnection.java:253)
	at org.apache.hc.core5.http.impl.io.HttpRequestExecutor.execute(HttpRequestExecutor.java:141)
	at org.apache.hc.core5.http.impl.io.HttpRequestExecutor.execute(HttpRequestExecutor.java:218)
	at org.apache.hc.client5.http.impl.io.PoolingHttpClientConnectionManager$InternalConnectionEndpoint.execute(PoolingHttpClientConnectionManager.java:717)
	at org.apache.hc.client5.http.impl.classic.InternalExecRuntime.execute(InternalExecRuntime.java:216)
	at org.apache.hc.client5.http.impl.classic.MainClientExec.execute(MainClientExec.java:116)
	at org.apache.hc.client5.http.impl.classic.ExecChainElement.execute(ExecChainElement.java:51)
	at org.apache.hc.client5.http.impl.classic.ConnectExec.execute(ConnectExec.java:188)
	at org.apache.hc.client5.http.impl.classic.ExecChainElement.execute(ExecChainElement.java:51)
	at org.apache.hc.client5.http.impl.classic.ProtocolExec.execute(ProtocolExec.java:192)
	at org.apache.hc.client5.http.impl.classic.ExecChainElement.execute(ExecChainElement.java:51)
	at org.apache.hc.client5.http.impl.classic.HttpRequestRetryExec.execute(HttpRequestRetryExec.java:113)
	at org.apache.hc.client5.http.impl.classic.ExecChainElement.execute(ExecChainElement.java:51)
	at org.apache.hc.client5.http.impl.classic.ContentCompressionExec.execute(ContentCompressionExec.java:152)
	at org.apache.hc.client5.http.impl.classic.ExecChainElement.execute(ExecChainElement.java:51)
	at org.apache.hc.client5.http.impl.classic.RedirectExec.execute(RedirectExec.java:116)
	at org.apache.hc.client5.http.impl.classic.ExecChainElement.execute(ExecChainElement.java:51)
	at org.apache.hc.client5.http.impl.classic.InternalHttpClient.doExecute(InternalHttpClient.java:170)
	at org.apache.hc.client5.http.impl.classic.CloseableHttpClient.execute(CloseableHttpClient.java:87)
	at org.apache.hc.client5.http.impl.classic.CloseableHttpClient.execute(CloseableHttpClient.java:55)
	at org.apache.hc.client5.http.classic.HttpClient.executeOpen(HttpClient.java:183)
	at org.springframework.http.client.HttpComponentsClientHttpRequest.executeInternal(HttpComponentsClientHttpRequest.java:99)
	at org.springframework.http.client.AbstractStreamingClientHttpRequest.executeInternal(AbstractStreamingClientHttpRequest.java:70)
	at org.springframework.http.client.AbstractClientHttpRequest.execute(AbstractClientHttpRequest.java:66)
	at org.springframework.web.client.DefaultRestClient$DefaultRequestBodyUriSpec.exchangeInternal(DefaultRestClient.java:492)
	at org.springframework.web.client.DefaultRestClient$DefaultRequestBodyUriSpec.retrieve(DefaultRestClient.java:460)
	at org.springframework.ai.openai.api.OpenAiAudioApi.createTranscription(OpenAiAudioApi.java:670)
	at org.springframework.ai.openai.OpenAiAudioTranscriptionModel.lambda$call$0(OpenAiAudioTranscriptionModel.java:153)
	at org.springframework.retry.support.RetryTemplate.doExecute(RetryTemplate.java:344)
	at org.springframework.retry.support.RetryTemplate.execute(RetryTemplate.java:217)
	at org.springframework.ai.openai.OpenAiAudioTranscriptionModel.call(OpenAiAudioTranscriptionModel.java:124)
	at io.github.clivelewis.assistant.chat.impl.openai.OpenAiTranscriptionService.transcribeAudio(OpenAiTranscriptionService.kt:24)
	at io.github.clivelewis.assistant.telegram.commands.ConversationCommand.transcribeVoiceMessage(ConversationCommand.kt:187)
	at io.github.clivelewis.assistant.telegram.commands.ConversationCommand.access$transcribeVoiceMessage(ConversationCommand.kt:28)
	at io.github.clivelewis.assistant.telegram.commands.ConversationCommand$transcribeVoiceMessage$1.invokeSuspend(ConversationCommand.kt)
	at kotlin.coroutines.jvm.internal.BaseContinuationImpl.resumeWith(ContinuationImpl.kt:33)
	at kotlinx.coroutines.DispatchedTask.run(DispatchedTask.kt:104)
	at kotlinx.coroutines.internal.LimitedDispatcher$Worker.run(LimitedDispatcher.kt:111)
	at kotlinx.coroutines.scheduling.TaskImpl.run(Tasks.kt:99)
	at kotlinx.coroutines.scheduling.CoroutineScheduler.runSafely(CoroutineScheduler.kt:584)
	at kotlinx.coroutines.scheduling.CoroutineScheduler$Worker.executeTask(CoroutineScheduler.kt:811)
	at kotlinx.coroutines.scheduling.CoroutineScheduler$Worker.runWorker(CoroutineScheduler.kt:715)
	at kotlinx.coroutines.scheduling.CoroutineScheduler$Worker.run(CoroutineScheduler.kt:702)
Caused by: com.fasterxml.jackson.databind.exc.InvalidDefinitionException: No serializer found for class java.io.ByteArrayInputStream and no properties discovered to create BeanSerializer (to avoid exception, disable SerializationFeature.FAIL_ON_EMPTY_BEANS) (through reference chain: org.springframework.util.LinkedMultiValueMap["file"]->java.util.ArrayList[0]->org.springframework.ai.openai.api.OpenAiAudioApi$1["inputStream"])
	at com.fasterxml.jackson.databind.exc.InvalidDefinitionException.from(InvalidDefinitionException.java:77)
	at com.fasterxml.jackson.databind.SerializerProvider.reportBadDefinition(SerializerProvider.java:1330)
	at com.fasterxml.jackson.databind.DatabindContext.reportBadDefinition(DatabindContext.java:414)
	at com.fasterxml.jackson.databind.ser.impl.UnknownSerializer.failForEmpty(UnknownSerializer.java:53)
	at com.fasterxml.jackson.databind.ser.impl.UnknownSerializer.serialize(UnknownSerializer.java:30)
	at com.fasterxml.jackson.databind.ser.BeanPropertyWriter.serializeAsField(BeanPropertyWriter.java:732)
	at com.fasterxml.jackson.databind.ser.std.BeanSerializerBase.serializeFields(BeanSerializerBase.java:770)
	at com.fasterxml.jackson.databind.ser.BeanSerializer.serialize(BeanSerializer.java:183)
	at com.fasterxml.jackson.databind.ser.impl.IndexedListSerializer.serializeContents(IndexedListSerializer.java:119)
	at com.fasterxml.jackson.databind.ser.impl.IndexedListSerializer.serialize(IndexedListSerializer.java:79)
	at com.fasterxml.jackson.databind.ser.impl.IndexedListSerializer.serialize(IndexedListSerializer.java:18)
	at com.fasterxml.jackson.databind.ser.std.MapSerializer.serializeFields(MapSerializer.java:808)
	at com.fasterxml.jackson.databind.ser.std.MapSerializer.serializeWithoutTypeInfo(MapSerializer.java:764)
	at com.fasterxml.jackson.databind.ser.std.MapSerializer.serialize(MapSerializer.java:720)
	at com.fasterxml.jackson.databind.ser.std.MapSerializer.serialize(MapSerializer.java:35)
	at com.fasterxml.jackson.databind.ser.DefaultSerializerProvider._serialize(DefaultSerializerProvider.java:502)
	at com.fasterxml.jackson.databind.ser.DefaultSerializerProvider.serializeValue(DefaultSerializerProvider.java:422)
	at com.fasterxml.jackson.databind.ObjectWriter$Prefetch.serialize(ObjectWriter.java:1570)
	at com.fasterxml.jackson.databind.ObjectWriter.writeValue(ObjectWriter.java:1061)
	at org.springframework.http.converter.json.AbstractJackson2HttpMessageConverter.writeInternal(AbstractJackson2HttpMessageConverter.java:483)
	... 45 common frames omitted