koog icon indicating copy to clipboard operation
koog copied to clipboard
Published 2 weeks ago • verified publisher icon JetBrains

Add capabilities integration tests and fix AbstractOpenAILLMClient for plain text

Open skarpovdev opened this issue 5 months ago • 1 comments

Tests

  1. Positive - check the model actually has specified capabilities by trying to use them.
  2. Negative - check that trying to use a capability that a model doesn't have results in an error.

Capabilities like Speculation and Temperature are excluded since there is no actual way to verify them.

Bugfix

It appears there is a bug in AbstractOpenAILLMClient which caused the plain text file attachment to fail with Unsupported file attachment content: class ai.koog.prompt.message.AttachmentContent$PlainText because it supports only Binary and throws for everything else. Attempting to send file text as a file:

is AttachmentContent.PlainText -> {
    val fileData = OpenAIContentPart.FileData(
        fileData = "data:$mimeType;text,${attachmentContent.text}",
        filename = fileName
    )
    OpenAIContentPart.File(fileData)
}

results in error like this:

[DefaultDispatcher-worker-3 @coroutine#23] ERROR ai.koog.prompt.executor.clients.openai.OpenAILLMClient - Error from OpenAILLMClient API: 400 Bad Request: {
  "error": {
    "message": "Invalid file data: 'messages[1].content[1].file.file_data'. Expected a base64-encoded data URL with an application/pdf MIME type (e.g. 'data:application/pdf;base64,SGVsbG8sIFdvcmxkIQ=='), but got unsupported MIME type 'text/plain'.",
    "type": "invalid_request_error",
    "param": "messages[1].content[1].file.file_data",
    "code": "invalid_value"
  }
}

so the solution is to send it as OpenAIContentPart.Text. Let me know what you think.

Refactoring

I've put executor, agent, and capabilities tests into corresponding packages to simplify navigation in integration tests.

Type of the change

  • [ ] New feature
  • [x] Bug fix
  • [ ] Documentation fix
  • [x] Tests improvement
  • [x] Refactoring

Checklist for all pull requests

  • [ ] The pull request has a description of the proposed change
  • [ ] I read the Contributing Guidelines before opening the pull request
  • [ ] The pull request uses develop as the base branch
  • [ ] Tests for the changes have been added
  • [ ] All new and existing tests passed
Additional steps for pull requests adding a new feature
  • [ ] An issue describing the proposed change exists
  • [ ] The pull request includes a link to the issue
  • [ ] The change was discussed and approved in the issue
  • [ ] Docs have been added / updated

skarpovdev avatar Aug 10 '25 16:08 skarpovdev

Qodana for JVM

772 new problems were found

Inspection name Severity Problems
Check Kotlin and Java source code coverage 🔶 Warning 741
Missing KDoc for public API declaration 🔶 Warning 24
Vulnerable imported dependency 🔶 Warning 6
String concatenation that can be converted to string template ◽️ Notice 1 
@@ Code coverage @@
+ 66% total lines covered
10842 lines analyzed, 7231 lines covered
# Calculated according to the filters of your coverage tool

☁️ View the detailed Qodana report

Detected 207 dependencies

Third-party software list

This page lists the third-party software dependencies used in koog-agents

Dependency Version Licenses
annotations-js 26.0.2 Apache-2.0
annotations-wasmjs 26.0.2 Apache-2.0
annotations 13.0 Apache-2.0
annotations 23.0.0 Apache-2.0
annotations 26.0.2 Apache-2.0
atomicfu-js 0.26.1 Apache-2.0
atomicfu-js 0.28.0 Apache-2.0
atomicfu-wasm-js 0.26.1 Apache-2.0
atomicfu-wasm-js 0.28.0 Apache-2.0
atomicfu 0.23.1 Apache-2.0
aws-config-jvm 1.5.16 Apache-2.0
aws-core-jvm 1.5.16 Apache-2.0
aws-credentials-jvm 1.5.4 Apache-2.0
aws-endpoint-jvm 1.5.16 Apache-2.0
aws-signing-common-jvm 1.5.4 Apache-2.0
bedrockruntime-jvm 1.5.16 Apache-2.0
config 1.4.3 Apache-2.0
dokka-core 2.0.0 Apache-2.0
dokka-gradle-plugin 2.0.0 Apache-2.0
fus-statistics-gradle-plugin 2.1.21 Apache-2.0
http-auth-api-jvm 1.5.4 Apache-2.0
http-auth-jvm 1.5.4 Apache-2.0
http-client-jvm 1.5.4 Apache-2.0
http-jvm 1.5.4 Apache-2.0
identity-api-jvm 1.5.4 Apache-2.0
jackson-annotations 2.12.7 Apache-2.0
jackson-annotations 2.19.1 Apache-2.0
jackson-core 2.12.7 Apache-2.0
jackson-core 2.19.1 Apache-2.0
jackson-databind 2.12.7.1 Apache-2.0
jackson-databind 2.19.1 Apache-2.0
jackson-dataformat-xml 2.12.7 Apache-2.0
jackson-module-jaxb-annotations 2.12.7 Apache-2.0
jackson-module-kotlin 2.12.7 Apache-2.0
jackson-module-kotlin 2.19.1 Apache-2.0
jakarta.activation-api 1.2.1 BSD-3-Clause
jakarta.annotation-api 2.1.1 Classpath-exception-2.0
EPL-2.0
GPL-2.0-only
jakarta.xml.bind-api 2.3.2 BSD-3-Clause
jet-sign 45.47 Apache-2.0
jul-to-slf4j 2.0.17 MIT
kotlin-dom-api-compat 2.1.21 Apache-2.0
kotlin-gradle-plugin-api 2.1.21 Apache-2.0
kotlin-gradle-plugin-model 2.1.21 Apache-2.0
kotlin-gradle-plugin 2.1.21 Apache-2.0
kotlin-logging-js 7.0.7 Apache-2.0
kotlin-logging-jvm 7.0.7 Apache-2.0
kotlin-logging-wasm-js 7.0.7 Apache-2.0
kotlin-logging 7.0.7 Apache-2.0
kotlin-reflect 2.0.21 Apache-2.0
kotlin-reflect 2.1.21 Apache-2.0
kotlin-sdk-jvm 0.6.0 MIT
kotlin-stdlib-js 2.1.21 Apache-2.0
kotlin-stdlib-wasm-js 2.1.21 Apache-2.0
kotlin-stdlib 2.0.21 Apache-2.0
kotlin-stdlib 2.1.21 Apache-2.0
kotlinx-collections-immutable-jvm 0.4.0 Apache-2.0
kotlinx-coroutines-core-js 1.10.2 Apache-2.0
kotlinx-coroutines-core-jvm 1.10.2 Apache-2.0
kotlinx-coroutines-core-jvm 1.8.0 Apache-2.0
kotlinx-coroutines-core-wasm-js 1.10.2 Apache-2.0
kotlinx-coroutines-core 1.10.2 Apache-2.0
kotlinx-coroutines-jdk8 1.10.2 Apache-2.0
kotlinx-coroutines-reactive 1.10.2 Apache-2.0
kotlinx-coroutines-slf4j 1.10.2 Apache-2.0
kotlinx-datetime-js 0.6.2 Apache-2.0
kotlinx-datetime-jvm 0.6.2 Apache-2.0
kotlinx-datetime-wasm-js 0.6.2 Apache-2.0
kotlinx-datetime 0.6.2 Apache-2.0
kotlinx-io-bytestring-js 0.7.0 Apache-2.0
kotlinx-io-bytestring-jvm 0.7.0 Apache-2.0
kotlinx-io-bytestring-wasm-js 0.7.0 Apache-2.0
kotlinx-io-bytestring 0.7.0 Apache-2.0
kotlinx-io-core-js 0.7.0 Apache-2.0
kotlinx-io-core-jvm 0.7.0 Apache-2.0
kotlinx-io-core-wasm-js 0.7.0 Apache-2.0
kotlinx-io-core 0.7.0 Apache-2.0
kotlinx-serialization-core-js 1.8.1 Apache-2.0
kotlinx-serialization-core-jvm 1.8.1 Apache-2.0
kotlinx-serialization-core-jvm 1.9.0 Apache-2.0
kotlinx-serialization-core-wasm-js 1.8.1 Apache-2.0
kotlinx-serialization-core 1.8.1 Apache-2.0
kotlinx-serialization-json-io-js 1.8.1 Apache-2.0
kotlinx-serialization-json-io-jvm 1.8.1 Apache-2.0
kotlinx-serialization-json-io-jvm 1.9.0 Apache-2.0
kotlinx-serialization-json-io-wasm-js 1.8.1 Apache-2.0
kotlinx-serialization-json-io 1.8.1 Apache-2.0
kotlinx-serialization-json-js 1.8.1 Apache-2.0
kotlinx-serialization-json-jvm 1.8.1 Apache-2.0
kotlinx-serialization-json-jvm 1.9.0 Apache-2.0
kotlinx-serialization-json-wasm-js 1.8.1 Apache-2.0
kotlinx-serialization-json 1.8.1 Apache-2.0
ktor-client-cio-js 3.2.2 Apache-2.0
ktor-client-cio-jvm 3.2.2 Apache-2.0
ktor-client-cio-wasm-js 3.2.2 Apache-2.0
ktor-client-cio 3.2.2 Apache-2.0
ktor-client-content-negotiation-js 3.2.2 Apache-2.0
ktor-client-content-negotiation-jvm 3.2.2 Apache-2.0
ktor-client-content-negotiation-wasm-js 3.2.2 Apache-2.0
ktor-client-content-negotiation 3.2.2 Apache-2.0
ktor-client-core-js 3.2.2 Apache-2.0
ktor-client-core-jvm 3.2.2 Apache-2.0
ktor-client-core-wasm-js 3.2.2 Apache-2.0
ktor-client-core 3.2.2 Apache-2.0
ktor-client-darwin 3.2.2 Apache-2.0
ktor-client-js-js 3.2.2 Apache-2.0
ktor-client-logging-js 3.2.2 Apache-2.0
ktor-client-logging-jvm 3.2.2 Apache-2.0
ktor-client-logging-wasm-js 3.2.2 Apache-2.0
ktor-client-logging 3.2.2 Apache-2.0
ktor-events-js 3.2.2 Apache-2.0
ktor-events-jvm 3.2.2 Apache-2.0
ktor-events-wasm-js 3.2.2 Apache-2.0
ktor-events 3.2.2 Apache-2.0
ktor-http-cio-js 3.2.2 Apache-2.0
ktor-http-cio-jvm 3.2.2 Apache-2.0
ktor-http-cio-wasm-js 3.2.2 Apache-2.0
ktor-http-cio 3.2.2 Apache-2.0
ktor-http-js 3.2.2 Apache-2.0
ktor-http-jvm 3.2.2 Apache-2.0
ktor-http-wasm-js 3.2.2 Apache-2.0
ktor-http 3.2.2 Apache-2.0
ktor-io-js 3.2.2 Apache-2.0
ktor-io-jvm 3.2.2 Apache-2.0
ktor-io-wasm-js 3.2.2 Apache-2.0
ktor-io 3.2.2 Apache-2.0
ktor-network-js 3.2.2 Apache-2.0
ktor-network-jvm 3.2.2 Apache-2.0
ktor-network-tls-js 3.2.2 Apache-2.0
ktor-network-tls-jvm 3.2.2 Apache-2.0
ktor-network-tls-wasm-js 3.2.2 Apache-2.0
ktor-network-tls 3.2.2 Apache-2.0
ktor-network-wasm-js 3.2.2 Apache-2.0
ktor-network 3.2.2 Apache-2.0
ktor-serialization-js 3.2.2 Apache-2.0
ktor-serialization-jvm 3.2.2 Apache-2.0
ktor-serialization-kotlinx-js 3.2.2 Apache-2.0
ktor-serialization-kotlinx-json-js 3.2.2 Apache-2.0
ktor-serialization-kotlinx-json-jvm 3.2.2 Apache-2.0
ktor-serialization-kotlinx-json-wasm-js 3.2.2 Apache-2.0
ktor-serialization-kotlinx-json 3.2.2 Apache-2.0
ktor-serialization-kotlinx-jvm 3.2.2 Apache-2.0
ktor-serialization-kotlinx-wasm-js 3.2.2 Apache-2.0
ktor-serialization-kotlinx 3.2.2 Apache-2.0
ktor-serialization-wasm-js 3.2.2 Apache-2.0
ktor-serialization 3.2.2 Apache-2.0
ktor-server-cio-js 3.2.2 Apache-2.0
ktor-server-cio-jvm 3.2.2 Apache-2.0
ktor-server-cio-wasm-js 3.2.2 Apache-2.0
ktor-server-cio 3.2.2 Apache-2.0
ktor-server-core-js 3.2.2 Apache-2.0
ktor-server-core-jvm 3.2.2 Apache-2.0
ktor-server-core-wasm-js 3.2.2 Apache-2.0
ktor-server-core 3.2.2 Apache-2.0
ktor-server-sse-js 3.2.2 Apache-2.0
ktor-server-sse-jvm 3.2.2 Apache-2.0
ktor-server-sse-wasm-js 3.2.2 Apache-2.0
ktor-server-sse 3.2.2 Apache-2.0
ktor-server-websockets-jvm 3.2.1 Apache-2.0
ktor-sse-js 3.2.2 Apache-2.0
ktor-sse-jvm 3.2.2 Apache-2.0
ktor-sse-wasm-js 3.2.2 Apache-2.0
ktor-sse 3.2.2 Apache-2.0
ktor-utils-js 3.2.2 Apache-2.0
ktor-utils-jvm 3.2.2 Apache-2.0
ktor-utils-wasm-js 3.2.2 Apache-2.0
ktor-utils 3.2.2 Apache-2.0
ktor-websocket-serialization-js 3.2.2 Apache-2.0
ktor-websocket-serialization-jvm 3.2.2 Apache-2.0
ktor-websocket-serialization-wasm-js 3.2.2 Apache-2.0
ktor-websocket-serialization 3.2.2 Apache-2.0
ktor-websockets-js 3.2.2 Apache-2.0
ktor-websockets-jvm 3.2.2 Apache-2.0
ktor-websockets-wasm-js 3.2.2 Apache-2.0
ktor-websockets 3.2.2 Apache-2.0
lettuce-core 6.5.5.release MIT
log4j-api 2.24.3 Apache-2.0
log4j-to-slf4j 2.24.3 Apache-2.0
logback-classic 1.5.13 EPL-1.0
LGPL-2.0-or-later
logback-classic 1.5.18 EPL-1.0
LGPL-2.0-or-later
logback-core 1.5.13 EPL-1.0
LGPL-2.0-or-later
logback-core 1.5.18 EPL-1.0
LGPL-2.0-or-later
netty-common 4.1.118.final Apache-2.0
opentelemetry-api 1.51.0 Apache-2.0
opentelemetry-context 1.51.0 Apache-2.0
opentelemetry-exporter-logging 1.51.0 Apache-2.0
opentelemetry-exporter-otlp 1.51.0 Apache-2.0
opentelemetry-sdk-common 1.51.0 Apache-2.0
opentelemetry-sdk-logs 1.51.0 Apache-2.0
opentelemetry-sdk-metrics 1.51.0 Apache-2.0
opentelemetry-sdk-trace 1.51.0 Apache-2.0
opentelemetry-sdk 1.51.0 Apache-2.0
reactive-streams 1.0.4 MIT-0
reactor-core 3.6.6 Apache-2.0
reactor-kotlin-extensions 1.2.3 Apache-2.0
runtime-core-jvm 1.5.4 Apache-2.0
slf4j-api 2.0.17 MIT
smithy-client-jvm 1.5.4 Apache-2.0
snakeyaml 2.4 Apache-2.0
spring-boot-autoconfigure 3.5.3 Apache-2.0
spring-boot-starter-logging 3.5.3 Apache-2.0
spring-boot-starter 3.5.3 Apache-2.0
spring-boot 3.5.3 Apache-2.0
spring-core 6.2.8 Apache-2.0
stax2-api 4.2.1 BSD-2-Clause
BSD-3-Clause
stdlib stdlib Apache-2.0
telemetry-api-jvm 1.5.4 Apache-2.0
woodstox-core 6.2.4 Apache-2.0
Contact Qodana team

Contact us at [email protected]

  • Or via our issue tracker: https://jb.gg/qodana-issue
  • Or share your feedback: https://jb.gg/qodana-discussions

github-actions[bot] avatar Aug 10 '25 16:08 github-actions[bot]

I see there're also prompt caching, OpenAIEndpoint completions, and OpenAIEndpoint responses capabilities not mentioned in the test. Was there a particular reason for that? If not, I'd suggest adding them to the basic check, like just verifying that there's no error if the model profile supports them (and an error is thrown if the model does not).

aozherelyeva avatar Sep 09 '25 10:09 aozherelyeva

I see there're also prompt caching

Same as for temperature and speculation. Providers' API ignores it in case the model doesn't support it and we don't check it on our end. I will add a positive check, but there are currently no models with this capability.

OpenAIEndpoint completions, and OpenAIEndpoint responses capabilities

It was added recently, after this PR was created. I'll see if I can add coverage for them.

skarpovdev avatar Sep 09 '25 16:09 skarpovdev

Qodana for JVM

863 new problems were found

Inspection name Severity Problems
Check Kotlin and Java source code coverage 🔶 Warning 820
Missing KDoc for public API declaration 🔶 Warning 21
Vulnerable imported dependency 🔶 Warning 21
String concatenation that can be converted to string template ◽️ Notice 1 
@@ Code coverage @@
+ 67% total lines covered
11450 lines analyzed, 7719 lines covered
# Calculated according to the filters of your coverage tool

☁️ View the detailed Qodana report

Contact Qodana team

Contact us at [email protected]

  • Or via our issue tracker: https://jb.gg/qodana-issue
  • Or share your feedback: https://jb.gg/qodana-discussions

github-actions[bot] avatar Sep 09 '25 17:09 github-actions[bot]