Conflicting Kotlin versions for DynamoDB local `1.16.0`

[spanierm42]

As indicated in the post to another issue, we are facing conflicts when using the latest DynamoDB local version 1.16.0 in a Kotlin project. The reason is that the DynamoDB local jar seems to contain a copy of the Kotlin standard lib which conflicts with the one we are using for our project. We tested this both with a 1.4.x and 1.5.x version of Kotlin and both failed. Can you please fix the setup so that we can use Kotlin and DynamoDB local, again?

---

@zoewangg Thanks for the information about the release of 1.16.0. We were successfully able to start an embedded DynamoDB local instance in a JUnit test. However, once we now face some compiler warning:

w: Some JAR files in the classpath have the Kotlin Runtime library bundled into them. This may cause difficult to debug problems if there's a different version of the Kotlin Runtime library in the classpath. Consider removing these libraries from the classpath
w: /home/markus/.gradle/caches/modules-2/files-2.1/com.amazonaws/DynamoDBLocal/1.16.0/f705aa07cee4eb8199e70a393c5bd8173e7290ac/DynamoDBLocal-1.16.0.jar: Library has Kotlin runtime bundled into it

Moreover, all integration tests fail due to conflicting Kotlin standard library files, see the following error details:

***************************
APPLICATION FAILED TO START
***************************
Description:
An attempt was made to call a method that does not exist. The attempt was made from the following location:
    kotlin.reflect.jvm.internal.impl.descriptors.Visibilities.<clinit>(Visibilities.kt:56)
The following method did not exist:
    'java.util.Map kotlin.collections.MapsKt.createMapBuilder()'
The method's class, kotlin.collections.MapsKt, is available from the following locations:
    jar:file:/home/markus/.gradle/caches/modules-2/files-2.1/com.amazonaws/DynamoDBLocal/1.16.0/f705aa07cee4eb8199e70a393c5bd8173e7290ac/DynamoDBLocal-1.16.0.jar!/kotlin/collections/MapsKt.class
    jar:file:/home/markus/.gradle/caches/modules-2/files-2.1/org.jetbrains.kotlin/kotlin-stdlib/1.5.0/29dae2501ca094416d15af0e21470cb634780444/kotlin-stdlib-1.5.0.jar!/kotlin/collections/MapsKt.class
The class hierarchy was loaded from the following locations:
    kotlin.collections.MapsKt: file:/home/markus/.gradle/caches/modules-2/files-2.1/com.amazonaws/DynamoDBLocal/1.16.0/f705aa07cee4eb8199e70a393c5bd8173e7290ac/DynamoDBLocal-1.16.0.jar
    kotlin.collections.MapsKt___MapsKt: file:/home/markus/.gradle/caches/modules-2/files-2.1/com.amazonaws/DynamoDBLocal/1.16.0/f705aa07cee4eb8199e70a393c5bd8173e7290ac/DynamoDBLocal-1.16.0.jar
    kotlin.collections.MapsKt__MapsKt: file:/home/markus/.gradle/caches/modules-2/files-2.1/com.amazonaws/DynamoDBLocal/1.16.0/f705aa07cee4eb8199e70a393c5bd8173e7290ac/DynamoDBLocal-1.16.0.jar
    kotlin.collections.MapsKt__MapsJVMKt: file:/home/markus/.gradle/caches/modules-2/files-2.1/com.amazonaws/DynamoDBLocal/1.16.0/f705aa07cee4eb8199e70a393c5bd8173e7290ac/DynamoDBLocal-1.16.0.jar
    kotlin.collections.MapsKt__MapWithDefaultKt: file:/home/markus/.gradle/caches/modules-2/files-2.1/com.amazonaws/DynamoDBLocal/1.16.0/f705aa07cee4eb8199e70a393c5bd8173e7290ac/DynamoDBLocal-1.16.0.jar
Action:
Correct the classpath of your application so that it contains a single, compatible version of kotlin.collections.MapsKt

Can you help us with the issue or guide us towrads a collegue that can provide more information?

We are using Kotlin 1.5, AWS SDK 2, Spring Boot, Corretto 11 as JRE, Gradle. Moreover, it seems like the DynamoDB local jar contains indeed a copy of the Kotlin standard library classes/files. Is it possible to remove those from the bundle?

Originally posted by @spanierm in https://github.com/aws/aws-sdk-java-v2/issues/982#issuecomment-843235519

[rafalwrzeszcz]

I assume the Kotlin SDK is bundled to allow stand-alone running of the .jar? Why not create separate jars - one for inclusion as dependency, and uber-jar for stand-alone calls?

[liamcassidy-eb]

Hello, are there any updates on this? Is there a workaround? Happy to help if there's anything we can do to assist 🙂

[liamcassidy-eb]

Posting this in case it helps anyone else out. I was able to get around this with the following Gradle configuration. Feel free to let me know if there is an easier way to accomplish this 🙂.

(I've only included the configuration relevant to local dynamo below)

// specifying a separate source set allows us to isolate the dependencies of DynamoLocal from the classpath
// of our test suite. Along with a source set we must specify a configuration:
// https://docs.gradle.org/current/userguide/declaring_dependencies.html#sec:defining-custom-configurations
sourceSets {
    create("dynamoLocal")
}
val dynamoLocalImplementation by configurations.getting {
    extendsFrom(configurations.implementation.get())
}

// allow the configuration to be resolved for use in tasks
configurations["dynamoLocalImplementation"].isCanBeResolved = true

// copies the sqlite native libraries into the build folder for resolution by local dynamo dependencies
tasks.register<Copy>("copySqliteNativeLibs") {
    from(configurations["dynamoLocalImplementation"]) {
        include("libsqlite4java-linux*.so")
        include("libsqlite4java*.dylib")

        // remove the version number in the native library files so that sqlite can resolve them
        rename("libsqlite4java-linux-amd64-.*\\.so", "libsqlite4java-linux-amd64.so")
        rename("libsqlite4java-linux-i386-.*\\.so", "libsqlite4java-linux-i386.so")
        rename("libsqlite4java-osx-.*\\.dylib", "libsqlite4java-osx.dylib")
    }
    into("build/libs")
}

// copies the transitive dependencies of the local dynamo jar into a dynamo-local folder
tasks.register<Copy>("copyDynamoDeps") {
    from(configurations["dynamoLocalImplementation"]) {
        exclude("kotlin/**/*")
        exclude("META-INF/kotlin-stdlib-coroutines.kotlin_module")
        exclude("META-INF/kotlin-stdlib.kotlin_module")
        exclude("META-INF/kotlin-stdlib-jdk8.kotlin_module")
        exclude("META-INF/kotlin-stdlib-jdk7.kotlin_module")
        exclude("META-INF/kotlin-stdlib-common.kotlin_module")
    }
    into("build/libs/dynamo-local")
}

// unzip the DynamoDBLocal-x.jar and filter out the kotlin libraries
// copy the resulting files into the dynamo-local folder
configurations["dynamoLocalImplementation"].isCanBeResolved = true
tasks.register<Copy>("filterDynamoLocal") {
    dependsOn(tasks["copyDynamoDeps"])
    val dynamoJar = zipTree(
        configurations["dynamoLocalImplementation"].filter {
            it.name == "DynamoDBLocal-1.16.0.jar"
        }.singleFile
    )

    from(dynamoJar) {
        exclude("kotlin/**/*")
        exclude("META-INF/kotlin-stdlib-coroutines.kotlin_module")
        exclude("META-INF/kotlin-stdlib.kotlin_module")
        exclude("META-INF/kotlin-stdlib-jdk8.kotlin_module")
        exclude("META-INF/kotlin-stdlib-jdk7.kotlin_module")
        exclude("META-INF/kotlin-stdlib-common.kotlin_module")
    }
    into("build/libs/dynamo-local")
}

dependencies {

    // download the DynamoLocal into an isolated dynamoLocal configuration
    dynamoLocalImplementation("com.amazonaws:DynamoDBLocal:1.16.0")

    // add dynamo-local filtered jar files to the test classpath
    testImplementation(
        files(layout.buildDirectory.dir("libs/dynamo-local")) {
            builtBy("filterDynamoLocal")
        }
    )

    // add transitive jar dependencies from dynamo local to the test classpath
    testImplementation(
        fileTree(
            mapOf(
                "dir" to "build/libs/dynamo-local",
                "include" to listOf("*.jar")
            )
        )
    )
}

tasks.test {
    // make sure we copy the sqlite libraries before running tests
    dependsOn(tasks["copySqliteNativeLibs"])
    // tells local dynamo where to find the native sqlite libraries
    systemProperty("java.library.path", "build/libs")
}

[mjsztainbok]

Any updates about this as it still happens with version 1.18.0? I got it working with the Gradle workaround above (although I also needed to exclude the DynamoDB Local JAR). However it would good if it just worked without requiring all of that as it complicates the Gradle build file.

[aeskreis-ta]

Bumping this. Still not working and this workaround is absurd. Please create a jar that is compatibile with Kotlin projects.

[sandeeptharayilGit]

Any update?

[debora-ito]

I've contacted the DynamoDB Local team again for an update, will add their reply here.

[maciejwalkowiak]

Not a fix but you can consider using Localstack with Testcontainers' Localstack Module as an alternative

(sample https://github.com/awspring/spring-cloud-aws/blob/main/spring-cloud-aws-dynamodb/src/test/java/io/awspring/cloud/dynamodb/DynamoDbTemplateIntegrationTest.java#L46)

[DanielJNoteboom]

@debora-ito Is there any update on this?

[Ad1tyaV]

@maciejwalkowiak that solution needs localstack running on a container though right ?

[rafalwrzeszcz]

Not a fix but you can consider using Localstack with Testcontainers' Localstack Module as an alternative

(sample https://github.com/awspring/spring-cloud-aws/blob/main/spring-cloud-aws-dynamodb/src/test/java/io/awspring/cloud/dynamodb/DynamoDbTemplateIntegrationTest.java#L46)

I just run DynamoDbLocal as a standalone jar, but still - as you mentioned - not a fix, just a dirty workardoung:

test-local:
	docker run -d --rm --name dynamodb -p 8000:8000 amazon/dynamodb-local:1.18.0 -jar DynamoDBLocal.jar -inMemory
	make test
	docker stop dynamodb

[maciejwalkowiak]

@maciejwalkowiak that solution needs localstack running on a container though right ?

@Ad1tyaV yep. Not an issue if you can are able to use containers.

[debora-ito]

Got a question from the DynamoDB Local team:

Do you see the error even when the Kotlin version used in your project matches the internal DynamoDBLocal Kotlin version?

DynamoDBLocal >= 1.19 uses Kotlin 1.6.21. DynamoDBLocal <= 1.18 uses Kotlin 1.3.72.

[chrscole]

Thank you for contacting the DynamoDB local team. Currently there are no plans to create a separate DynamoDB local version without an internal Kotlin library. However, we have updated DDB local jar to the latest Kotlin version which allows to fix the conflict. Please give this a try and see if this addresses your issue.

[github-actions[bot]]

It looks like this issue has not been active for more than five days. In the absence of more information, we will be closing this issue soon. If you find that this is still a problem, please add a comment to prevent automatic closure, or if the issue is already closed please feel free to reopen it.

[vgiguere]

Can we reopen this issue and kindly ask to remove these kotlin classes from the jar? This is literally preventing folks to use it in a kotlin application.

The problem is occurring with DynamoDBLocal 2.0 and the kotlin companion object TimeSource.Monotonic defined twice: DynamoDBLocal-2.0.0.jar!/kotlin/time/TimeSource$Monotonic.class kotlin-stdlib-1.8.21.jar!/kotlin/time/TimeSource$Monotonic.class

DynamoDBLocal is a great tool, don't force us to go use LocalStack please.

Can you simply do as suggested above by @rafalwrzeszcz : create 2 distributions of DynamoDBLocal.jar

• One uber jar with the kotlin classeses
• One smaller without the kotlin classes

That would be very much appreciated