testcontainers-java icon indicating copy to clipboard operation
testcontainers-java copied to clipboard
verified publisher icon testcontainers

[Bug]: 1.18.1 to 1.18.3 gives Out Of Heap Memory exception for DockerComposeContainer

Open frankjkelly opened this issue 2 years ago • 24 comments

Module

Core

Testcontainers version

1.18.1 to 1.18.3

Using the latest Testcontainers version?

Yes

Host OS

MacOS

Host Arch

x86 Intel Core i7

Docker version

$ docker version
Client:
 Cloud integration: v1.0.31
 Version:           23.0.5
 API version:       1.42
 Go version:        go1.19.8
 Git commit:        bc4487a
 Built:             Wed Apr 26 16:12:52 2023
 OS/Arch:           darwin/amd64
 Context:           default

Server: Docker Desktop 4.19.0 (106363)
 Engine:
  Version:          23.0.5
  API version:      1.42 (minimum version 1.12)
  Go version:       go1.19.8
  Git commit:       94d3ad6
  Built:            Wed Apr 26 16:17:45 2023
  OS/Arch:          linux/amd64
  Experimental:     false
 containerd:
  Version:          1.6.20
  GitCommit:        2806fc1057397dbaeefbea0e4e17bddfbd388f38
 runc:
  Version:          1.1.5
  GitCommit:        v1.1.5-0-gf19387a
 docker-init:
  Version:          0.19.0
  GitCommit:        de40ad0

What happened?

When upgrading only test containers from 1.18.0 to 1.18.1 got heap error on startup of my Docker Compose based Integration tests. Also tried 1.18.3.

Relevant log output

org.gradle.api.internal.tasks.testing.TestSuiteExecutionException: Could not complete execution for Gradle Test Executor 1.
	at org.gradle.api.internal.tasks.testing.SuiteTestClassProcessor.stop(SuiteTestClassProcessor.java:64)
	at [email protected]/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at [email protected]/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
	at [email protected]/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at [email protected]/java.lang.reflect.Method.invoke(Method.java:566)
	at org.gradle.internal.dispatch.ReflectionDispatch.dispatch(ReflectionDispatch.java:36)
	at org.gradle.internal.dispatch.ReflectionDispatch.dispatch(ReflectionDispatch.java:24)
	at org.gradle.internal.dispatch.ContextClassLoaderDispatch.dispatch(ContextClassLoaderDispatch.java:33)
	at org.gradle.internal.dispatch.ProxyDispatchAdapter$DispatchingInvocationHandler.invoke(ProxyDispatchAdapter.java:94)
	at com.sun.proxy.$Proxy5.stop(Unknown Source)
	at org.gradle.api.internal.tasks.testing.worker.TestWorker$3.run(TestWorker.java:193)
	at org.gradle.api.internal.tasks.testing.worker.TestWorker.executeAndMaintainThreadName(TestWorker.java:129)
	at org.gradle.api.internal.tasks.testing.worker.TestWorker.execute(TestWorker.java:100)
	at org.gradle.api.internal.tasks.testing.worker.TestWorker.execute(TestWorker.java:60)
	at org.gradle.process.internal.worker.child.ActionExecutionWorker.execute(ActionExecutionWorker.java:56)
	at org.gradle.process.internal.worker.child.SystemApplicationClassLoaderWorker.call(SystemApplicationClassLoaderWorker.java:113)
	at org.gradle.process.internal.worker.child.SystemApplicationClassLoaderWorker.call(SystemApplicationClassLoaderWorker.java:65)
	at app//worker.org.gradle.process.internal.worker.GradleWorkerMain.run(GradleWorkerMain.java:69)
	at app//worker.org.gradle.process.internal.worker.GradleWorkerMain.main(GradleWorkerMain.java:74)
Caused by: java.lang.OutOfMemoryError: Java heap space
	at java.base/java.util.Arrays.copyOf(Arrays.java:3745)
	at java.base/java.io.ByteArrayOutputStream.grow(ByteArrayOutputStream.java:120)
	at java.base/java.io.ByteArrayOutputStream.ensureCapacity(ByteArrayOutputStream.java:95)
	at java.base/java.io.ByteArrayOutputStream.write(ByteArrayOutputStream.java:156)
	at org.apache.commons.compress.utils.CountingOutputStream.write(CountingOutputStream.java:48)
	at org.apache.commons.compress.utils.FixedLengthBlockOutputStream$BufferAtATimeOutputChannel.write(FixedLengthBlockOutputStream.java:244)
	at org.apache.commons.compress.utils.FixedLengthBlockOutputStream.writeBlock(FixedLengthBlockOutputStream.java:92)
	at org.apache.commons.compress.utils.FixedLengthBlockOutputStream.maybeFlush(FixedLengthBlockOutputStream.java:86)
	at org.apache.commons.compress.utils.FixedLengthBlockOutputStream.write(FixedLengthBlockOutputStream.java:122)
	at org.apache.commons.compress.archivers.tar.TarArchiveOutputStream.write(TarArchiveOutputStream.java:462)
	at java.base/java.io.InputStream.transferTo(InputStream.java:705)
	at java.base/java.nio.file.Files.copy(Files.java:3119)
	at org.testcontainers.utility.MountableFile.recursiveTar(MountableFile.java:362)
	at org.testcontainers.utility.MountableFile.recursiveTar(MountableFile.java:371)
	at org.testcontainers.utility.MountableFile.recursiveTar(MountableFile.java:371)
	at org.testcontainers.utility.MountableFile.recursiveTar(MountableFile.java:371)
	at org.testcontainers.utility.MountableFile.recursiveTar(MountableFile.java:371)
	at org.testcontainers.utility.MountableFile.recursiveTar(MountableFile.java:371)
	at org.testcontainers.utility.MountableFile.transferTo(MountableFile.java:327)
	at org.testcontainers.containers.ContainerState.copyFileToContainer(ContainerState.java:306)
	at org.testcontainers.containers.ContainerState.copyFileToContainer(ContainerState.java:282)
	at org.testcontainers.containers.GenericContainer$$Lambda$482/0x0000000800420840.accept(Unknown Source)
	at java.base/java.util.LinkedHashMap.forEach(LinkedHashMap.java:684)
	at org.testcontainers.containers.GenericContainer.tryStart(GenericContainer.java:430)
	at org.testcontainers.containers.GenericContainer.lambda$doStart$0(GenericContainer.java:344)
	at org.testcontainers.containers.GenericContainer$$Lambda$447/0x0000000800393c40.call(Unknown Source)
	at org.rnorth.ducttape.unreliables.Unreliables.retryUntilSuccess(Unreliables.java:81)
	at org.testcontainers.containers.GenericContainer.doStart(GenericContainer.java:334)
	at org.testcontainers.containers.GenericContainer.start(GenericContainer.java:322)
	at org.testcontainers.containers.ContainerisedDockerCompose.invoke(DockerComposeContainer.java:727)
	at org.testcontainers.containers.DockerComposeContainer.runWithCompose(DockerComposeContainer.java:339)
	at org.testcontainers.containers.DockerComposeContainer.createServices(DockerComposeContainer.java:260)

Additional Information

No response

frankjkelly avatar Jun 22 '23 21:06 frankjkelly

i would like to contribute this issue, kindly assign this to me

Saimurugeshwari avatar Jun 28 '23 20:06 Saimurugeshwari

This was caused by https://github.com/testcontainers/testcontainers-java/pull/6945/commits/00a509c72a295f61e93e56fd31721fd565a7011b

linus-learns avatar Jul 06 '23 09:07 linus-learns

is your docker-compose file that big?

eddumelendez avatar Jul 06 '23 10:07 eddumelendez

If I interpret the code correctly, you are trying to copy the whole directory where the compose file is located into the container causing the oom: final String pwd = dockerComposeBaseFile.getAbsoluteFile().getParentFile().getAbsolutePath(); final String containerPwd = convertToUnixFilesystemPath(pwd); ... withCopyFileToContainer(MountableFile.forHostPath(pwd), containerPwd);

from https://github.com/testcontainers/testcontainers-java/blob/595076c21573b96b73c4d8b1c6808ad6134e99ee/core/src/main/java/org/testcontainers/containers/ContainerisedDockerCompose.java#L33 and onwards

linus-learns avatar Jul 06 '23 10:07 linus-learns

Thanks! good catch! I didn't fall into that because the docker-compose files are inside different folders.

I need to figure out how to fix it properly. The workaround would be to move the docker-compose.yml to a folder

eddumelendez avatar Jul 11 '23 16:07 eddumelendez

Depending on what your docker-compose.yml contains, it might be necessary to copy stuff. As an example if you mount config files for a container. So a mechanism like .dockerignore could be of help.

As workaround use withLocalCompose(true) if possible.

markush81 avatar Aug 22 '23 14:08 markush81

Still have issue on 1.19.0. Only working with 1.18.0

jandry avatar Sep 06 '23 09:09 jandry

So sounds like two work-arounds so far

  1. Move docker-compose file to dedicated folder as per https://github.com/testcontainers/testcontainers-java/issues/7239#issuecomment-1631170376
  2. Use use withLocalCompose(true) as per https://github.com/testcontainers/testcontainers-java/issues/7239#issuecomment-1688298605

frankjkelly avatar Sep 06 '23 12:09 frankjkelly

@frankjkelly - fancy meeting you here, long time no see :).
So i ran into the same problem, but we are on version 1.19.0 and withLocalCompose(true) did not do the trick for me, but we have a developer investigating both work arounds further. Curious if you had any luck on your end with the workarounds?

Michael-Rosa-Imprivata avatar Sep 12 '23 23:09 Michael-Rosa-Imprivata

Hey there @Michael-Rosa-Imprivata - wow fancy meeting you here!? Hope all is well. I haven't tried any of the workarounds yet so we just pinned the testcontainers version :-(

frankjkelly avatar Sep 13 '23 12:09 frankjkelly

FYI I just tried work-around of moving the docker-compose.yml to it's own directory but still no luck

Caused by: org.testcontainers.containers.ContainerLaunchException: Container startup failed for image docker/compose:1.29.2
	at app//org.testcontainers.containers.GenericContainer.doStart(GenericContainer.java:361)
	at app//org.testcontainers.containers.GenericContainer.start(GenericContainer.java:334)
	at app//org.testcontainers.containers.ContainerisedDockerCompose.invoke(ContainerisedDockerCompose.java:64)
	at app//org.testcontainers.containers.ComposeDelegate.runWithCompose(ComposeDelegate.java:254)
	at app//org.testcontainers.containers.ComposeDelegate.createServices(ComposeDelegate.java:163)
	at app//org.testcontainers.containers.DockerComposeContainer.start(DockerComposeContainer.java:137)
	at app//com.cogito.platform.signal.AbstractIntegrationTest.<clinit>(AbstractIntegrationTest.java:102)
	... 97 more
Caused by: org.rnorth.ducttape.RetryCountExceededException: Retry limit hit with exception
	at app//org.rnorth.ducttape.unreliables.Unreliables.retryUntilSuccess(Unreliables.java:88)
	at app//org.testcontainers.containers.GenericContainer.doStart(GenericContainer.java:346)
	... 103 more
Caused by: org.testcontainers.containers.ContainerLaunchException: Could not create/start container
	at app//org.testcontainers.containers.GenericContainer.tryStart(GenericContainer.java:565)
	at app//org.testcontainers.containers.GenericContainer.lambda$doStart$0(GenericContainer.java:356)
	at app//org.rnorth.ducttape.unreliables.Unreliables.retryUntilSuccess(Unreliables.java:81)
	... 104 more
Caused by: java.lang.IllegalStateException: Container did not start correctly.
	at org.testcontainers.containers.GenericContainer.tryStart(GenericContainer.java:497)
	... 106 more

frankjkelly avatar Sep 13 '23 14:09 frankjkelly

Yea same here Frank. Work arounds are not doing the trick. We decided to revert back to the previous working version as well for now.

Michael-Rosa-Imprivata avatar Sep 28 '23 14:09 Michael-Rosa-Imprivata

@Michael-Rosa-Imprivata just out of interest

So withLocalCompose(true) still runs into java.lang.OutOfMemoryError: Java heap space.

That sounds odd (or different issue), since the LocalDockerCompose doesn't copy, compared to ContainerisedDockerCompose in GenericContainer.

Could you paste stacktrace/output/code snippet, when using withLocalCompose(true).

markush81 avatar Sep 29 '23 15:09 markush81

Hi, I tried a workaround : I forced a downgrade org.apache.commons:commons-compress to 1.22.0. But it's seems not working also. Still only working workaround, stay with 1.18.0 :(

Exemple in gradle

fun Configuration.resolutionStrategyForSecurity() {
    resolutionStrategy.eachDependency {
        if (requested.group == "org.apache.commons"  && (requested.name == "commons-compress") && (requested.version == "1.23.0" || requested.version == "1.24.0")) {
            useVersion("1.22")
            because("Memory issue in integration test or system test with testcontainers 1.18.3/1.19.0 (java.lang.OutOfMemoryError: Java heap space in ContainerState.copyFileToContainer) https://github.com/testcontainers/testcontainers-java/issues/2863")
        }
    }
}

jandry avatar Oct 04 '23 10:10 jandry

withLocalCompose(true) helped to migrate to 1.19.1. Could somebody, please, explain what property withLocalCompose doing?

eugene-kuntsevich avatar Oct 05 '23 12:10 eugene-kuntsevich

@eugene-kuntsevich https://java.testcontainers.org/modules/docker_compose/#local-compose-mode

In short, it executes docker-compose (or v2 docker compose) on your machine (or the CI machine) instead of doing it, in a container - and therefore has no need to copy stuff.

markush81 avatar Oct 05 '23 13:10 markush81

When I try withLocalCompose(true)( with 1.18.0 and 1.19.1) I get

Caused by: java.lang.ExceptionInInitializerError
	at com.cogito.platform.admin.AbstractIntegrationTest$ContextConfiguration.<clinit>(AbstractIntegrationTest.java:136)
	at java.base/java.lang.Class.forName0(Native Method)
	at java.base/java.lang.Class.forName(Class.java:467)
	at org.springframework.cglib.core.ReflectUtils.defineClass(ReflectUtils.java:602)
	at org.springframework.cglib.core.AbstractClassGenerator.generate(AbstractClassGenerator.java:363)
	at org.springframework.cglib.proxy.Enhancer.generate(Enhancer.java:585)
	at org.springframework.cglib.core.AbstractClassGenerator$ClassLoaderData$3.apply(AbstractClassGenerator.java:110)
	at org.springframework.cglib.core.AbstractClassGenerator$ClassLoaderData$3.apply(AbstractClassGenerator.java:108)
	at org.springframework.cglib.core.internal.LoadingCache$2.call(LoadingCache.java:54)
	at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264)
	at org.springframework.cglib.core.internal.LoadingCache.createEntry(LoadingCache.java:61)
	... 86 more
Caused by: org.testcontainers.containers.ContainerLaunchException: Container startup failed for image alpine/socat:1.7.4.3-r0
	at app//org.testcontainers.containers.GenericContainer.doStart(GenericContainer.java:349)
	at app//org.testcontainers.containers.GenericContainer.start(GenericContainer.java:322)
	at app//org.testcontainers.containers.DockerComposeContainer.startAmbassadorContainers(DockerComposeContainer.java:359)
	at app//org.testcontainers.containers.DockerComposeContainer.start(DockerComposeContainer.java:190)
	at app//com.cogito.platform.admin.AbstractIntegrationTest.<clinit>(AbstractIntegrationTest.java:101)
	... 97 more
Caused by: org.rnorth.ducttape.RetryCountExceededException: Retry limit hit with exception
	at app//org.rnorth.ducttape.unreliables.Unreliables.retryUntilSuccess(Unreliables.java:88)
	at app//org.testcontainers.containers.GenericContainer.doStart(GenericContainer.java:334)
	... 101 more
Caused by: org.testcontainers.containers.ContainerLaunchException: Could not create/start container
	at app//org.testcontainers.containers.GenericContainer.tryStart(GenericContainer.java:553)
	at app//org.testcontainers.containers.GenericContainer.lambda$doStart$0(GenericContainer.java:344)
	at app//org.rnorth.ducttape.unreliables.Unreliables.retryUntilSuccess(Unreliables.java:81)
	... 102 more
Caused by: org.testcontainers.containers.ContainerLaunchException: Aborting attempt to link to container tcntmtj1rvmc_entity-service_1 as it is not running
	at app//org.testcontainers.containers.GenericContainer.applyConfiguration(GenericContainer.java:839)
	at app//org.testcontainers.containers.GenericContainer.tryStart(GenericContainer.java:378)
	... 104 more

Is there a second part to this workaround that is needed?

frankjkelly avatar Oct 05 '23 13:10 frankjkelly

withLocalCompose(true) helped to migrate to 1.19.1. Could somebody, please, explain what property withLocalCompose doing?

Like explained here https://java.testcontainers.org/modules/docker_compose/, it means that testcontainers use the local binary of docker compose instead of using a docker image with docker compose inside. So locally no issue, but in CI this means that your runner image must have a docker compose binary

jandry avatar Oct 05 '23 18:10 jandry

Any update on this?

Gregory-Berkman-Imprivata avatar Oct 20 '23 19:10 Gregory-Berkman-Imprivata

Any update?

odwrotnie avatar Nov 17 '23 21:11 odwrotnie

Is there any workaround for this?

erwin-ramirez-da avatar Jan 11 '24 10:01 erwin-ramirez-da

#1348 is i think the most highly relevant piece - if copying the files is really non-negotiable (maybe making this bind vs copy behavior configurable is agreeable to testcontainers-java maintainers?) then implementing .dockerignore support in docker-java is probably the way to go.

https://github.com/testcontainers/testcontainers-java/compare/main...alexanderankin:testcontainers-java:reduce_7239

alexanderankin avatar Jan 26 '24 04:01 alexanderankin

Hi @alexanderankin , thanks a lot ! I tried your fix build a snapshot version locally and it working fine for me

gba-foundever avatar Jan 27 '24 11:01 gba-foundever

https://github.com/testcontainers/testcontainers-java/pull/2864 was replacing some in-memory buffering (ByteArrayOutputStream) with a temp file. Not sure if it still is relevant.

findepi avatar Jan 30 '24 19:01 findepi

#8409 has been merged and it should fix the issue.

eddumelendez avatar Jul 04 '24 22:07 eddumelendez