testcontainers-java
testcontainers-java copied to clipboard
testcontainers
GenericContainer#withCopyFileToContainer is not efficient memory-wise
Hi,
We've recently seen OOM in our tests and after debugging by generating a heap dump and opening it, we found that for the following code that we use in XWiki tests, TC is copying all the content of the sourceDirectory in memory:
MountableFile mountableDirectory = MountableFile.forHostPath(sourceDirectory);
container.withCopyFileToContainer(mountableDirectory, targetDirectory);
Since our directory contains 300MB+ of files, the memory we give to the JVM is not enough.
Note that we copy and don't bind because we need to have it work in the DOOD use case. Our code is:
protected void mountFromHostToContainer(GenericContainer<?> container, String sourceDirectory,
String targetDirectory)
{
// Note 1: File mounting is awfully slow on Mac OSX. For example starting Tomcat with XWiki mounted takes
// 45s+, while doing a COPY first and then starting Tomcat takes 8s (+5s for the copy).
// Note 2: For the DOOD use case, we also do the copy instead of the volume mounting since that would require
// to have the sourceDirectory path mounted from the host and this would put and leave files on the host which
// would not work with parallel executions (think about multiple CI jobs executing in parallel on the same host)
// and would also not be clean.
String osName = System.getProperty("os.name").toLowerCase();
if (isInAContainer() || osName.startsWith("mac os x")) {
MountableFile mountableDirectory = MountableFile.forHostPath(sourceDirectory);
container.withCopyFileToContainer(mountableDirectory, targetDirectory);
} else {
container.withFileSystemBind(sourceDirectory, targetDirectory);
}
}
It would be great if TC could stream the data instead of keeping it in memory. Is that possible?
Thanks
Guys, any take on this?
FYI here's the stack trace we got. It's taking more than 300MB in memory which is very inefficient:
Thread 'main' with ID = 1
java.lang.OutOfMemoryError.<init>(OutOfMemoryError.java:48)
java.util.Arrays.copyOf(Arrays.java:3236)
java.io.ByteArrayOutputStream.grow(ByteArrayOutputStream.java:118)
java.io.ByteArrayOutputStream.ensureCapacity(ByteArrayOutputStream.java:93)
java.io.ByteArrayOutputStream.write(ByteArrayOutputStream.java:153)
org.apache.commons.compress.utils.CountingOutputStream.write(CountingOutputStream.java:48)
org.apache.commons.compress.utils.FixedLengthBlockOutputStream$BufferAtATimeOutputChannel.write(FixedLengthBlockOutputStream.java:244)
org.apache.commons.compress.utils.FixedLengthBlockOutputStream.writeBlock(FixedLengthBlockOutputStream.java:92)
org.apache.commons.compress.utils.FixedLengthBlockOutputStream.maybeFlush(FixedLengthBlockOutputStream.java:86)
org.apache.commons.compress.utils.FixedLengthBlockOutputStream.write(FixedLengthBlockOutputStream.java:122)
org.apache.commons.compress.archivers.tar.TarArchiveOutputStream.write(TarArchiveOutputStream.java:454)
java.nio.file.Files.copy(Files.java:2909)
java.nio.file.Files.copy(Files.java:3069)
org.testcontainers.utility.MountableFile.recursiveTar(MountableFile.java:326)
org.testcontainers.utility.MountableFile.recursiveTar(MountableFile.java:335)
org.testcontainers.utility.MountableFile.recursiveTar(MountableFile.java:335)
org.testcontainers.utility.MountableFile.recursiveTar(MountableFile.java:335)
org.testcontainers.utility.MountableFile.transferTo(MountableFile.java:300)
org.testcontainers.containers.ContainerState.copyFileToContainer(ContainerState.java:276)
org.testcontainers.containers.ContainerState.copyFileToContainer(ContainerState.java:253)
org.testcontainers.containers.GenericContainer$$Lambda$410.accept(Native method)
java.util.LinkedHashMap.forEach(LinkedHashMap.java:684)
org.testcontainers.containers.GenericContainer.tryStart(GenericContainer.java:410)
org.testcontainers.containers.GenericContainer.lambda$doStart$0(GenericContainer.java:325)
org.testcontainers.containers.GenericContainer$$Lambda$390.call(Native method)
org.rnorth.ducttape.unreliables.Unreliables.retryUntilSuccess(Unreliables.java:81)
org.testcontainers.containers.GenericContainer.doStart(GenericContainer.java:323)
org.testcontainers.containers.GenericContainer.start(GenericContainer.java:311)
org.xwiki.test.docker.internal.junit5.DockerTestUtils.startContainerInternal(DockerTestUtils.java:204)
org.xwiki.test.docker.internal.junit5.DockerTestUtils.startContainer(DockerTestUtils.java:167)
org.xwiki.test.docker.internal.junit5.AbstractContainerExecutor.start(AbstractContainerExecutor.java:56)
org.xwiki.test.docker.internal.junit5.servletengine.ServletContainerExecutor.startContainer(ServletContainerExecutor.java:258)
org.xwiki.test.docker.internal.junit5.servletengine.ServletContainerExecutor.start(ServletContainerExecutor.java:132)
org.xwiki.test.docker.internal.junit5.XWikiDockerExtension.startServletEngine(XWikiDockerExtension.java:429)
org.xwiki.test.docker.internal.junit5.XWikiDockerExtension.beforeAllInternal(XWikiDockerExtension.java:184)
org.xwiki.test.docker.internal.junit5.XWikiDockerExtension.beforeAll(XWikiDockerExtension.java:114)
org.junit.jupiter.engine.descriptor.ClassBasedTestDescriptor.lambda$invokeBeforeAllCallbacks$8(ClassBasedTestDescriptor.java:368)
org.junit.jupiter.engine.descriptor.ClassBasedTestDescriptor$$Lambda$313.execute(Native method)
org.junit.platform.engine.support.hierarchical.ThrowableCollector.execute(ThrowableCollector.java:73)
org.junit.jupiter.engine.descriptor.ClassBasedTestDescriptor.invokeBeforeAllCallbacks(ClassBasedTestDescriptor.java:368)
org.junit.jupiter.engine.descriptor.ClassBasedTestDescriptor.before(ClassBasedTestDescriptor.java:192)
org.junit.jupiter.engine.descriptor.ClassBasedTestDescriptor.before(ClassBasedTestDescriptor.java:78)
org.junit.platform.engine.support.hierarchical.NodeTestTask.lambda$executeRecursively$5(NodeTestTask.java:136)
org.junit.platform.engine.support.hierarchical.NodeTestTask$$Lambda$263.execute(Native method)
org.junit.platform.engine.support.hierarchical.ThrowableCollector.execute(ThrowableCollector.java:73)
org.junit.platform.engine.support.hierarchical.NodeTestTask.lambda$executeRecursively$7(NodeTestTask.java:129)
org.junit.platform.engine.support.hierarchical.NodeTestTask$$Lambda$262.invoke(Native method)
org.junit.platform.engine.support.hierarchical.Node.around(Node.java:137)
org.junit.platform.engine.support.hierarchical.NodeTestTask.lambda$executeRecursively$8(NodeTestTask.java:127)
org.junit.platform.engine.support.hierarchical.NodeTestTask$$Lambda$261.execute(Native method)
org.junit.platform.engine.support.hierarchical.ThrowableCollector.execute(ThrowableCollector.java:73)
org.junit.platform.engine.support.hierarchical.NodeTestTask.executeRecursively(NodeTestTask.java:126)
org.junit.platform.engine.support.hierarchical.NodeTestTask.execute(NodeTestTask.java:84)
org.junit.platform.engine.support.hierarchical.SameThreadHierarchicalTestExecutorService$$Lambda$267.accept(Native method)
java.util.ArrayList.forEach(ArrayList.java:1259)
org.junit.platform.engine.support.hierarchical.SameThreadHierarchicalTestExecutorService.invokeAll(SameThreadHierarchicalTestExecutorService.java:38)
org.junit.platform.engine.support.hierarchical.NodeTestTask.lambda$executeRecursively$5(NodeTestTask.java:143)
org.junit.platform.engine.support.hierarchical.NodeTestTask$$Lambda$263.execute(Native method)
org.junit.platform.engine.support.hierarchical.ThrowableCollector.execute(ThrowableCollector.java:73)
org.junit.platform.engine.support.hierarchical.NodeTestTask.lambda$executeRecursively$7(NodeTestTask.java:129)
org.junit.platform.engine.support.hierarchical.NodeTestTask$$Lambda$262.invoke(Native method)
org.junit.platform.engine.support.hierarchical.Node.around(Node.java:137)
org.junit.platform.engine.support.hierarchical.NodeTestTask.lambda$executeRecursively$8(NodeTestTask.java:127)
org.junit.platform.engine.support.hierarchical.NodeTestTask$$Lambda$261.execute(Native method)
org.junit.platform.engine.support.hierarchical.ThrowableCollector.execute(ThrowableCollector.java:73)
org.junit.platform.engine.support.hierarchical.NodeTestTask.executeRecursively(NodeTestTask.java:126)
org.junit.platform.engine.support.hierarchical.NodeTestTask.execute(NodeTestTask.java:84)
org.junit.platform.engine.support.hierarchical.SameThreadHierarchicalTestExecutorService.submit(SameThreadHierarchicalTestExecutorService.java:32)
org.junit.platform.engine.support.hierarchical.HierarchicalTestExecutor.execute(HierarchicalTestExecutor.java:57)
org.junit.platform.engine.support.hierarchical.HierarchicalTestEngine.execute(HierarchicalTestEngine.java:51)
org.junit.platform.launcher.core.EngineExecutionOrchestrator.execute(EngineExecutionOrchestrator.java:108)
org.junit.platform.launcher.core.EngineExecutionOrchestrator.execute(EngineExecutionOrchestrator.java:88)
org.junit.platform.launcher.core.EngineExecutionOrchestrator.lambda$execute$0(EngineExecutionOrchestrator.java:54)
org.junit.platform.launcher.core.EngineExecutionOrchestrator$$Lambda$222.accept(Native method)
org.junit.platform.launcher.core.EngineExecutionOrchestrator.withInterceptedStreams(EngineExecutionOrchestrator.java:67)
org.junit.platform.launcher.core.EngineExecutionOrchestrator.execute(EngineExecutionOrchestrator.java:52)
org.junit.platform.launcher.core.DefaultLauncher.execute(DefaultLauncher.java:96)
org.junit.platform.launcher.core.DefaultLauncher.execute(DefaultLauncher.java:75)
org.apache.maven.surefire.junitplatform.JUnitPlatformProvider.invokeAllTests(JUnitPlatformProvider.java:150)
org.apache.maven.surefire.junitplatform.JUnitPlatformProvider.invoke(JUnitPlatformProvider.java:124)
org.apache.maven.surefire.booter.ForkedBooter.invokeProviderInSameClassLoader(ForkedBooter.java:384)
org.apache.maven.surefire.booter.ForkedBooter.runSuitesInProcess(ForkedBooter.java:345)
org.apache.maven.surefire.booter.ForkedBooter.execute(ForkedBooter.java:126)
org.apache.maven.surefire.booter.ForkedBooter.main(ForkedBooter.java:418)
Thanks!
Thanks for your investigation @vmassol. You are right and we will look into improving this.
I'm currently look into this issue.
I can reproduce the error. Getting OOM errors for files 100mb locally.
Just from what I've gathered, it seems like it's possible to stream to container, I think under the hood it uses:
https://docs.docker.com/engine/api/v1.41/#tag/Container/operation/PutContainerArchive
Which requires us to package up the files into a Tar archive. I think the problem is that we create the Tar archive in memory - and so the whole file gets loaded into memory. One possible solution is to stream the tar archive to disk and then stream it to docker? Still looking into solutions though.
@REslim30 Thanks for looking into this, that sounds indeed like the right direction. @eddumelendez might share some more details since he was recently looking into a similar topic in the context of Jib integration.
Hello guys. FTR we're still getting this issue on the XWiki build.
01:36:19,047 [ERROR] There was an error in the forked process
01:36:19,047 [ERROR] Java heap space
01:36:19,047 [ERROR] java.lang.OutOfMemoryError: Java heap space
01:36:19,047 [ERROR] at java.base/java.util.Arrays.copyOf(Arrays.java:3745)
01:36:19,047 [ERROR] at java.base/java.io.ByteArrayOutputStream.grow(ByteArrayOutputStream.java:120)
01:36:19,047 [ERROR] at java.base/java.io.ByteArrayOutputStream.ensureCapacity(ByteArrayOutputStream.java:95)
01:36:19,047 [ERROR] at java.base/java.io.ByteArrayOutputStream.write(ByteArrayOutputStream.java:156)
01:36:19,047 [ERROR] at org.apache.commons.compress.utils.CountingOutputStream.write(CountingOutputStream.java:62)
01:36:19,047 [ERROR] at org.apache.commons.compress.utils.FixedLengthBlockOutputStream$BufferAtATimeOutputChannel.write(FixedLengthBlockOutputStream.java:91)
01:36:19,047 [ERROR] at org.apache.commons.compress.utils.FixedLengthBlockOutputStream.writeBlock(FixedLengthBlockOutputStream.java:259)
01:36:19,047 [ERROR] at org.apache.commons.compress.utils.FixedLengthBlockOutputStream.maybeFlush(FixedLengthBlockOutputStream.java:169)
01:36:19,047 [ERROR] at org.apache.commons.compress.utils.FixedLengthBlockOutputStream.write(FixedLengthBlockOutputStream.java:206)
01:36:19,047 [ERROR] at org.apache.commons.compress.archivers.tar.TarArchiveOutputStream.write(TarArchiveOutputStream.java:713)
01:36:19,047 [ERROR] at java.base/java.io.InputStream.transferTo(InputStream.java:705)
01:36:19,047 [ERROR] at java.base/java.nio.file.Files.copy(Files.java:3119)
01:36:19,047 [ERROR] at org.testcontainers.utility.MountableFile.recursiveTar(MountableFile.java:362)
01:36:19,047 [ERROR] at org.testcontainers.utility.MountableFile.recursiveTar(MountableFile.java:371)
01:36:19,047 [ERROR] at org.testcontainers.utility.MountableFile.recursiveTar(MountableFile.java:371)
01:36:19,047 [ERROR] at org.testcontainers.utility.MountableFile.recursiveTar(MountableFile.java:371)
01:36:19,047 [ERROR] at org.testcontainers.utility.MountableFile.transferTo(MountableFile.java:327)
01:36:19,047 [ERROR] at org.testcontainers.containers.ContainerState.copyFileToContainer(ContainerState.java:306)
01:36:19,047 [ERROR] at org.testcontainers.containers.ContainerState.copyFileToContainer(ContainerState.java:282)
01:36:19,047 [ERROR] at org.testcontainers.containers.GenericContainer$$Lambda$603/0x00000001003ca040.accept(Unknown Source)
01:36:19,047 [ERROR] at java.base/java.util.LinkedHashMap.forEach(LinkedHashMap.java:684)
01:36:19,047 [ERROR] at org.testcontainers.containers.GenericContainer.tryStart(GenericContainer.java:430)
01:36:19,047 [ERROR] at org.testcontainers.containers.GenericContainer.lambda$doStart$0(GenericContainer.java:344)
01:36:19,047 [ERROR] at org.testcontainers.containers.GenericContainer$$Lambda$571/0x00000001003a1440.call(Unknown Source)
01:36:19,047 [ERROR] at org.rnorth.ducttape.unreliables.Unreliables.retryUntilSuccess(Unreliables.java:81)
01:36:19,047 [ERROR] at org.testcontainers.containers.GenericContainer.doStart(GenericContainer.java:334)
01:36:19,047 [ERROR] at org.testcontainers.containers.GenericContainer.start(GenericContainer.java:322)
01:36:19,047 [ERROR] at org.xwiki.test.docker.internal.junit5.DockerTestUtils.startContainerInternal(DockerTestUtils.java:218)
01:36:19,047 [ERROR] at org.xwiki.test.docker.internal.junit5.DockerTestUtils.startContainer(DockerTestUtils.java:181)
01:36:19,047 [ERROR] at org.xwiki.test.docker.internal.junit5.AbstractContainerExecutor.start(AbstractContainerExecutor.java:56)
01:36:19,047 [ERROR] at org.xwiki.test.docker.internal.junit5.servletengine.ServletContainerExecutor.startContainer(ServletContainerExecutor.java:323)
01:36:19,048 [ERROR] at org.xwiki.test.docker.internal.junit5.servletengine.ServletContainerExecutor.start(ServletContainerExecutor.java:155)
01:36:19,048 [ERROR]
01:36:19,048 [ERROR] at org.apache.maven.plugin.surefire.booterclient.ForkStarter.fork(ForkStarter.java:628)
01:36:19,048 [ERROR] at org.apache.maven.plugin.surefire.booterclient.ForkStarter.run(ForkStarter.java:285)
01:36:19,048 [ERROR] at org.apache.maven.plugin.surefire.booterclient.ForkStarter.run(ForkStarter.java:250)
01:36:19,048 [ERROR] at org.apache.maven.plugin.surefire.AbstractSurefireMojo.executeProvider(AbstractSurefireMojo.java:1203)
01:36:19,048 [ERROR] at org.apache.maven.plugin.surefire.AbstractSurefireMojo.executeAfterPreconditionsChecked(AbstractSurefireMojo.java:1055)
01:36:19,048 [ERROR] at org.apache.maven.plugin.surefire.AbstractSurefireMojo.execute(AbstractSurefireMojo.java:871)
Seems same as https://github.com/testcontainers/testcontainers-java/issues/7239
Seems same as https://github.com/testcontainers/testcontainers-java/issues/7239
I am now also hit by this when using 1.19.1
Currently it is not possible to copy larger files or a directory into the container without getting an OOM.
There are a couple of other tickets which are also related to this OOM (#1348, #7239,#2863, #4203), so I think this is not an edge usecase.
After checking the code, its quite obvious that all the copy operations are done with an in-memory BAOS instead of (batched) streaming. Without major rework in ContainerState, Transferable and MountableFile) there seems to be no easy solution.
The only workaround I can use right now, is calling the docker executable to copy the files with the cp command into the running container.
Of course this does not allow to inject the files before container startup...
I just tried to upgrade to 1.19.3 (from 1.17.6) and I am also hit with OutOfMemoryError: Java heap space:
org.apache.maven.surefire.booter.SurefireBooterForkException: There was an error in the forked process
Java heap space
java.lang.OutOfMemoryError: Java heap space
at java.base/java.util.Arrays.copyOf(Arrays.java:3541)
at java.base/java.io.ByteArrayOutputStream.toByteArray(ByteArrayOutputStream.java:187)
at org.testcontainers.containers.ContainerState.copyFileToContainer(ContainerState.java:337)
at org.testcontainers.containers.ContainerState.copyFileToContainer(ContainerState.java:308)
at org.testcontainers.containers.GenericContainer$$Lambda/0x00007f8cc0cd0860.accept(Unknown Source)
at java.base/java.util.LinkedHashMap.forEach(LinkedHashMap.java:986)
at org.testcontainers.containers.GenericContainer.tryStart(GenericContainer.java:444)
at org.testcontainers.containers.GenericContainer.lambda$doStart$0(GenericContainer.java:357)
at org.testcontainers.containers.GenericContainer$$Lambda/0x00007f8cc0cc4000.call(Unknown Source)
at org.rnorth.ducttape.unreliables.Unreliables.retryUntilSuccess(Unreliables.java:81)
at org.testcontainers.containers.GenericContainer.doStart(GenericContainer.java:347)
at org.testcontainers.containers.GenericContainer.start(GenericContainer.java:333)
at org.testcontainers.containers.ContainerisedDockerCompose.invoke(ContainerisedDockerCompose.java:64)
at org.testcontainers.containers.ComposeDelegate.runWithCompose(ComposeDelegate.java:254)
at org.testcontainers.containers.ComposeDelegate.createServices(ComposeDelegate.java:163)
at org.testcontainers.containers.DockerComposeContainer.start(DockerComposeContainer.java:137)
Yes, I tried a lot of workaround but I'm stuck. I keep version 1.8.0 😕 no choice
A fix proposal has been pushed here https://github.com/testcontainers/testcontainers-java/issues/7239 it's working for me
@gba-foundever Do you mean this patch: https://github.com/alexanderankin/testcontainers-java/commit/21cbc05d6633bcff443478c7844851fbf5e1ea66 ? Thx
@kiview Hello! Any chance of having the patch at https://github.com/alexanderankin/testcontainers-java/commit/21cbc05d6633bcff443478c7844851fbf5e1ea66 be included in the next release of TC by any chance? :) Thx
The patch seems to work around the problem and not fix it from what I see. See the IF at https://github.com/alexanderankin/testcontainers-java/commit/21cbc05d6633bcff443478c7844851fbf5e1ea66#diff-f4e8b058bc84a0af53093405bca79fe426972bd141dc2f03a9d54345000e6b28R47
In our case we use Docker in Docker and we have to use withCopyFileToContainer(). See the code in the first message above.
Is it possible to give some priority to this? I believe putting the docker compose file in the root of the project is something a lot of people do (Even start.spring.io generates it like that). All the duplicates of this issue (#7239, #1348, #2863) are a testament to this being quite an issue for a lot of people currently.
Even some mechanism to tell testcontainers to not copy every file, but just .env for example would already be great I guess.